Testing Complexity
Some real-world examples of Testing Complexity
Sometimes bugs are missed by QA because they are unique. They can be hard to find during normal regression testing.
Here are a couple of examples of bugs making it to Production due to complexity of testing:
France Translation
There was a bug that only occurred for French users of a website that I was testing. The bug occurred because the translated text was too long and as a result, a dialog action button was not visible.
QA missed this in testing because the focus on testing was on English. QA wasn't notified that a particular dialog box was being translated to French - which wasn't an issue at the time since text content wasn't part of QA testing.
JBoss Installation Instructions
At another company that I worked at, I was responsible for software QA while at the same time doing some installs and training for customers. The company developers put together some documentation on how customers can install the product. The problem was that the document only handled a particular set of customers and several customers started complaining about the inaccuracy of the documentation.
QA missed this because the document was working for many customers. There were no major changes in the application that would have resulted in needing to test the installation process with the customer instructions. What happened was the sales team was selling the product to a different set of customers that required QA to check the documentation.
I then walked through the installation with a couple of customers while at the same time updating the document to make sure that it was clear for particular environments and make sure the terminology matched the audience that was doing the install.
Five Things I Learned to Handled Future situations
While the above is very specific examples, there are many more similar bug patterns that I have seen over my many years of QA testing. Here are some things that I have learned:
- It doesn't hurt to every once in a while to take a step back and manually go through the sales flow of the application. Are things working as they should be? How does the product look to new customers.
- Work with Developers to get some QA tools to help with testing. The French problem was being solved by having a special URL query for QA to force the page to load with a particular translation. This tool makes it easy to test the key languages when major changes happen in the application. Also, it makes it easy for automation to test the button visibility against various languages.
- Review the code changes. It doesn't hurt to check out the code review to see what has been changed. Many times I have found that a code change was made without thinking of other consequences - for example, what happens if customers use non triditional UTF-8 characters.
- Learn from the Bugs that Excape QA. One of my weekly tasks is to review the causes of customer reported issues and to see how it was missed.
- Learn new QA tools. There's always something new to learn in QA. There's always some new Chrome Extension, JQuery tip, database query and security lock down.