It’s easy for testers to get distracted by processes or the latest tools but really, it’s all about finding bugs. Lovely, horrible bugs. So what do we know about them?
Bugs live in hives
Bugs are sociable creatures. Where there is one, there is very likely to be a whole family of them huddled together in the dark. A tester who finds a bug is well advised to dig a little deeper or push a little harder in the same area and see what else is lurking. He will often be rewarded with more finds.
Bugs are sociable creatures
Why does this happen?
- The area of the application was particularly complex and fiddly to develop increasing the likelihood of errors.
- The component under test is dependent on other components that are themselves buggy.
- This part of the code was written in a hurry. Possibly just before a deadline was due. Possibly on a Friday afternoon. Possibly at three in the morning. All these things would make human error more likely.
- The developer may have misunderstood the requirement. If one part of a requirement has been misunderstood then it’s quite likely that other parts of it have been misunderstood too.
They are hard to kill
Just because a bug has been found and fixed doesn’t mean it can forgotten about. It’s very easy for bugs to re-spawn even after they’ve been squished by a code fix. For example, if the fix for a bug was only patched into a test environment rather than being properly included in the code base and deployed, subsequent versions of the code won’t contain the fix and the bug will regress. If the tests are manual the tester needs to make sure all his bug fixes from at least two or three versions of code back have stayed fixed by including the tests for them in his regression test pack. This is where automated tests are great, but only if either the automated test found the bug in the first place, or a new automated test has been written specifically to test for the bug.
Perhaps controversially, in my experience automated tests are not very good at finding bugs because they are designed to pass rather than designed to fail. Look out for another post on this topic later, so save your hate mail until then please.
They are seldom where you expect them to be
Many bugs that evade testers are a result of a failure of test analysis rather than a failure of test execution.
Testers seldom make mistakes and miss bugs when they are executing test scripts (although if course it does happen). This means that if a tester has identified a test, designed and executed it, he will usually find the bug if there is one lurking in the code. If he’s looking for it, he’ll probably find it.
Missed bugs are most often a failure of test analysis rather than a failure of test execution.
Most bugs that get missed are where the tester didn’t even think to look. This is really frustrating for testers, especially when the bugs would have been really easy to find if they had even had a quick look in the affected area. It’s also frustrating for everyone else, because people start thinking the testers can’t be relied upon to find even simple bugs.
The analogy here is someone who has lost their kitten. They hunt high and low, looking in all the cupboards, checking the drains outside, checking in the attic, and eventually even taking up their floorboards and dismantling the staircase to try and find the kitten. In short they do a very thorough job of destroying their house in their hunt for the kitten which is actually with the next door neighbours enjoying a saucer of milk.
The lessons to be learned
Coverage is king. It is more important to get your test coverage right rather than the perfect the detail of each test.
Be prepared to change the focus of your testing as you learn more about the thing you are testing. Don’t waste your time slavishly running thousands of low priority tests which all seem to be passing when you are suspicious about an area that has shown to have bugs.
Make sure any defects you find are covered by tests in your regression suite, and make sure you run that suite often.
If you’re reviewing tests, (either your own or other people’s) spend less time reviewing the tests that have been written, and more time reviewing what test haven’t been written. This is where the bugs will be.
Have your critical thinking reviewed by your peers and have heated discussions with them about where to spend your time.
In my next blog in this category, I’ll discuss the different sorts of places where bugs like to hang out.