Tag: test execution

Bug hunting - Where to look

In my previous post I wrote about knowing your prey. In this post, I’ll talk about good places to go hunting. All these tips are dependent on knowing the application you are testing. The better you know the ground, the more you will know where to look and the easier it will be to spot when something’s not right.

Functionality bugs

This is where most testers spend most of their time looking. Testers are pretty good at finding bugs here because they can be uncovered using your normal boundary value analysis type tests and other simple destructive tests like putting rubbish values into input fields.
They will enjoy a brief period of success when they find a few high priority bugs. Sadly many testers then continue to invest their time running less and less important tests of this type and find less and less important defects. A more experienced tester will recognise the rapidly diminishing rate of return on the investment of his time, and have a look in some other areas instead.

A trap so called “automation testers” or “developers in test” often fall into is to limit their testing to this area, because the tests are easy to express in test code and are easy to make pass.

Integration bugs

Integration testing is an often discussed and frequently practised type of testing, but again the temptation is to gain false confidence by testing only one aspect of it. Can the two systems talk to each other and does data that is sent from System A successfully arrive in System B? Yes? Good! A lot of integration testing stops at this point, but there’s much more fun to be had.

What happens if the message doesn’t arrive successfully in System B? A number of things could happen. Data in the two systems could get out of sync, or all subsequent messages could get locked up behind the one that didn’t get though, or the maybe no-one would notice that the connectivity had been lost until days later.

Systems operate in real life, and sadly real life involves outages that happen for any number of reasons. A good tester will want to know how his system behaves in this part of the life cycle.

Data quality bugs

Any experienced test will tell you the creation of useful quantities of test data is hard work. As a result, one of two things frequently happen. Sometimes the test data is created specifically for the tests. This is great for proving the various happy path tests, but this artificially created data often doesn’t reflect the myriad of different types of data that the system will have to deal with in the real life. Alternatively, the only data that is available is a copy, or more likely a subset of the live data. It can be often be hard to get the specific data sets you need to test all your edge cases. In both cases, it’s easy to find there are tests you might want to run, but can’t.

Another pit fall is to assume that all the data in the production environment is of good quality and is complete. All too often it isn’t. How will your system cope when it encounters bad data?

Environmental issues

Have you ever worked anywhere where there are stable and plentiful test environments that are accurately represent the live environment? Me neither. Just because something works in your test environment doesn’t necessarily mean it will work in your production environment where the config and architectural topography is likely to very different. For example, connectivity between the different systems in your test environment might be dead easy because it is a “de-militarised-zone” and is a virtualised environment, but your live environment is likely to have all sorts of firewalls.

I’m sure you’ll think of many more places to look. Please leave your ideas in the comments below!

Bug hunting - Know your prey

It’s easy for testers to get distracted by processes or the latest tools but really, it’s all about finding bugs. Lovely, horrible bugs. So what do we know about them?

Bugs live in hives

Bugs are sociable creatures. Where there is one, there is very likely to be a whole family of them huddled together in the dark. A tester who finds a bug is well advised to dig a little deeper or push a little harder in the same area and see what else is lurking. He will often be rewarded with more finds.

Bugs are sociable creatures

Why does this happen?

  • The area of the application was particularly complex and fiddly to develop increasing the likelihood of errors.
  • The component under test is dependent on other components that are themselves buggy.
  • This part of the code was written in a hurry. Possibly just before a deadline was due. Possibly on a Friday afternoon. Possibly at three in the morning. All these things would make human error more likely.
  • The developer may have misunderstood the requirement. If one part of a requirement has been misunderstood then it’s quite likely that other parts of it have been misunderstood too.

They are hard to kill

Just because a bug has been found and fixed doesn’t mean it can forgotten about. It’s very easy for bugs to re-spawn even after they’ve been squished by a code fix. For example, if the fix for a bug was only patched into a test environment rather than being properly included in the code base and deployed, subsequent versions of the code won’t contain the fix and the bug will regress. If the tests are manual the tester needs to make sure all his bug fixes from at least two or three versions of code back have stayed fixed by including the tests for them in his regression test pack. This is where automated tests are great, but only if either the automated test found the bug in the first place, or a new automated test has been written specifically to test for the bug.

Perhaps controversially, in my experience automated tests are not very good at finding bugs because they are designed to pass rather than designed to fail. Look out for another post on this topic later, so save your hate mail until then please.

They are seldom where you expect them to be

Many bugs that evade testers are a result of a failure of test analysis rather than a failure of test execution.

Testers seldom make mistakes and miss bugs when they are executing test scripts (although if course it does happen). This means that if a tester has identified a test, designed and executed it, he will usually find the bug if there is one lurking in the code. If he’s looking for it, he’ll probably find it.

Missed bugs are most often a failure of test analysis rather than a failure of test execution.

Most bugs that get missed are where the tester didn’t even think to look. This is really frustrating for testers, especially when the bugs would have been really easy to find if they had even had a quick look in the affected area. It’s also frustrating for everyone else, because people start thinking the testers can’t be relied upon to find even simple bugs.

The analogy here is someone who has lost their kitten. They hunt high and low, looking in all the cupboards, checking the drains outside, checking in the attic, and eventually even taking up their floorboards and dismantling the staircase to try and find the kitten. In short they do a very thorough job of destroying their house in their hunt for the kitten which is actually with the next door neighbours enjoying a saucer of milk.

The lessons to be learned

Coverage is king. It is more important to get your test coverage right rather than the perfect the detail of each test.

Be prepared to change the focus of your testing as you learn more about the thing you are testing. Don’t waste your time slavishly running thousands of low priority tests which all seem to be passing when you are suspicious about an area that has shown to have bugs.

Make sure any defects you find are covered by tests in your regression suite, and make sure you run that suite often.

If you’re reviewing tests, (either your own or other people’s) spend less time reviewing the tests that have been written, and more time reviewing what test haven’t been written. This is where the bugs will be.

Have your critical thinking reviewed by your peers and have heated discussions with them about where to spend your time.

In my next blog in this category, I’ll discuss the different sorts of places where bugs like to hang out.

LinkedIn discussion on this posts here.