I love testing for the same reason that I love science. To be a scientist is to be curious about the world around you and learn about it through experimentation. To be a tester is to be curious about the software around you and learn about it through experiments that we call tests. We can benefit greatly from applying the scientific method to our testing.
In introducing me simultaneously to scepticism and to wonder they [my parents] taught me the two uneasily co-habiting modes of thought that are central to the scientific method.
– Carl Sagan
Software behaves differently to the physical cosmos. The laws of physics are constant but the the laws of software are infinitely fluid and different on every system you will test. It’s as if we testers travel to a new universe with every new application. We must approach each new universe with a sense of wonder and curiosity about the laws that govern it. We are told how it should behave in the requirements and the developers who made it tell us how they think it behaves. But we must be sceptical about all that what we are told, and learn for ourselves how each new testing universe really behaves.
Turning hypotheses into theories
The word “theory” is often mis-used in a dismissive way; “The theory of evolution is only a theory“. This mis-use of the word suggests that a theory is just someone’s idea that might not be true. This is just wrong. A theory is a model that has been proven by a number of observations, and can be used to predict the results of future observations. For example, the theory of gravity is based on lots of experiments and measurements, starting with Newton’s, and is really useful in predicting with great accuracy things like by how much passing asteroids will miss Earth.
This is different to a hypothesis, which is a proposed explanation without any expectation or evidence of its truth. This is often what people mean why they say have a theory about something. They actually have an idea about something which demands to be tested to see whether it’s true or not. The development of a hypothesis into a theory, and the subsequent revision of that that theory as we learn more is the foundation of how to do science. Science isn’t perfect, but it is by far the very best tool we have to learn about and describe the world, or in our case, our software. We must approach testing using the same technique scientists use to learn about the world.
1. Ask a question
First, be curious. What will happen if I follow this? How does this part of the system work? Are there any security flaws in this system? Is this system able to handle the demand from all our customers?
2. Do some research
Learn as much as possible about what you are researching or testing. What other research and testing has been done before? What was learned and what theories were strengthened or destroyed by this work? What methodological problems were experienced and how were they overcome?
It’s also good to get as familiar with the territory as possible. There’s no point trying to do a complex experiment in quantum mechanics unless you’ve got a strong understanding of the physics first, and likewise there’s no point trying to do tests in a complex application unless you have a strong understanding of the product first. A tester must be an expert on the product before he can be expert in testing it.
3. Come up with a hypothesis
Hypotheses can be positive or negative. Proving a positive hypothesis will increase your confidence in the quality of that part of the system under test, and disproving it will result in a bug or bugs being raised. Proving a negative hypothesis will result in a bug being raised. Disproving increases your confidence in the quality of that part of the software under test.
In testing, we set out to unequivocally prove or disprove hypotheses.
Hypotheses can also be used to describe why certain bugs happen. This is very helpful in focussing investigative work. For example:
The reason the error message is received is because the firewall is rejecting the in-bound message
There may be several such competing hypotheses, each suggesting a different cause for the problem. Each hypothesis can then be tested individually, ultimately disproving all of them apart from those that are true.
4. Design an experiment to test the hypothesis
One of the great things about the scientific method is that the results from experiments are published for peer review with enough information for the reviewing peers to repeat the experiments for themselves and see if their results are the same. This is also a perfect lesson for testers designing test scripts.
Scientific methods are designed to isolate and measure the effect of just one variable that is under test. Everything else in the experiment stays the same, including the method of measurement.
5. Conduct the experiment
Sadly, conducting testing experiments is often the least interesting part of the job. It can be seen as a mundane process of following instructions and clicking buttons over and over again.
When conducting the experiment, we must be engaged, dispassionate and un-emotive. We must record what we see, whether that supports an eager project manager’s aspirations or not and whether it supports our own expectations or not. This is where test automation can be really helpful.
We tend not be very critical when confronting evidence that supports our conclusion.
– Carl Sagan.
Imagine the scene: A tester has spent all day testing something; he’s feeling good about it, all his tests have passed and he’s looking forward to signing it off. But then something weird happens that doesn’t seem to be reproducible. Does he gloss over it, putting it down to a “glitch” and sign off, or does he suddenly doubt his previous tests and redouble his efforts to reproduce the unexpected result? This is when the psychology of a true tester is most apparent.
6. Analyse the results
The human brain can switch off and not notice anomalies, or equally as bad, can start to make up excuses for anomalies we might see. Although the human brain is brilliant at spotting patterns, it can sometimes see patterns that aren’t there.
Respect the facts, even when they are disquieting and they seem to contradict conventional wisdom.
– Carl Sagan
7. Construct a theory
The more tests you have run in a particular area, the more you learn about it, and you will soon be able to come up with a theory. The theory might be that a particular component always fails under certain specific conditions, or it may be that a theory that the system is secure against the 3 most common forms of security breach, or anything that is relevant. For any theory you propose, you must have the evidence to support it.
8. Repeat the process to strengthen or revise your theory
Once you’ve got your theory, every further test you run will either strengthen that theory and give you more confidence in it, or will disprove your theory.
Many positive test results can support a theory. It only takes one to destroy it.
– Stephen Hawkins
Depending on how you look at it this can either be dispiriting or uplifting. (Assuming of course the test that destroyed the theory is valid, repeatable, reliable, etc.) A true scientist will be excited by his theory being blown. He will think, “Wow! Everything we thought about this is wrong! Something else is happening that we don’t understand! We need to think harder and differently and go back to the drawing board to try to understand this! The world is not as it seems! How exciting!”
We must let let reality speak for itself, supporting a theory when its predictions are confirmed and challenging a theory when its predictions proven to be false
What this means for testers
The ability to think and reason is what separates us from the machines. This is the stuff that human beings are best at. Let’s let the machines do the boring, dirty and dangerous tasks of collecting the evidence for us so we can spend as much of our time as possible thinking critically.