Exploratory testing.

Not a long time ago I had a discussion with someone from my organization about testing and the way we test our product at the organization I work for. We had an almost endless discussion about what exploratory testing is and why we practice this kind of testing. I was shocked to hear him say that: “Exploratory testing should be a tester’s only thing”. From my perspective this doesn’t have to be tester’s only, testers could surely use the input of consultants and / or developers when exploratory testing. And when needed perhaps we should test together! I got the feeling that he had no clue about what Exploratory testing really means and what it could mean for the stability of our product. Perhaps you have experienced the same discussion, or experiencing it right now. So here is what I think about exploratory testing.

So what is exploratory testing?

Let’s start off with why we do exploratory testing. Many of you reading this article will probably have heard of the term “Exploratory testing” before since there is already written so many about this topic. Many testers already apply this way of testing  for some time.

In short, Exploratory testing is testing a system under test (or a part of it) without following a strict test script. Take a part of the system under test where multiple components come together and test the interaction of these components. It will give you information about how the system works and if the interaction between components is correct and understandable for the (most of the times less technical) end user. So you will be designing and executing test cases on the spot. Use your curiosity to experiment with the components and see if what ever happens is as you expected it to.

By exploring the functionality you probably execute existing test cases (happy path) but you will also test paths not covered in test cases (perhaps resulting in new (automated) test cases to be executed). The more you know about the product that is developed and the more you are experienced with testing, the better the result of the exploratory test.

Is exploratory testing only for testers?

From my perspective I would say: No, not totally! Perhaps a exploratory test is at best when executed by a tester since our testing skills and mindset would help when testing. But like I mentioned before: The more you know about the product that is developed…. That would indicate that everyone from developers to consultants could be involved in exploratory testing. Not only testers have knowledge about the product developed. For example: Consultants usually know how one or more customers use the product. This knowledge can be of great value when they are testing. Not all developers and tester will have this knowledge.

Developers for example know how they developed the product and where to find “weak” spots in the application. They can point them out for you to test, but can also test these themselves, learn, fix bugs and deliver a better (part of the) product. Developers can also practice exploratory testing when doing a code review. Besides literally only reviewing the code, click trough the component if possible and look for some simple bugs. By saying that I’m not saying that testers are obsolete, I think testers just shouldn’t be bothered with finding bugs that are blocking the flow just after two mouse clicks.

Why exploratory testing is so important:

Following test scripts is repetitive work, testing a new release for the second time by following a testscript will probably not give you new results. It’s most unlikely that these test will now suddenly fail. By exploring (parts of) the system under test you will probably find new results, find new testcases, write them down and perhaps automate them. The first thing I do when I’m done with testing a new version of our application via a manual regression testscript is exploratory testing. This can also be done when your in an agile team and the user store is completed because it reached its definition of done, after that there is nothing holding you back to explore the functionality and put it to done when you as tester are completely satisfied with the result.

How to overcome some of the downsides of Exploratory testing

Unfortunately exploratory testing also has some downsides. The main problem I experienced was that you can get easily lost in time when exploratory testing, you’ll have to watch out that it down not result in some kind of exhaustive testing. Best practice for me is to time box each exploratory testing session. For each time frame I pick one or two (when testing interaction) particular parts of the system under test, set some objectives what to test, write down or log my findings and when the time is up I move on to the next test.

An easy and lightweight tool to help you with creating sessions for exploratory testing is Sessiontester. As described by their website: “It has a timer so you limit your test sessions to a particular length, and it provides an easy way to record session notes.”

When to apply exploratory testing

Here is a summary of situations in which you could prefer applying exploratory testing:

  • When testing with test scripts is completed but you do not feel totally confidant with the delivered functionality.
  • When fast feedback about a functionality is needed.
  • When the functionality is already mature and no or minor changes has been made.
  • When there is no time to execute a (scripted) functional test but feedback is still required.
  • When testing the frontend of a webapplication with multiple browsers (Internet explorer, Firefox, Safari etc.).

Some interesting links:

James bach about exploraty testing.
A PPT about Exploratory testing in Agile by Elisabeth Hendrikson
The sience of Exploratory testing