How to measure exploratory tests ?

Hello guys,

Many people that are not from the QA area doesn’t know how to measure or what are the advantages of doing exploratory tests, but it is a technique really powerful when used correctly. Its effectiveness depends on several intangibles: the skill of the tester, their intuition, their experience, and their ability to follow hunches.

Value

detects subtle or complex bugs in a system (that are not detected in targeted testing)
provides user-oriented feedback to the team

Exploratory testing aims to find new and undiscovered problems. It contrasts with other more prescribed methods of testing, such as test automation, which aims to show scripted tests can complete successfully without defects. It will help you write new automated tests to ensure that problems aren’t repeated.

If you have any doubs about Exploratory tests, like examples and what are the advantages of doing it, have a look on the video below first:

99 Second Introduction to Exploratory Testing | MoT

When should you perform exploratory tests

Exploratory testing works best on a system that has enough functionality for you to interact with it in a meaningful way. This could be before you release your first minimum viable product in beta or before you release a major new feature in your service.

How to measure

Always test in sessions:

Charter
Time Box
Debriefing
Mind Maps

Charter

Mission for the session
What should be tested, how it should be tested, and what problems to look for
It is not meant to be a detailed plan
Specific charters provide better focus, but take more effort to design: “Test clip art insertion. Focus on stress and flow
techniques, and make sure to insert into a variety of documents. We’re concerned about resource leaks or anything else that might degrade performance over time.”

99 Second Introduction to Charters | MoT

Time Box

Focused test effort of fixed duration
Brief enough for accurate reporting
Brief enough to allow flexible scheduling
Brief enough to allow course correction
Long enough to get solid testing done
Long enough for efficient debriefings
Beware of overly precise timing

Sessions time:

Short: 60 minutes (+-15)
Normal: 90 minutes (+-15)
Long: 120 minutes (+-15)

Debriefing

Measurement begins with observation
Session metrics are checked
Charter may be adjusted
Session may be extended
New sessions may be chartered
Coaching happens

Mind maps

Mind maps can be useful to document exploratory testing in a diagram, instead of writing the scenarios. It is a visual thinking tool and are quick and easy to record as they don’t follow a linear approach.

Session metrics

The session metrics are the primary means to express the status of the exploratory test process. They contain the following elements:

Number of sessions completed
Number of problems found
Function areas covered
Percentage of session time spent setting up for testing
Percentage of session time spent testing
Percentage of session time spent investigating problems

Coverage

Coverage areas can include anything
Areas of the product
Test configuration
Test strategies
System configuration parameters
Use the debriefings to check the validity of the specified coverage areas

Reporting

Create a charter
Features you’ve tested
Notes on how you conducted the testing
Notes on any bugs you found
A list of issues (questions and concerns about the product or project that arose during testing)
Extra materials you used to support testing
How much time you spent creating and executing tests
How much time you were investigating and reporting bugs
How much time you were setting up the session

Tools

I like to use Katalon or Jing, but to be honest this is just to record and take screenshots of the test sessions. To do these kind of tests you just need a paper and a pen to write your notes, concerns and questions.

Resources:

http://www.satisfice.com/sbtm/

http://www.satisfice.com/presentations/htmaht.pdf

https://www.gov.uk/service-manual/technology/exploratory-testing

Exploratory tests as a complement of regression pack

Hey guys, today I will post about a technique to test new/old features when you don’t have a proper regression pack, or when the product is not stable, or the time is really limited.

Why should we perform exploratory tests combined to regression tests when the product is not stable ?

Because testers will be involved in minimum planning and maximum test execution (Allowing to find more bugs than just following the regression pack)
It will find different approach to test the same scenario, allowing to have a better coverage about the software, so it will have a higher possibility to find a trick bug
This is an approach that is most useful when there are no or poor specifications and when time is severely limited

How to perform the exploratory tests ?

The test design and test execution activities are performed in parallel typically without formally documenting the test conditions, test cases or test scripts. This does not mean that other, more formal testing techniques will not be used.

There is too much evidence to test, tools are often expensive, so investigators must exercise judgment. The investigator must pick what to study, and how in order to reveal how, in order to reveal the most needed information. This takes time and you need a huge knowledge of the software, until there you are gaining experience doing exploratory tests on te software.

[When my boss asks how the tests are going]

In this case it’s serving to complement the regression pack, which is a more formal testing, helping to establish greater confidence in the software. Exploratory testing can be used as a check on the formal test process by helping to ensure that the most serious/tricks defects have been found.

Programs fail in many ways:

Why should you not automate exploratory tests?

With a script, you miss the same things every time
Automated scripts are completely blind by design, this is more about a human interaction and different approaches
Different programmers tend to make different errors. (This is a key part of the rationale behind the PSP). A generic test suite that ignores authorship will overemphasize some potential errors while underemphasizing others
The environment in which the software will run ( platform, competition, user expectations, new exploits) changes over time

So, if is not stable what are the types of the defects I am finding?

A manufacturing defect appears in an individual instance of the product. This is the type of the defect you are finding on non stable softwares and what you try to find doing exploratory tests.
A design defect appears in every instance of the product. The challenge is to find new design errors, not to look over and over and over again for the same design error

To end this post, Exploratory testing is called when you want to go beyond the obvious or when I don’t trust the software, which is most of the time.

Resources:

http://istqbexamcertification.com/what-is-exploratory-testing-in-software-testing/

http://www.kaner.com/pdfs/QAIExploring.pdf

http://www.satisfice.com/articles/what_is_et.shtml

Tests Coverage

As a tester you have a different way to think about the scenarios. You know that you need to think beyond the scenarios. So, how do you know when it will be enough ? When will you have 100% of tests coverage ?

You probably already found a bug out of the requirements, and in a specific sequence of steps. I normally find these kind of bugs with exploratory tests, when I have time to free my creative side and start to do different ways to test the same thing. Developers follow the requirements, they don’t do exploratory tests, usually they think even the user has the possibility to do the same step in a different way, they shouldn’t (Because it’s not the right way).

In my humble opinion, if your software allow to do the same function in 1000 different ways, you should be prepare to test every single “invalid” way, because this will increase the trust in your software. If I find a single stupid bug in an application, like an error when I send invalid characters, I start to think what type of software was delivered, like neither the basic simple stupid scenario of invalid characters was tested, imagine the more complex ones… This could be low priority, but if you ignore this, you need to face there are many people like me (critical detail vision), that see these kind of things and lose the confidence on the software and to be honest, the respect too.

Imagine you have all the requirements:

And you have the system software in this another circle:

But in the real world, we don’t have every requirement covered by the system, we have something like this:

Which means you will have some parts in your system not covered by your requirements and you have some parts in your requirements not covered by your system. It is exactly in this part that we, testers should start to think about. We need knowledge of both of the parts, and this takes time.

So, you don’t need to worry cover 100% of your tests in the beginning, you need to worry if you know everything about what you are testing, some scenarios you just figure out when you are testing, because you are pretending you are an user. You need a good background, someone to sit next to you, or some good documentation about what you will test, spend some time exploring the app before you start the scenarios. This will create your first impression of the software and you will be more into it and the user experience.

Finally, my advice to know if you have a good test coverage is:

Exploratory tests, this will help you to find unknown scenarios between system and requirements. This is a type of test you can’t automate, it involves more about your creativity than objective steps. Sometimes is just a different sequence that you do and you can find a critical bug.
Kick-off requirements, this is another thing that helps to reduce the unknown scenarios, like if you have an explanation about what a new function will do, you can raise and think in points which you already know and maybe nobody thought yet. As I said before, it’s better if you have a good background about what is coming.
System flow, the last key is try to understand the gaps and the flow of the software, like what is the flow to a function update something in database. It seems very technical, but this will help you to think about scenarios that might crash when you do in a different sequence or if you do many times, or if you don’t wait a specific time, this is quality assurance 🙂

See you next week !