How to measure exploratory tests ?

Hello guys,

Many people that are not from the QA area doesn’t know how to measure or what are the advantages of doing exploratory tests, but it is a technique really powerful when used correctly. Its effectiveness depends on several intangibles: the skill of the tester, their intuition, their experience, and their ability to follow hunches.

 

Value

• detects subtle or complex bugs in a system (that are not detected in targeted testing)
• provides user-oriented feedback to the team

Exploratory testing aims to find new and undiscovered problems. It contrasts with other more prescribed methods of testing, such as test automation, which aims to show scripted tests can complete successfully without defects. It will help you write new automated tests to ensure that problems aren’t repeated.

If you have any doubs about Exploratory tests, like examples and what are the advantages of doing it, have a look on the video below first:

99 Second Introduction to Exploratory Testing | MoT

 

When should you perform exploratory tests

Exploratory testing works best on a system that has enough functionality for you to interact with it in a meaningful way. This could be before you release your first minimum viable product in beta or before you release a major new feature in your service.

How to measure

Charter

• Mission for the session
• What should be tested, how it should be tested, and what problems to look for
• It is not meant to be a detailed plan
• Specific charters provide better focus, but take more effort to design: “Test clip art insertion. Focus on stress and flow
• techniques, and make sure to insert into a variety of documents. We’re concerned about resource leaks or anything else that might degrade performance over time.”

99 Second Introduction to Charters | MoT

 

Time Box

• Focused test effort of fixed duration
• Brief enough for accurate reporting
• Brief enough to allow flexible scheduling
• Brief enough to allow course correction
• Long enough to get solid testing done
• Long enough for efficient debriefings
• Beware of overly precise timing

• Short: 60 minutes (+-15)
• Normal: 90 minutes (+-15)
• Long: 120 minutes (+-15)

Debriefing

 

Mind maps

Mind maps can be useful to document exploratory testing in a diagram, instead of writing the scenarios. It is a visual thinking tool and are quick and easy to record as they don’t follow a linear approach.

 

Session metrics

The session metrics are the primary means to express the status of the exploratory test process. They contain the following elements:

• Number of sessions completed
• Number of problems found
• Function areas covered
• Percentage of session time spent setting up for testing
• Percentage of session time spent testing
• Percentage of session time spent investigating problems

 

Coverage

• Coverage areas can include anything
• Areas of the product
• Test configuration
• Test strategies
• System configuration parameters
• Use the debriefings to check the validity of the specified coverage areas

 

Reporting

• Create a charter
• Features you’ve tested
• Notes on how you conducted the testing
• Notes on any bugs you found
• A list of issues (questions and concerns about the product or project that arose during testing)
• Extra materials you used to support testing
• How much time you spent creating and executing tests
• How much time you were investigating and reporting bugs
• How much time you were setting up the session

 

Tools

I like to use Katalon or Jing, but to be honest this is just to record and take screenshots of the test sessions. To do these kind of tests you just need a paper and a pen to write your notes, concerns and questions.

 

Resources:

http://www.satisfice.com/sbtm/

http://www.satisfice.com/presentations/htmaht.pdf

https://www.gov.uk/service-manual/technology/exploratory-testing