Risk Analysis when descoping scenarios

Hey hey hey guys, today I will post about how to manage a risk analysis and what type of questions you should ask when descoping scenarios from your regression pack.

Regression testing must cover certain conditions in order to be effective.

Regression coverage isn’t a question of the number of cases as much as covering the conditions that

  1. Ensure functionality works
  2. Bugs found have been resolved
  3. A functional area can handle some amount of negative or destructive behaviour

Regression testing that addresses all three aspects should guide your testing efforts more than focusing on the number of cases.

Assessing coverage by the number of test cases is difficult — one case can cover many conditions or one case could provide coverage of only a single condition. If I provided a response by a metric of one case will cover what is needed, you might be underestimating regression testing that such a response would be potentially dangerous or misleading. So addressing the question how I would plan regression testing might be more practical. The questions you should keep in mind are:

  • Is this feature stable ?
  • Is this feature dependent on other less-stable features ?
  • What will happen if this feature fails in a subtle/medium/catastrophic manner ?


You can find some techniques to develop a risk analysis:

  • A close reading of the requirements specification, design specifications, user documentation and other items.
  • Pair-wise, you can use a tool to mix the scenarios with different combinations and reduce the scope of your regression pack.
  • A¬†sequence of one-on-one or small-group sessions with the business and technology experts in the company.
  • A tool built from Six Sigma DMAIC model. You can simply write down each function and it’s permutations in any manner that works for your business. The whole development team provides input, while the QA professional manages the document and detail of the information and input gathered.

If you are in a small company, like a startup, my opinion is: You can write the scenarios for you regression pack and ask for some developer to review it. So, you will have a regression pack built with their product code knowledge and you with your global view. QA ends up with factual data on what areas and functions of the application are most critical.

Finally, you put together a regression pack that focuses the higher priority functions while exercising the lesser functions less in depth. The key is to test the higher risk areas first and frequently, while still testing the lesser risk functions more superficially or via rotating test suites based on priority or risk.

It seems too simple, but it works to improve communication between team members while also documenting application functionality.

Stay updated with the last features, your regression pack¬†needs to be fresh with the scenarios. Each new version will contain some new feature as it also can remove an old one. It’s important to try to execute that final test cycle with the freshest view you can.







Exploratory tests as a complement of regression pack

Hey guys, today I will post about a technique to test new/old features when you don’t have a proper regression pack, or when the product is not stable, or the time is really limited.

Why should we perform exploratory tests combined to regression tests when the product is not stable ?

  • Because testers will be¬†involved in minimum planning and maximum test execution (Allowing to find more bugs than just following the regression pack)
  • It will find different approach to test the same scenario, allowing to have a better coverage about the software, so it will have a higher¬†possibility to find a trick bug
  • This is an approach that is most useful when there are no or poor specifications and when time is severely limited


How to perform the exploratory tests ?

The test design and test execution activities are performed in parallel typically without formally documenting the test conditions, test cases or test scripts. This does not mean that other, more formal testing techniques will not be used.

There is too much evidence to test, tools are often expensive, so investigators must exercise judgment. The investigator must pick what to study, and how in order to reveal how, in order to reveal the most needed information. This takes time and you need a huge knowledge of the software, until there you are gaining experience doing exploratory tests on te software.


[When my boss asks how the tests are going]



In this case it’s serving to complement the regression pack, which is a more¬†formal testing, helping to establish greater confidence in the software. Exploratory testing can be used as a check on the formal test process by helping to ensure that the most serious/tricks defects have been found.

Programs fail in many ways:

Screen Shot 2016-02-24 at 23.14.00


Why should you not automate exploratory tests?

  • With a script, you miss the same things every time
  • Automated scripts are completely blind by design, this is more about a human interaction and different approaches
  • Different programmers tend to make different errors. (This is a key part of the rationale behind the PSP). A generic test suite that ignores authorship will overemphasize some potential errors while underemphasizing others
  • The environment in which the software will run ( platform, competition, user expectations, new exploits) changes over time


So, if is not stable what are the types of the defects I am finding?

  • A manufacturing defect appears in an individual instance of the product.¬†This is the type of the defect you are finding on¬†non stable softwares and what you try to find doing¬†exploratory tests.
  • A design defect appears in every instance of the product. The challenge is to find new design errors, not to look over and over and over again for the same design error


To end this post, Exploratory testing is called when you want to go beyond the obvious or when I don’t trust the software, which is most of the time.






10 Steps to setting up the QA Area

Hello guys, today I will post about some steps that you can follow to create the QA area from the scratch.

1 – Make questions like:

  • Do you have any written scenarios ?
  • Who is writing the scenarios and in what phase ?
  • QA team will need create its own scenarios, what phase may¬†the team do that ?
  • Who will need to look¬†the scenarios, just QA and dev area ?
  • Who will run the tests DEV¬†and QA¬†only ?
  • How the application is working ?
  • What are the critical scenarios ? Like, what are the scenarios which will crash the functions/app/process ?
  • What are the most used scenarios (Like create an account…) ?
  • What are the scenarios which are¬†more unstable, like if you change a simple thing you will need to test this scenario every time ?
    • What are the most repetitive tests ?
  • Do you need run the regression everyday ? before every release ? (regression tests, exploratory tests, monkey tests, stress tests, performance tests) every time after a push ? (smoke tests)
  • Do you¬†have a QA environment ?
  • If you are in a mobile/web project, remember to make these questions:
    • What are the most used devices/browsers/OS ?
    • What are the most unstable devices/browsers/OS ?
    • What are the most used versions on¬†these¬†browsers/OS ?
    • What are the most unstable versions on¬†these¬†browsers/OS ?
  • If you are in a mobile project:
    • The app is hybrid, native or web app ?
  • If you are in a web project:
    • Is this site responsive ?
  • Do you have a process of QA, like SCRUM, Kanban ?
  • If you find a bug in a task, do you return the task or create the bug to be fix separately ?

2 – Understand what will be first priority and until where you will reach (Like performance tests, integration ui tests, etc…), so you can have a big picture of the project

3 – Remember that if you are using BDD, it is just 3 layers (Don’t expose your code)

4 РDecide what are the scenarios that must be in the regression (Priority).  What are the most repetitive tests ?

5 – Use tags for all the type of scenarios, like smoke, regression, iOS, android, manual. Even manual tests should be in the features inside the automation project

6 – Don’t use too many complex steps in your scenarios

7 – Try to re-use steps so you don’t need waste time

8 – Use examples

9 – Regression tests – Automated tests

  • Choose the tool, do a POC with different frameworks
    • On¬†this POC, choose the most simple and important scenario (E.g. Create an account)
  • Considerate the language which the¬†developers are using (You never know when you will need help)
  • Distribute the right level of tests between unit (70%), integration (20%) and manual tests (10%) – new functions
  • Include the Automation in your development
  • Decide if you need to create the automation project in¬†the same¬†or separated project
  • Use CI since the beginning
  • Decide what must to have in the report
  • Choose a good report from¬†the beginning
  • Structure of Scripts.¬†Create¬†or follow a Code Style Guideline

10 – Form a team of automation and equality the level of knowledge. Make some meetings to prepare people in your team to use the tool with wisdom.

I think it’s pretty much this. Sorry guys if I forgot something… If I remember anything else I will update here. Thank you ! Feel free for comments and your opinion always ! See you next week¬†!

How To Optimize Regression Test Suite Effectively

Hi guys, I read a article about Optimize Regression Tests and for this reason I will summarize with my words what you can do more effectively with Regression Tests. According to different surveys, more than 70% of software work relates to application maintenance and improvement. This means that you will have a lot of work updating your regression test code and also keeping focus on the newer application functionality on the same time.

regression testing, optimize regression test suitesThere are a few factors that affect the effectiveness of the regression test suites. So, how to maintain the effectiveness of the regression test suite, or optimize regression test suite effectively?

  • RTS – Regression Test Selection is one of the most popular methods for test case suite optimization. Divide the test suite into reusable test cases, retestable test cases and obsolete test cases. Apart from all these, it also creates new test cases that test the program for areas not covered in current test cases.
  • TRACK MECHANISM – The effectiveness of the regression test suite can be easily maintained by monitoring the changes to the test suite. A clearly outlined process will ensure that only tests that are useful to the entire testing strategy get added to the test suite, which ensures the efficiency and usability of the test harness at a high level.
  • PERIODIC CLEANUP – Considering periodic cleanup of old tests, all the existing tests in the test suite need to be analyzed for their effectiveness in a specific scenario. In such cases, the relevant regression test suites should also be eased out. It will ensure robustness of the regression test suite for a long period of time. You can also measure the effectiveness of regression test suites on a release-to-release basis. It will allow you to know the root cause for reduction in the effectiveness of the test harness if any, and enable you to take appropriate action on the same.
  • METRICS ANALYSIS – Collection of some metrics and their analysis could also be useful when it comes to the effectiveness of the regression test suite. You can consider different metrics such as percentage of defects found by the regression tests suite, their importance, etc.

You can consider optimization of tests when the tests get lengthy or huge, evaluate different regression test selection strategies and plan and explore changes in the regression test suite framework in order to maintain the effectiveness of the regression test suite.

If you have another method, tell to us commenting below.

Thank you again ūüôā

References: http://intersog.com/blog/How-To-Optimize-Regression-Test-Suite-Effectively

%d bloggers like this: