Chaos Engineering: Why Breaking Things Should be Practiced

Hello guys,

Last week I went to the WebSummit 2018 Conference in Lisbon and I managed to join some of the AWS talks. The talk that I am posting today is about chaos engineering, which specifically address the uncertainty of distributed systems at scale. The aim of this practice is to uncover the system weakness and build confidence in the system’s capability. 

The harder it is to disrupt the steady state, the more confidence we have in the behavior of the system.  If a weakness is uncovered, we now have a target for improvement before that behavior manifests in the system at large.

Today I am going to post the video on the exact moment that this talk starts.

https://player.twitch.tv/?autoplay=false&t=02h05m17s&video=v333130731

This talk is presented by AWS Technical Evangelist Adrian Hornsby.

You can find tools to help you with the tests in this repo:

https://github.com/dastergon/awesome-chaos-engineering#notable-tools

 

References:

https://principlesofchaos.org/

https://www.twitch.tv/videos/333130731

Amazing repo with content/links about the topic: https://github.com/dastergon/awesome-chaos-engineering

How to measure exploratory tests ?

Hello guys,

Many people that are not from the QA area doesn’t know how to measure or what are the advantages of doing exploratory tests, but it is a technique really powerful when used correctly. Its effectiveness depends on several intangibles: the skill of the tester, their intuition, their experience, and their ability to follow hunches.

 

Value

  • detects subtle or complex bugs in a system (that are not detected in targeted testing)
  • provides user-oriented feedback to the team

Exploratory testing aims to find new and undiscovered problems. It contrasts with other more prescribed methods of testing, such as test automation, which aims to show scripted tests can complete successfully without defects. It will help you write new automated tests to ensure that problems aren’t repeated.

If you have any doubs about Exploratory tests, like examples and what are the advantages of doing it, have a look on the video below first:

99 Second Introduction to Exploratory Testing | MoT

 

When should you perform exploratory tests

Exploratory testing works best on a system that has enough functionality for you to interact with it in a meaningful way. This could be before you release your first minimum viable product in beta or before you release a major new feature in your service.

How to measure

Always test in sessions:
  1. Charter
  2. Time Box
  3. Debriefing
  4. Mind Maps

 

Charter

  • Mission for the session
  • What should be tested, how it should be tested, and what problems to look for
  • It is not meant to be a detailed plan
  • Specific charters provide better focus, but take more effort to design: “Test clip art insertion. Focus on stress and flow
  • techniques, and make sure to insert into a variety of documents. We’re concerned about resource leaks or anything else that might degrade performance over time.”

99 Second Introduction to Charters | MoT

 

Time Box

  • Focused test effort of fixed duration
  • Brief enough for accurate reporting
  • Brief enough to allow flexible scheduling
  • Brief enough to allow course correction
  • Long enough to get solid testing done
  • Long enough for efficient debriefings
  • Beware of overly precise timing
Sessions time:
  • Short: 60 minutes (+-15)
  • Normal: 90 minutes (+-15)
  • Long: 120 minutes (+-15)

Debriefing

  • Measurement begins with observation
  • Session metrics are checked
  • Charter may be adjusted
  • Session may be extended
  • New sessions may be chartered
  • Coaching happens

 

Mind maps

Mind maps can be useful to document exploratory testing in a diagram, instead of writing the scenarios. It is a visual thinking tool and are quick and easy to record as they don’t follow a linear approach.

 

Session metrics

The session metrics are the primary means to express the status of the exploratory test process. They contain the following elements:

  • Number of sessions completed
  • Number of problems found
  • Function areas covered
  • Percentage of session time spent setting up for testing
  • Percentage of session time spent testing
  • Percentage of session time spent investigating problems

 

Coverage

  • Coverage areas can include anything
  • Areas of the product
  • Test configuration
  • Test strategies
  • System configuration parameters
  • Use the debriefings to check the validity of the specified coverage areas

 

Reporting

  • Create a charter
  • Features you’ve tested
  • Notes on how you conducted the testing
  • Notes on any bugs you found
  • A list of issues (questions and concerns about the product or project that arose during testing)
  • Extra materials you used to support testing
  • How much time you spent creating and executing tests
  • How much time you were investigating and reporting bugs
  • How much time you were setting up the session

 

Tools

I like to use Katalon or Jing, but to be honest this is just to record and take screenshots of the test sessions. To do these kind of tests you just need a paper and a pen to write your notes, concerns and questions.

 

Resources:

http://www.satisfice.com/sbtm/

http://www.satisfice.com/presentations/htmaht.pdf

https://www.gov.uk/service-manual/technology/exploratory-testing

What is the cost of a bug?

If you have wondered how much a bug cost and how to measure this, today I am going to show some research about this. You must be asking, but why invest in testing if you can just fix your mistake after? What’s the true cost of a software bug? The cost depends on when the bug, or defect, is found during the SDLC (Software Development LifeCycle.)

In 2016, the software failures cost the worldwide economy $1.1 trillion. These failures were found at 363 companies, affected 4.4 billion customers, and caused more than 315 years of time lost. Most of these incidents were avoidable, but the software was simply pushed to production without proper tests.

 

 

It’s time to pay attention to how much software errors cost to your company, and start taking steps to recoup those costs.

To illustrate: if a bug is found in the requirements-gathering phase, the cost could be $100. If the product owner doesn’t find that bug until the QA testing phase, then the cost could be $1500. If it’s not found until production, the cost could be $10,000. And if the bug is never found, it could be secretly costing the company money. A 2003 study commissioned by the Department of Commerce’s National Institute of Standards and Technology found that software bugs cost the US economy $59.5 billion annually.

The cost of a bug goes up based on how far down the SDLC (Software Development Life Cycle) the bug is found

Then there’s the domino effect to think about. The software development approach often need to change to accommodate the code fix, which can in turn bump back other code changes. So not only is the bug going to cost more to fix as it moves through a second round of SDLC, but a different code change could be delayed, which adds cost to that code change as well.

 

Test early, test often. Prevention is better than the cure

To ensure that bugs are fixed at an earlier stage, take advantage of the following security testing practices:

  • Get together the team to help identifying issues during the design phase of software development.
  • Implement code review stage
  • Create an automated regression or smoke tests pack and run them often

 

In development, you often have less data, use one browser and use the software exactly as intended. Plus you probably already have a debugger on the machine. The major problem with bugs in production is in the absence of supporting tools.

 

Better testing

The truth is when automated testing is still under-development you still need to do manual testing. A poor testing methodology costs a lot of money in wasted time and can result in a reducing the scale of return.

For instance, if your testing process is too thorough, your developers will be producing new features faster than you can test them, and you end up with a backlog of features waiting to be deployed to production. The same could happen if you don’t have the correct scale between how many people you have in your QA and in your DEV team.

Always add unit tests when software errors are found. It is a bit more painful when handling legacy codes without a good coverage of unit tests.

 

Prioritise your bug fixes

Prioritise software errors measuring the impact that they have on the end users. This way will allow you to allocate your resources accordingly, saving valuable costs throughout the SDLC and reduce the cost of software errors.

Software errors expose your end users to slow, buggy software, compromise the security and safety of your products. Many businesses don’t have visibility on their software errors, so measuring them and their impact can be hard.

Software errors were responsible for a majority of the software failures in Government, entertainment, finance, retail, services and transportation in 2016 thanks to research conducted by tricentis.com:

 

What’s worse is software errors have multiple consequences varying on impact, so you can’t always pinpoint the cause and effect. The effects trickle down ultimately to:

  • Developer costs spent on finding and fixing errors
  • Ongoing lost revenue from unsatisfied customers

Using a few industry averages, we can help you calculate the cost in lost development time for your company.

 

How to calculate the cost of developer labour caused by software errors

Taking the industry averages from 2018, we can estimate the financial costs to your company and investigate where the money is going.

  • The median developer wage for the UK April. 2018 £30,651

The errors in your applications need to be fixed or they will affect end users and cause lost revenue. This is where support costs start to mount. Fixing software errors is low cost, reactive work.

You should be aiming for around 20% reactive work (finding and fixing errors, support costs), 80% proactive work (building features and improving product) rather than vice versa. This is where you’ll add the true value to the business and your users according to Raygun.

Based on a 40 hour work week at the average wage of GBP £30,651, the average software developer could spend 8 hours each week (32 hours each month). Costing around £6,130.20 per year, fixing errors and replicating issues.

This is time spent away not building new and important features for your customers. So, think twice when building your QA team and not bringing key people to earlier stages of a new implementation. Share as early as possible, gather opinions and different perspectives of the platform.

 

Resources:

https://www.payscale.com/

http://blog.celerity.com/the-true-cost-of-a-software-bug

https://crossbrowsertesting.com/blog/development/software-bug-cost/

https://raygun.com/blog/cost-of-software-errors/

https://www.synopsys.com/blogs/software-security/cost-to-fix-bugs-during-each-sdlc-phase/

Leading by example

Hello guys,

Today I am going to post something interesting that I learned through time with my work experience. This time I’m not talking about technical skills, I’m talking about the soft skills and how you can be a better person every day by learning from the constructive critics and ditching the destructive ones that don’t give you any benefit.

I am also sharing what some successful companies are already doing in the Silicon Valley which Jacqueline Yumi Asano explained in her article (it is in Portuguese, but you can check it bellow in the Resources part).

 

Inspire people

 

Say sorry, you are not a machine and you are going to fail at some point

If you want to be a trustworthy person at your workplace, you need to show that you are fair and you recognise when you fail. Saying sorry will show that you are humble enough to have the trust of anyone.

 

Strength the individual

Nothing is more empowering than supporting people to learn something and improve themselves. Everybody wins when this happens. Ask yourself if 20% of your time is about learning or you just do the same task over and over again. Do you have challenges in your company ? Are you growing in this company ?

You might be asking how can you identify if a company will improve you or not before even starting there? Check how they write their specs and roles. Are they asking for specific technologies or are they looking for generalist professionals ? A good company will look for someone that is a machine learning person and not someone that has a fixed technology in their skillset. That means the company will follow the latest technologies and you will be learning most of the time.

We don’t’ assign the work for our developers. They assign up for work
LiveRamp

 

Don’t under estimate the power of the vision and direction

Do you have a clear version of your goal ? Does your company is taking you to this goal ? This is extremely important for your future and you don’t have time to lose. So, double check if your company helps you to achieve this goal and if you have a clear vision and direction to follow.

If not, there is not too much you can do apart from start looking for a place where you can clearly see it leading you to your goals in, let’s say, a years time.

 

Get away from tyrant bosses, look for a strong bond of trust on both sides

If you ever worked in a place where your manager use his lead position to decide something rather than arguments, what are you even doing in a place like this? It is definitely a toxic environment and with this lack of trust it is better to look for something new because discussions and arguments would be pointless. Your opinion will mean nothing, even if you have arguments, analysis and your own carrier experience. Your manager will always ignore what you are saying.

Free choices matter and you should work in a place where your opinion is not invalidated.


Our product is great because we show why we are recommending
an specific insight. This way we build trust.
François Lopitaux, Director of Product, Salesforce

 

Surround yourself with passionate, but not blind people

Passionate is different from blindness. I just want to highlight that I have found many people that come across as passionate people but in reality they were completely blind and never accepted to have failures on their ideas/projects/implementation. This is quite common among developers, for that reason there is this assumption that a QA will always be a developer enemy, since it is part of our job to show bugs on their implementations.

A smart person knows how beneficial is to have a constructive critic and feel glad to have them. So, surround yourself with passionate and smart people. Be smart, take responsibility, accept the failures and improve them.

 

Get others opinions to empower people

I can only see advantages on doing that. Share knowledge as soon as possible with everybody in your team. Is this a new feature ? So, get everybody together and expose your idea, get others opinions. Don’t ignore them, every idea has a value. You can find bugs in the early stages and also this builds an ownership feeling.

Does the company take your opinion into account ? Do you feel that you have autonomy ? There is nothing more empowering than showing that you care about everybody opinions. Everybody feels valued and this will increase the trust on your work since they know that they can be honest with you.

We don’t need as PMs to tell UX Designer what they need to look at
Pinterest

 

Resources:

https://medium.com/mulheres-de-produto/o-que-eu-aprendi-no-vale-do-sil%C3%ADcio-sobre-product-innovation-7f3128f33e3

http://qablog.practitest.com/leading-by-example/

http://www.soulcraft.co/essays/lead_by_example.html

How to use mind maps to clarity your tests

To improve the communication about a project you don’t need to have infinite docs and articles. For someone who is starting or to quick understand the product you need to have something smaller, prettier, and more focused to the audience. Mind maps are a lightweight form of testing documentation, because communicating effectively with the team is the key of a good quality implementation.

Revealing the Complexity of a Problem

Imagine that you have to test an app. You know that you need to test the functionalities and if the behaviour of the app is not clunky and unstable. You can have articles on Confluence explaining the behaviour of the app or you can have a mind map which is more focused and simple.

For example:

Click to expand

 

This mind map will help you to remember of all of the type of tests that you can perform on a mobile app.

The mind map communicated the logic of how our code would be written without the need of looking at code. It can cover all of your use cases and extract connections in a way that would have been difficult to do in a list.

When creating the mind map you can follow Heuristic Test Design, which is a model of tests with different patterns of quality criteria, techniques, elements and environment. It helps testers to remember and design different combinations while creating the test plan.

 

Using mind maps for regression tests

You can use mind maps for so many things, for example as a guide to your regression tests. I think it is far much more easy than reading a list of checklists. Also, it helps people who are just arriving at the company to understand the flow and the connections across the features. This guide helps you to decide whether what’s happening is something you should expect. Not everybody agrees about having mind maps for regression tests which I can understand why, but you can decide this with your team.

Imagine that you have a checklist like this:

You need to follow this checklist to be sure your release is good to go, but imagine that you have a map to follow, wouldn’t be more clear and easy to understand ? You can find the mind map correspondent to this checklist here:

Click to expand

When a button changes, for example, the mind map should be the first thing to look at. You can check if nothing was changed below or up that node (feature). You look at the parent node to see what pressing the button did and make sure it still does that. You update the mind map with the new button shape so that future testers know how it works now.

Mind maps help us test not just the change at hand but the consistency of that change relative to the rest of the product, the product’s history, and the feature’s purpose.

 You should share this process and ask for the development team input their thoughts and this will build trust in the regression pack. Also as I always recommend, share and review always. You are not working alone and it’s important to remember that we are not machines and we have blind spots which can be solved by the involvement of a properly engaged team.
Tools
You can use some of these free tools to create your Mind Map, I usually prefer the online ones, but feel free to choose the best one for you:

 

Resources:

https://dojo.ministryoftesting.com/lessons/mind-maps-made-easy

http://www.satisfice.com/tools/htsm.pdf

Blockchain Testing Tools

If you are wondering what Blockchain is, I will give you a quick introduction. Blockchain is a data structure that is distributed at once in many different places and as you can’t ever delete from it, it is extremely difficult to make amendments. This makes the record more secure and more trustworthy.

So, what are the kinds of test that you can perform ? You can use the traditional testing, since it is just normal development work with normal testing criteria. So, boundary value analysis, decision tables, test driven development and behavior driven development techniques.

There is also a set of questions that can help you to build your test scenarios, for example:

  • How does it handles valid and invalid inputs?
  • How does it cope with a wide range of input data?
  • How does it handle missing state, or existing state?
  • How does it handle error cases?
  • How does it handle security and access control?

You don’t need to test the Blockchain because the algorithms are well-established, because it is a distributed system, but the transactions still require some kind of validation. For example, you may need to check if your transaction is valid before it can be approved. There are approval authorities for different blockchains, and they must test the integrity of the transactions.

 

What is Smart Contract ?

Smart Contract is an API and defines the rules for transactions in a Blockchain network. A Smart Contract is a set of rules in the form of programmable constructs that are capable of automatically enforcing themselves when pre-defined conditions are met.

It has public functions which might be called by anyone registered on the Blockchain network. However, unlike a traditional API, a Smart Contract cannot call external web APIs.

 

What do you need to know to test Ethereum Smart Contracts ?

Test automation requires that the platform being tested must have hooks so that external automated scripts can instruct the platform, observe the outcome, and verify that the outcome is what is expected. Legacy platforms in banking often do not have these hooks, and that makes automation much more difficult. When you compare smart contracts to older software used in banks, you can automate testing much earlier and much faster.

 

I will show some of the tools that you can use to perform tests on Blockchain applications:

 

Ethereum Tester

This github has a project for you to test Ethereum Blockchain Applications. You will need to clone Eth-Tester. The Eth-tester library strictly enforces some input formats and types.

 

Truffle

Truffle is one of the most popular Ethereum development frameworks and has testing functionality, it is a scaffolding framework for smart contracts used by UI builders. You have the ability to write automated tests for your contracts in both JavaScript and Solidity and get your contracts developed quickly.

Ganache

Ganache is the most-used library for testing Ethereum contracts locally. It mocks a blockchain that gives you access to accounts you can run tests, execute commands, etc.

 

Populus

By default tests run against an in-memory ethereum blockchain and as you can see here Populus supports writing contracts that are specifically for testing.

 

Manticore

Manticore is a symbolic execution tool for analysis of binaries and smart contracts. It is supported on Linux and requires Python 2.7. Ubuntu 16.04 is strongly recommended. It has a Python programming interface which can be used to implement custom analyses. You can see more about here on the wiki.

 

Hyperledger Composer

Hyperledger Composer supports three types of testing: interactive testing, automated unit testing and automated system testing. It’s a framework for rapidly building Blockchain business networks on top of a Blockchain platform, such as Hyperledger Fabric.

This framework allows you to automatically deploy the business network definition, and then interact with it by submitting queries and transactions in order to test how that business network really behaves.

 

Corda Testing Tools

Corda is a blockchain-inspired, open-source distributed ledger platform. There are several distinct test suites each with a different purpose: Unit tests, Integration tests, Smoke tests, Load tests and other. These tests are mostly written with JUnit and can run via Gradle.

 

BitcoinJ

This tool BitcoinJ allows you to interact with Bitcoin connecting directly to the bitcoin network. So, you can simulate send and receive transactions in real time, also you don’t need to have a local copy of the Bitcoin Core.

You can follow this guide to get start with this tool.

 

Resources:

https://www.joecolantonio.com/2018/02/01/blockchain-testing-tools/

https://joecolantonio.com/testtalks/175-blockchain-application-testing-rhian-lewis/

http://searchsoftwarequality.techtarget.com/answer/Heres-everything-you-need-to-know-about-testing-blockchain

https://www.capgemini.com/2017/01/testing-of-smart-contracts-in-the-blockchain-world/

http://www.bcs.org/content/conWebDoc/56020

https://medium.com/@mrsimonstone/test-your-blockchain-business-network-using-hyperledger-composer-c8e8f112da08

Quality Engineer Mindsets

Hello guys, today I am going to post about the different types of QA mindsets that exists and how you can identify them. As a QA Engineer your aim is to improve the quality of the product from the very beginning. If this is the main objective why we have so many different titles for a professional that works in the QA area ?

This is because you can find QAs that focus more on the business and product side of the project. Others are more technical and focus on the automation and improving the quality and the release speed of the product, and some others that like to do both, the business and the automation part.

What is the future of the QA professionals ?

We can notice that some QA pros are instead of running the tests and handling the hands-on work, they are transitioning into a more consultative and distributed role to help developers learn how to write better tests and improve their approach to screening for quality. Because, at the end of the day, developers are still developers and while they’ve not been trained to look for quality issues, they’re going to miss things.

QAs pros just need to adapt to a rewritten organizational hierarchy—and a completely different mindset about testing objectives.

You can spread this mindset to your entire team, not only QAs. Uncaught bugs and untested features could easily gradually destroy confidence in the new product and lead to a loss in traction. Knowing your development team can not only ship features quickly but avoid these costly mistakes may be the edge your app needs.

With the latest trends toward faster and more efficient software teams having developers doing their own quality assurance is more important than ever. It takes dedication and practice to build a QA mindset but the benefits of how it can help your team and product make it worth.

I am going to separate the QA mindset profiles following the market trends.

Traditional QE

  • Perform Manual tests only
  • Focus more on the business part of the project
  • Able to perform Functional tests
  • Create a lot of documentation, like Test Plan and Test Suite
  • No contact with the feature until is Ready to Test
  • Don’t have integration with development team

 

Agile QE

  • Perform Automation and Manual tests
  • Focus on the business and the technical part of the project
  • Able to perform Functional tests
  • Create Test Scenarios for automation and for the feature
  • Help the Developers and PO to write the scenarios upfront
  • Have some kind of integration with development team

 

Pos Agile QE

  • Perform Automation tests only
  • Responsible for configuring CI/CD
  • Focus on the business and the technical part of the project
  • Create Test Scenarios for automation and for the feature
  • Able to perform Functional and Non-Functional tests
  • Help the Developers and PO to write the scenarios upfront
  • Share the automation knowledge with the developers

 

Developer QE

  • Developer with interested in QA
  • Implement the code and the automation code
  • Focus on the implementation, but also on the quality
  • Tend to over complicate the Automation project
  • Has a QA curious and critical thinking when implementing
  • Able to perform Functional and Non-Functional tests
  • Test his own implementation thinking on the end user

 

There is also a misconception about QE:

Wannabe QE

  • Doesn’t have a QA background
  • Worked only on one type of industry or only on crowd tests
  • Usually doesn’t know anything about type of tests (Functional or Non-functional)
  • It only knows about the business related to the background they have
  • Doesn’t know the importance of trivial tests like boundary tests
  • Doesn’t really have interest about QA best practices

What I suggest for this last one is to study. If you want to work in this area, don’t undervalue the professionals that have studied and continue studying everyday to be here.

In general QEs are extremely curious and go always for the simple and cleanest solution. They are skeptical and nitpicker which are the most annoying character of a normal human hates to posses. But when it comes to a testing profession, these becomes a must have characteristics for a tester.

 

Resources:

https://haughtcodeworks.com/blog/software-development/qa-mindset/

http://randsinrepose.com/archives/the-qa-mindset/

https://techbeacon.com/qa-devops-reinventing-testers-role-face-automation

http://blog.qmetry.com/quality-engineering-mindset-devops/

https://dzone.com/articles/dev-vs-qa-should-there-be

https://qa.siliconindia.com/news/How-an-Ideal-Software-Testing-Mindset-should-be-nid-105831.html

https://www.slideshare.net/juliodelimas/mindset-do-qa-em-diferentes-contextos?trk=v-feed