As any other model, this doesn’t say this should fit to every company, but this is the typical idea and what Google follows. You can follow this, but you need to be aware that maybe this is not the best model for your company.
This pyramid shows how much of each type of tests you should focus more. So, in numbers this should be:
- Unit – 70%
- Component Regression – 20%
- Acceptance Criteria – 10%
The top of the pyramid is about test the logs, this is not really a common test, but it is about making sure the behaviour is correct, testing the logs according to the action, the base of your system. For example, Google saves all the data from the users in the logs and keep all these information making sure the data is protected. From bottom to top each stage of test requires high level of domain knowledge, high amount of setup costs, investment, time, machine resources.
The canary tests approach is about pushing the release to a first batch of end users, who are unaware that they are receiving this new code. Because the canary is only distributed to a small number of users, if your company is global remember to have this small group of end users divided per each location, its impact is relatively small and changes can be reversed quickly should the new code prove to be buggy.
You need to consider all these aspects to build robust automation frameworks that have a utility and usability through the time. You need to consider the velocity of your tests and of course the quality that you can achieve with small and frequently releases, keep in mind they need to work together.
A bad model of test strategy
- Never test in production environment;
- Release as much as you can, possibly every time a new feature is aproved;
- Focus first on unit/smoke tests not the whole system;
- Create good metrics to show the evolution of the product quality
Google has grown on maturated taking into account different devices, platforms, features.
A good model of test strategy
- Focus on quality of your product and the infrastructure;
- Stable automation frameworks/tools;
- Cross functional tools, don’t rely on only one tool for tests.
Take care of feature duplication. When a company grows, the number of tests with different goals grow together, but duplicating the same code. This leads to dead code and codes doing the same thing.
Metrics
You can create metrics about defect leakage of your automation, how long is taking to run the tests, everything will help you to know where you can improve. So, try as much as possible to have clear objectives.
This is the test and release model that Google has at the moment, with some canary testing, monitoring step, frequent releases and feedbacks.
Most of the challenges that were said are the challenges of always, as a company you need to think constantly in growing and hire more people in different teams to work together, also every year we have hundreds of new devices to test and make sure your system is supported.
- Integration tests between even more components;
- Make multiple teams work together;
- Support multiple platforms.
The complete video is here: https://www.youtube.com/watch?v=2yN53k9jz3U
Rafaela Azevedo 