Monitoring & Alert Test Strategy

Having an alert system is crucial to have confidence in your product. There are 2 main points you need to consider, Monitoring and Alerts:

  • Monitoring consists of dashboards and reports to display the metrics.
  • Alerts involve taking some type of action such as notifying someone, writing to a log, and raising an alert on a dashboard.

Having an alert system is crucial to delivering the right information to the right people at the right time

Important Metrics to Watch

The list is long and you can get lost on all the possible metrics you can watch and raise an alert upon. A good strategy is starting with the riskiest ones, the ones that can stop the system to work, stability, and performance. Here is a list of some of the most-watched metrics:

CPU utilization 
Memory utilization
Memory breakup
Load balancer (Number of instances running)
Services and processes running
Processor queue length  
Disk usage
Network up 
Crontab executed
Event logs generated
Application details
Expiring Certificates
Docker/Kubernetes containers

These metrics can be swapped or removed or you can even add others according to your needs. The goal here is to find the best metrics to watch and raise different levels of alerts to the specific group of people at the right time. These metrics can help the team to check the system’s health and also increase the confidence and reliability not only in the recovery steps when something goes sideways but also in the alert system itself.

Different Types of Metrics

I found this good framework to create a good alert. It needs to have these properties:

Actionable: Indicates a problem for which the recipient is well placed to take immediate action.
Investigable: Indicates a problem whose solution is not yet known by the organization.

Do you need to take an action every time? No, you can specify a capacity limit, and then once the metric reaches this limit you can have the first support acting up on it, then when it reaches the second limit then you have a different action like calling the second line support, or calling a script to restart a service, etc.

Sometimes there is no action to take. For example, “CPU utilization” or “Packet loss.” These are some classic FYI alerts. Instead of alerting, these things should appear on a dashboard for use in troubleshooting when a problem is already known to exist.

Do you need to send the alert straight away? Also no, especially when you have mechanisms of self-healing. There’s no need to bother a human to fix the issue; the response should be automated.

Depending on your recovery strategy, you can wait 5, 10, or 15 minutes to check if the metric is still showing a problem, and then, if it is you can raise the alert accordingly. In the alert, you can always have a link to the documentation with recovery steps (e.g. a “runbook” indicating steps to perform).

Remember to have an FYI for when the problem is fixed as well, so in case something has failed in the middle of the evening and in the morning is working, you get the latest update about the problem.

Always keep records and logs to investigate what happened and how the system reacted.

Dan Slimmon has a good spreadsheet example of an alert framework:

Monitoring and Alert Tests

Independently the tool you are using for your metrics and alerts: Grafana, Cloudwatch, Sensu, Prometheus, or New Relic. It is important to have a test strategy to cover this critical part of your system.

Some common tests you need to perform are much more Devops related, like scaling down some instances on AWS, simulating a high memory load with scripts, or even creating scripts specific to remove log files from the Docker container.

Points to keep in mind:

Is it worth automating?

Think about if you have a lot of changes and maintenance to the project. How much effort would be to create the automation for these scenarios? How complex these scenarios would be? Does automating them bring any real value?

There are some tools that you can use to perform chaos engineering:

AWS FIS (Fault injection Actions Simulator) to automate some of the scenarios and add to your continuous delivery pipeline. It’s a fully managed service for running fault injection experiments on AWS that makes it easier to improve an application’s performance, observability, and resiliency.

Chaos Mesh utilizes chaos experiments within Kubernetes environments. It’s able to use various types of scenarios related to fault simulations within a distributed system.

Many other tools can be used and you can check the list in this post.

At what stages do you want to test and what scope?

Check if you need to do manual smoke tests for each feature and then have a bigger regression after merging the tickets, or manual test for the feature and at the same time add the automation for this scenario, so at the end of the pipeline, you have the full regression.

Maybe you don’t even need to have automation running for this project and just doing manual feature tests for the tickets and having a report with the full regression scenarios is enough

Whatever is the strategy that you choose, make sure you have the max confidence you can have with the current constraints and a plan to improve in the near future.

Automating these tests is complex and always involves a good knowledge of infrastructure. I personally had and still have to do a lot of research when it comes to it.


Testing Trends 2021

Hello all, I have been wondering what will be the trends for this year. Last year, I noticed an increase in accessibility and performance tests concerns because of the increased number of users going online due to the pandemic.

I believe this is going to continue this year and then we will definitely have an increase on the other trends that were already around for the past years. I have condensed some of these trends here:


Think about how the world changed after the internet, everybody is connected. Billions and billions of transactions are made per second, because of that, the concern with the security is even more important. Blockchain arrived with the idea of making it difficult or impossible to change, hack, or cheat the system. Blockchain is essentially a digital ledger of transactions and smart contract (chaincode) services to applications that is distributed across an entire network of computer systems.

This has been a hot topic in the past years, so nothing really new, but you will definitely see an increase as people are still learning and understanding what are the benefits of it and what are the problems that Blockchain solves. Once you understand you never go back.

Augmented Reality

Again nothing really new here, you can see videogames have been using this a lot and it was definitely a boom when you could catch Pokemons in the middle of the street !

With Augmented reality (AR) you can have a real-world interactive experience. The objects from the real world are enhanced by the computer-generated perceptual information, or even combined with digital created objects. You can see an increase on this trend as again people are getting more and more online and avoiding going to the stores because of the pandemic. Now you have the ability to try clothes online for example.

I have never really joined a project with Augmented Reality, but this would be another challenge since you need to keep in mind things like speed you move, lights, objects and other variants.

Chatbots and Artificial Intelligence

I know many people hate chatbots but, for someone that hates calls or just wants to make a quick question, or even just want things to be solved as fast as possible like me, this is the right option. Of course there is still a lot to improve on the machine learning side of the technology, but depending on the how you were able to train the bot, you can even think you are talking to a real person.

I am not going to deny that there is a lot of training of words and phrases until it feels like talking to a human but, some chatbots are really useful, for example if you need to return a purchase on Wish app. Today, chatbots are used most commonly in the customer service space, assuming roles traditionally performed by living, breathing human beings such as Tier-1 support operatives and customer satisfaction reps.

Artificial Intelligence (AI) is here to stay since the beginning of its creation. Big companies like Tesla, Amazon (Alexa), Apple (Siri) are more and more sophisticated. You can expect Machine Learning to be used in the automated tests more and more and also to think about the tests you will need to perform on AI projects.

Mobile Apps (Fitness, Mental Health, E-commerce, Services in general)

It is rare to find a person that doesn’t have a smartphone nowadays, even kids need to have so the parents can contact them to see if everything is okay. The pandemic brought more people to look for mental health and fitness apps than ever, is this going to stay even after our current situation ?

Mobile tests are way more full of details, things you need to consider, like the speed of the internet connection, orientation of the screen, sending the app to the background and opening again… The market is flooded with millions of apps and we have too many options now, so how will new apps find their place? Quality of the service, Usability and Price.

Usability and Accessibility Testing

This is another hot topic as the pandemic pushed more people to go for online services. Keeping accessibility tests in mind you will help people with disabilities to be able to navigate and interact with websites and tools. It also means that they can contribute equally without barriers.

Usability Tests is more about the design of the products. People don’t want to tap 300 times to be able to login or pay for a product, your app needs to be efficient, effective and satisfying. A lot of people don’t realise how this is important, but once you lose a client because of a small mistake it might be hard to get it back.

Internet Of Things (IoT)

Share data and automate services are a must have ! Nowadays you need to have the option to share your fitbit steps with your scale, for example. There is a huge demand to access, create, use and share data from any device.

The thrust is to provide greater insight and control, over various interconnected IOT devices. Which means you need to have the option to tell Alexa to turn on or off the lights when you are too tired to get out of the bed to do it.

I particularly didn’t see too much movement in this area last year, maybe because of all the other more urgent things we had to deal with, but this is another thing that is going to continue increasing, even though I can see a lot of security concerns around having everything connected. Imagine if somebody is able to hack your printer and then getting access to your front door lock ?

Data Protection and Security Tests

This is something you need to keep in mind every time you have a new feature coming up, or even updating a current feature. We need to identify the threats in the system and measure its potential vulnerabilities, so the threats can be encountered and the system does not stop functioning or can not be exploited.

Security tests were always important, but I always felt it was a bit neglected until something bad happens, then was treated as first priority. Here in Europe we need to be super careful with the GDPR rules and make sure we are not exposing personal data anywhere. So, another thing to keep in mind as if we have more people accessing the internet, it means we have more people trying to steal data as well.

Coding Skills and QAOps

Another name for the QA role, like we don’t have enough 😂 (QA, Tester, SDET, Software Developer in Test, Test Automation Engineer, Full Stack QA and the list goes on…). What is this new QAOps name ?

It is a combination of Quality Assurance (QA) and software operations. QAOps combines QA practices with software development and IT operations to develop a long-term, integrated operational delivery model.


What is Quality and Who Cares About It?

Hey guys, this is a long due meetup post that I haven’t shared here. It was a great discussion, sharing ideas and meeting new people with different experiences. I highly recommend everyone to join a meetup like this one day or another as it is really dynamic and enjoyable.

The link to the meetup is here, but as you can see it is from the middle of the last year. 

What is Quality and Who Cares About It? Roundtable Event (No Spaces Available)

Thursday, Jul 23, 2020, 5:30 PM

Online event

1 Advocates Went

We’ll be sharing a recording of a roundtable discussion, as we have set up a small group to discuss the meaning behind quality within organisations.

Check out this Meetup →

Testing Blockchain Applications

Hello all, today I am going to post about a topic that I am a huge enthusiast about: Blockchain !

Nowadays data is one of the most valuable product that you can own, and as we are surrounded by fake data coming from thousands of different places, it is hard to know what is trustworthy or not.

What is Blockchain ?

It is basically the second phase of the internet. The Internet democratized the exchange of information, blockchain promises to democratize the exchange of real value. It was created in 2008 by Satashi Nakamoto, which the identity is still unknown.

An example of how blockchain can be used is when you want to make a transaction of £10 from a X to Y. Nowadays what happens is this transaction goes through a third-party app or payment processing system. Then, X’s bank will need to check the details of Y’s bank and once this is done the transfer of the amount will initiate with a certain amount of deductions, of course. In the end both banks will record the transaction deducting the transaction fee, meaning Y will receive something like £9.95. Even though this process is quite secure and has many redundancies in place to make sure that it stays secure and accurate, there are some basic fundamental issues with this process:

  • Delay in processing (using this middle agent and all the bureaucracy)
  • Dependency on a single intermediary whose effectiveness is never 100 percent (dependency on a national state)
  • In case of any gap in the transaction, no one takes the responsibility and people keep on blaming each other (not concrete proof to check the transaction)

Blockchain is a system where you no longer have to worry about these problems, you just need to perform a transaction and there is a chain of people who are able to validate your transactions every second. This mechanism is called as Proof of block and this is done based on the public key provided for the encrypted data, and it is done by all listeners in the peer-to-peer network. They are real people/identities, not bots. Since there is not a single transaction validating authority (banks, national state) who are centralising the transaction service, the process is effectively decentralized, public and transparent for everybody to see.

Once a specified number of people validates the transaction, the transaction details are stored in the form of a block and that block is added to the existing blockchain. Once the blocks are validated, nobody can change them, just add another one on top, they are immutable. These blocks have a specific hash associated with every block. These hashes are like fingerprints and are unique to every block. Every person that is validating the transaction process is called a miner. The more miners, the better efficiency of the transaction.

A block contains data, a hash, and the hash of the previous block. Since it contains a hash of a previous block, all the blocks contain data for the previous blocks and it becomes almost impossible for a blockchain to be corrupt.

What types of tests can I perform in a Blockchain app ?

  • API Testing: In API testing, we ensure that the interaction between applications in the blockchain ecosystem is as expected
  • Block/Peer/Node Testing: All the blocks on the Network should be tested individually to ensure proper cooperation. All diverse nodes on the Network must be tested independently to ensure smooth cooperation. Do the nodes in the network sync with other validating peers? Is the integrity of the network and shared ledger maintained throughout the testing?
  • Functional Testing: In Functional Testing, we evaluate the work of various functional parts of the Blockchain (e.g., smart contracts).
  • Performance Testing: Details like network latency based on block size, network size, expected transaction size, and how long a query takes to return the output with the specialized authentication protocol
  • Security Testing: Here, we ensure that the application is not vulnerable to attacks and the systems can protect the data and are capable of handling malicious attacks, etc.
  • Integration Testing: In Integration testing, we ensure that all the components of the application are integrated properly and performing the actions appropriately
  • Smart Contract Testing: Smart Contract testing is about performing detailed functional testing of business logic and process. It refers to the set of software constructs that automatically execute transactions when predefined conditions and business logic are met. Testing smart contracts involve simulation of all possible expected and unexpected variables for every contract and the triggers that execute transactions.

Benefits of Blockchain Apps

  • Decentralized System: No need to have a middle agent for the transactions. Beneficial in various industries like finance, real estate etc.
  • Better Security: As it uses multiple nodes to complete and authenticate transactions, you don’t rely only on a transaction system that is controlled by a centralized group of identities.
  • Authenticity: Allows the unique algorithm to process data, like a fingerprint.
  • Increased Capacity: Increases the capacity of the entire Network as it is completely decentralized and real people can verify the blockchain.

Challenges in Blockchain Testing

  • Understanding the Technology: Blockchain is a new technology and understanding it is very important in order to test it. Where does regulatory compliance overlap with quality? How do you fix a defect that has been deployed as part of an immutable smart contract? How do you predict transaction fees and determine the correct behaviour of your application if transaction fees and network volume increase unexpectedly?
  • Lack of Blockchain Testing Tools: Blockchain-based applications testing is all about tools. Selecting the right tool as per application is one of the important decisions.
  • Defining Test Strategy: Designing Test Strategy for Blockchain application is quite complex exactly because you need to have a good understanding and in-depth knowledge of the technology.
  • Block and Chain Size: Testing for block size and chain size is quite important as the application may fail without proper validation of block size and chain size and creating a false-positive verification.
  • Integration Testing: Should be done properly and frequently to test that all the components are properly integrated.
  • Performance and Load: You need to perform load testing to give better insight into how the Blockchain application performs in the real world, check if it is scalable and what are the limits. Check the example that was done to Ethereum and the Cryptokitties.
  • Security: Securing the data should be the most important thing in a Blockchain Application, to be honest, in any application. Even bitcoin is not bug free !

What do you need to consider when testing a Blockchain Application?

Here it goes an example with Bitcoin transaction tests:

  1. Block Size: The maximum fixed limit of a block is 1 megabyte. After the introduction of Bitcoin, the average size of a block for the first 18 months came out to be under 30 KB. But in December 2017, it hovered around 1 MB. What if the size of a block exceeds beyond 1 MB?
  2. Chain Size: There is no limit on the size of the chain. So, it is fun to test it for its function and performance. For example, the Bitcoin chain’s size keeps on increasing day by day.
  3. Load: With so many people on the blockchain, the load becomes a major parameter to test in a blockchain. Bitcoin currently has a maximum throughput of 3.3 – 7 transactions per second, but what if the transaction/second increases as in the case of Visa(2000), Paypal(193), etc? The load remains a major problem with blockchain because performance drops when the load increases.
  4. Security: Since there are many miners involved with a transaction, ensuring security is a little complex. Well, there is a multi-layered security system in a blockchain. If one of the layers has been hacked, the instantaneous transactions cannot be stopped. It is, therefore, to be tested that one security layer doesn’t affect the other.
  5. Transmission of data: Encrypted and decrypted data is transferred from computer to computer, so it is necessary to test if the transmission process is working flawlessly. Is the data being sent received on the other end, or is there a loss in between?
  6. Addition of block: Every new block is added to the chain once the transaction’s validity is authenticated. So, it must be tested that there should not be any leak in the block addition system and the block must be added after authentication.
  7. Cryptographical data: Cryptography is the backbone of blockchain technology. Therefore, it is necessary to make sure that the data is properly encrypted and decrypted.

Blockchain Testing Tools

  1. Ethereum Tester: It is an open-source testing library available as a Github repo. Its setup is pretty easy with a manageable API support for various Testing requirements.
  2. BitcoinJ: It is a Java-based framework built for Bitcoin-based apps that enables you to interact with the real BTC network and various testing activities. In order to use it, you don’t have to download the standard BTC Core files from You can even approach a user forum in case you need clarification or are facing hiccups in the testing process. It is an open network available for assistance.
  3. Populus: This framework has the testing functionality of Ethereum embedded in the form of a set of features for test contract deployment. It’s developed around the py.test framework. Hence, it is relatively easy to implement.
  4. Truffle: It’s a commonly referred name for Ethereum developers, which brings in good testing features, such as automated contract testing. The framework holds capabilities beyond just testing functionality within the Blockchain application.
  5. Embark: It is a testing framework that focuses on developing decentralized applications (dApps) that run on various systems or nodes. It has integrations with Ethereum blockchain, IPFS, and a decentralized communication platforms such as Whisper and Orbit.

Future of Blockchain

Even though Blockchain is often discussed in financial services and cryptocurrencies, this technology offers a broader range of potential applications. The use case of Blockchain technology is picking up pace in various industries and segments. A World Economic Forum report predicts that by 2025, 10% of Global Gross Domestic Product (GDP) is stored on Blockchains or Blockchain-based technology.

Free courses


PACT CI/CD Workshop

Hello all, after a small burnout and a month break from my extra work activities (2nd lockdown here in London and winter vibes definitely didn’t help), I am finally back on track and I thought it was worth sharing this CI/CD Pact workshop that was published in the middle of this year.

They use Node, but you don’t need to be an expert to try this workshop. Once you have finished, you will be able to fit Pact and Pactflow into your CI/CD pipelines and understand the workflow when changing the consumer and provider.

I followed all the steps and after around 1h20m I finished it. It is pretty straight forward and it includes the new feature pending pact that is equivalent to the pending tests tag, just make sure you are copying the right tokens and using the right URLs.

The link to the workshop with the instructions and requirements are here:

In the end I got the results:

Travis Passed

✅ Provider Passed Locally

✅ Consumer Passed Locally

✅ Pactflow verified

Thanks to Matt Fellows and Beth Skurrie for that !

Special thanks to my friend Marie who suggested me to use Excalidraw for my awesome diagrams !

Preparing yourself for the CTFL Exam

A friend of mine has recently passed the CTFL Exam after one week of a lot of study and effort. CTFL is a popular Certified Tester Foundation Level exam in software testing. It examines your professional knowledge around software testing discipline. The exam has 40 questions and takes 60 minutes.

Should I take the exam ?

This is a very debatable subject and people can discuss for hours about it’s value.

If you are thinking about if this is for you or not, these are some things that you can keep in mind and help you to decide:

  • Experience always has more value than any certification. Certifications don’t provide the exposure and training you get while working on real life projects !
  • THIS CERTIFICATION CAN HELP YOU TO GET YOUR FIRST QA JOB. If you are changing careers, never worked with Software Testing before, then this certification might help you to be selected for an interview. This would be part of your portfolio and as you might not have any experience in this field, this can be used as a parameter when filtering the candidates.
  • THE CERTIFICATION DOESN’T MEAN YOU ARE AN EXPERT OR THAT YOU ARE GETTING THE JOB. Most of the professionals applied to this certification in the beginning of their career and then never study the Syllabus again. This is because you start to use your experience more than the base knowledge you got when studying for the exam. Again the certification doesn’t mean you will get the job, experience and exposure to different projects will.

Online Course

Mock Exams/Material and Syllabus links

When do I know I am ready ?

One technique that I use for actually most of the exams that I apply is practicing the mock exams and once I pass 3 times in a row then it means I am mostly ready to take the real one. You might have other ways to see when you are actually ready, but whatever you follow make sure to prepare and dedicate yourself !

Test Strategy Templates

Hello people,
Following up these meetups: Developing a Test Strategy and Desenvolvendo uma Estratégia de Testes, I realised it would be useful to share some templates and examples that I have seen in my previous projects.
Every company/project adapted this document and had their own template. There is no right or wrong as long as you have the needed information there to the best of your knowledge at the time is okay.


Template 1


Template 2


Template 3


Template 4


Template 5


You can mix them, pick one session from one and another session from the other, feel free to create your own Test Strategy according to what you need !

Developing a Test Strategy

Hello everybody,

In case you have missed here it is the link for the meetup about Developing a Test Strategy 30/07/2020.


If you can speak portuguese and don’t feel comfortable with english yet you can also watch the portuguese version here:

Open Banking Functional Conformance Suite Test Cases

What is Open Banking ?

Open banking allows the use of open APIs enabling third-party developers to build applications and services around financial institutions. It comes to bring more financial transparency options for account holders ranging from open data to private data.

Open Banking Use Cases (for Users) | by Ştefan Alexandru Băluţ ...

Open Banking Functional Conformance Suite

To be able to get the Functional Conformance Certificate, Open Banking provides a Functional Conformance Tool to allow implementers to check if your API has successfully developed all required functional elements of the OBIE Read/Write API Specifications.

This Open Banking tool allows an ASPSP (Account Servicing Payment Service Provider) and a TPP (Third Party Provider) to test the response of any API endpoint and validate that the JSON and data formats meet the schema, permissions and interfaces against the Functional API standard.

How to identify Test Cases covered in the OB Functional Conformance Suite ?

How do you know what else needs to be covered and if there is indeed something more to cover ? After digging into the project on bitbucket, I found some useful json files where you can check the assertions for each test case, the test cases itself and another file to translate the list of the assertions.

So, you can find the asserts that are being done for each test case inside the manifests folder.

For example, this one contains the assertions for this test case: The x-fapi-interaction-id is replayed for an Account. You can find the file with the accounts transactions test cases here.

Screenshot 2020-07-13 at 17.48.08

Then you would need to check what this assertion actually means, and you can find the dictionary of the assertions on this file.

Screenshot 2020-07-13 at 17.52.47

Remember that all the tests currently assume that consent is granted at the ASPSP portal for each requested PSU Consent (Payment Service User Consent).

Also, you will find that some test cases are missing for instance what should happen when you send an invalid token to the payments endpoint, but you can see there is a test case for the accounts endpoints for when you send a token without the required permissions to get a 401 response.

In this example, you can see that for payments the consent model is a bit different because each access token doesn’t have a range of permissions, but is associated with a single payment consent id. So, in order to get a 401 response, the request can present the wrong token along with a payment call or present no token at all. The conformance tool is not sending any token in this instance.

So make sure you are aware and cover the missing test cases with another approach.

I found quite hard to have a straight answer about what are all the test cases they are covering and also the details, so hope this helps to have a bit more clarity in case you are having the same issues.


Load Tests: Jmeter vs Gatling

Hello guys,

Continuing on reviewing some performance test tools, today is the turn of Jmeter and Gatling, which looks like more and more people are using nowadays. Remember always check your other options and see what better fits for your project.


Jmeter is a great and powerful tool, but depending on what you really need (something more lighter) then Jmeter might become an overcomplex, slow, hard to maintain tool.

Jmeter Gatling
In-built Protocols Support
  • HTTP
  • FTP
  • JDBC
  • SOAP
  • LDAP
  • TCP
  • JMS
  • SMTP
  • POP3
  • IMAP
  • HTTP
  • JMS
  • MQTT
Speed to write tests
  • Slow
  • Fast
Support of “Test as Code”
  • GUI oriented
  • Possibility to create scripts, but too complex and lack of documentation
  • Weak (Java)
  • Hard to maintain
Ramp-up Flexibility
  • Plugins available to be able to configure flexible load
  • Supports ramp-up phases and flexible load
Test Results Analyzing
  • Yes
  • Yes
Resources Consumption
  • Heavy to run tests with multiple users on a single machine, more memory consumption
  • Lighweight and doesn’t take up so much memory of your machine

Screenshot 2020-06-20 at 14.38.36

Easy to use with Version Control Systems
  • No
  • Yes
Number of Concurrent Users
  • Thousands, under restrictions
  • Thousands
Recording Functionality
  • Yes
  • Yes
Distributed Execution
  • Yes
  • Yes
Load Tests Monitoring
  • Add listeners, but consume more memory
  • Yes, logs through the console and reports are created at the end
Screenshot_2020-06-20 Gatling Stats - Global Information


Jmeter is most used when:

  • You need to perform a complex load including different protocols
  • You can record scenarios
  • Robust support and training ecosystem
  • Require that a full scenario be written for every test
  • If you need to simulate specific load with some custom ramp-up patterns
  • If you just prefer UI desktop app for scripts creation, or you just do not know Javascript/YAML/JSON well enough


Gatling solves some specific problems: