Load Tests: Locust vs Jmeter

Hello guys,

Today I am going to post a comparison of these two different load tests framework. I know most people use Jmeter and it has been longer in the market, but I have recently used Locust and also Artillery (which will post a comparison later) and the results are great, the team was able to improve the creation of the tests and also the maintainability.

In the end of the day you need to use the right tool for your needs, Jmeter is a great and powerful tool, but depending on what you really need (something more lighter) then Jmeter might become an overcomplex, slow, hard to maintain tool.

Jmeter Locust
In-built Protocols Support
  • HTTP
  • FTP
  • JDBC
  • SOAP
  • LDAP
  • TCP
  • JMS
  • SMTP
  • POP3
  • IMAP
  • HTTP, but ability to add a customise protocol
Speed to write tests
  • Slow
  • Fast
Support of “Test as Code”
  • GUI oriented
  • Possibility to create scripts, but too complex and lack of documentation
  • Weak (Java)
  • Hard to maintain
  • Scripts oriented
  • Strong (Python)
  • Easier to maintain
Ramp-up Flexibility
  • Plugins available to be able to configure flexible load
  • Ability to specify a linear load only (so no complex scenarios)
Test Results Analyzing
  • Yes
  • Yes
Resources Consumption
  • Heavy to run tests with multiple users on a single machine, more memory consumption
  • Light to run tests with multiple users on a single machine, less memory consumption
  • Doesn’t take up so many of your machines’ resources.
Easy to use with Version Control Systems
  • No
  • Yes
Number of Concurrent Users
  • Thousands, under restrictions
  • Thousands
Recording Functionality
  • Yes
  • No
Distributed Execution
  • Yes
  • Yes
Load Tests Monitoring
  • Add listeners, but consume more memory
  • Ability to monitor a basic load

 

Jmeter is most used when:

  • You need to perform a complex load including different protocols
  • If you need the script recording functionality
  • If you need to simulate specific load with some custom ramp-up patterns
  • If you just prefer UI desktop app for scripts creation, or you just do not know Python well enough

 

Locust solves some specific problems:

  • You can write performance scripts pretty fast
  • Push to your VCS and easily maintain the scripts
  • Spend minimum time on maintenance without additional GUI applications
  • Simulate thousands of test users on local machine without the need to have multiple slaves

 

Reducing the Scope of Load Tests with Machine Learning

Hello hello,

Today I am going to share this really interesting webinar of my friend Julio de Lima, he is one of the top QA influencers in Brazil, and in this video he is talking about how to reduce the scope of load tests using Machine Learning.

Load testing execution produces a huge amount of data. Investigation and analysis are time-consuming, and numbers tend to hide important information about issues and trends. Using machine learning is a good way to solve data issues by giving meaningful insights about what happened during test execution.

Julio de Lima will show you how to use K-means clustering, a machine learning algorithm, to reduce almost 300,000 records to fewer than 1,000 and still get good insights into load testing results. He will explain K-means clustering, detail what use cases and applications this method can be used in, and give the steps to help you reproduce a K-means clustering experiment in your own projects. You’ll learn how to use this machine learning algorithm to reduce the scope of your load testing and getting meaningful analysis from your data faster.

 

Thank you Julio 🙂

Lead your Team Through the Crisis

I’ve recently watched a webinar about how to lead and communicate better with your team during this world crisis. Everybody is dealing in a different way and some of us are struggling in this unprecedented time of lockdown. I particularly work much better at home and without the distractions of the office, but not everybody is the same.

R. Michael Anderson talks about mental health during this COVID-19 and how we can overcome the issues and communicate better, you can watch the meetup recording below.

He shares his views and real life experience on: 

  • The systems and processes you need to not just survive, but to thrive over the next few months.
  • That real reason leaders cave under the pressure of a crisis…and how to not be one of those casualties.
  • What exactly your team needs from you right now, and how to give that to them.
  • How to find another level of leadership within yourself in just a few quick steps.
  • AND how to do all of this right away so that you and your team perform at a high level when it’s needed the most.

 

 

One of the suggestions is to apply the DiSC assessment, which I am going to talk a bit deeper here. You can use it to help team members better understand each other and what is the way they prefer to communicate. Doing that you can improve the workplace and it can become a more productive and comfortable environment.

The whole point of this kind of test is to build emotional intelligence between its members, personal growth and self awareness. Reminding each other that we’re all human beings (not machines) with different perspectives, plans, experiences, and skills. This understanding is essential and helps teams to perform better and feel like a team avoiding a blaming/competitive culture.

When letting the team members know about the test, it’s important they understand it’s strictly a tool to understand each other better and work well together. There are no right or wrong answers !

By taking the personality assessment test and then reviewing the results together, the team will learn more about the other members and also themselves, noticing what is the best way to interact with each other.

The DiSC model organizes personalities among four quadrants, with people usually being dominant in one of them:

  • dominance, an emphasis on accomplishing results
  • influence, an emphasis on relationships and influencing others
  • steadiness, an emphasis on cooperation and dependability
  • conscientiousness, an emphasis on quality, accuracy and competency

Results from the DiSC model are presented on a circle, with a dot in one of the quadrants, so test takers can see where their personality lies and identify how much of each quadrant is relevant to them. Keeping track of the results can help everybody to better evaluate conflicts, recognize high performers and address communication issues on teams.

According to this detailed analysis from EQ experts Dr. Vanessa Druskat and Dr. Steven Wolff:

“Our research tells us that three conditions are essential to a group’s effectiveness: trust among members, a sense of group identity, and a sense of group efficacy.”

Here it is an example of a DiSC assessment:

This also reminds me of the “benefits” the companies offer: fruits in the office, table football, ps 4, bonus and the list goes on… When the only thing you want is to be able to work from home to stay with your dog and go to your local pub while it is still sunny.

It is impossible to make everybody happy, so it is better if you can understand what motivates EACH one of the team members instead of pushing things they don’t want/care.

You can take the personality test for free and understand more about yourself and how to better communicate with others here: https://www.truity.com/test/disc-personality-test

 

Resources:

https://www.tonyrobbins.com/disc/

https://www.paycom.com/resources/blog/how-to-use-personality-assessment-tests-to-improve-workplace-culture-and-communication/

https://blog.trello.com/personality-tests-for-teams-at-work

Your automation framework smells !

Code smell is everything that can slow down your process or increase the risk of implementing bugs when doing maintenance. 

The vast majority of the places that I have worked think it is okay to have an automation project with poor quality. Unfortunately, this is an idea shared by many QAs as well. Who should test the test automation ? It is probably alright to have duplicated code, layers and layers of abstraction… After all, it is not the product code, why should we bother, right ?

 

Automation is a development project that should follow the same best practices to avoid code smells. You need to ensure the minimum of the quality on your project, so: add a code review process, a code quality tool, and also test your code before pushing the PR (like changing the expectations and see if it is going to fail).

Of course you don’t need to go too deep and create unit/integration/performance tests for your automation project (who test the tests right ?), but you definitely need to ensure you will have a readable, maintainable, scalable automation project. This is going to be maintained by the team, it needs to be simple, direct and easy to understand. If you spend the same amount of time on your automation and on your development code, something is wrong.

You want to have an extremely simple and easy to read automation framework, so you can have a lot more confidence that your tests are correct. 

I will post here some of the most common anti-patterns that I have found during my career. You might have come across some others as well.

 

Common code smells in Automation framework

– Long class(God object), you need to scroll for hours to find something, it has loads of methods and functions. You don’t even know what this class is about anymore.

 

– Long BDD scenarios, try to be as simple and straight forward as possible, if you create a long scenario it is going to be hard to maintain, to read and to understand.

 

– BDD scenarios with UI actions, your tests should not rely on the UI, no actions like click, typed, etc. Try to use more generic actions like send, create, things that even if the UI changes the action doesn’t need to change.

 

– Fragile locators / Xpath from hell, any small change on the UI would fail the tests and require to update the locator.

 

– Duplicate code, identical or very similar code exists in more than one location. Even variables should be pain free maintenance. Any change means changing the code in multiple spots.

 

– Overcomplexity, forced usage of overcomplicated design patterns where simpler design would be enough. Do you really need to use dependency injection ?

 

– Indecent Exposure, too many classes can see you, limit your scope.

 

– Shotgun surgery, a single change needs to be applied to multiple classes at the same time.

 

– Inefficient waits, it slows down the automation test pipeline, can make your tests flaky.

 

– Variable mutations, very hard to refactor code since the actual value is unpredictable and hard to reason about.

 

– Boolean blindness, easy to assert on the opposite value and still type checks.

 

– Inappropriate intimacy, too many dependencies on implementation details of another class.

– Lazy class / freeloader, a class that doesn’t do much.

– Cyclomatic complexity, too many branches or loops, this may indicate a function needs to be broken into smaller functions, or that it has potential for simplification.

 

– Orphan variable or constant class, a class that typically has a collection of constants which belong elsewhere where those constants should be owned by one of the other member classes.

 

– Data clump, occurs when a group of variables are passed around together in various parts of the program, a long list of parameters and it is hard to read. In general, this suggests that it would be more appropriate to formally group the different variables together into a single object, and pass around only this object instead.

 

– Excessively long identifiers, in particular, the use of naming conventions to provide disambiguation that should be implicit in the software architecture.

 

– Excessively short identifiers, the name of a variable should reflect its function unless the function is obvious.

 

– Excessive return of data, a function or method that returns more than what each of its callers needs.

 

– Excessively long line of code (or God Line), a line of code which is too long, making the code difficult to read, understand, debug, refactor, or even identify possibilities of software reuse.

 

How can you fix these issues ?

  • Follow SOLID principles ! Class, methods should have a single responsibility !
  • Add a code review process and ask the team to review (developers and other QAs).
  • Lookout how many parameters you are sending. Maybe you should just send an Object.
  • Add locators that are resistant to UI changes, focus on ids first.
  • Return an object with the group of the data you need instead of returning loads of variables.
  • Focus to name the methods and classes as direct as possible, remember SOLID principles.
  • If you have a method that just type a text in a textfield, it maybe grouped together to a function that is going to perform the login().
  • If you have long lines of code, you might want to split it up into functions and move some of them to a variable and then formatting this variable, for example.
  • Think twice about the boolean assertions, add a comment if you think it is not straight forward.
  • Follow POM structure with helpers and common shared steps/functions to avoid long classes.
  • Do you really need this wait ? You might be able to use a retry or maybe your current framework have ways to deal with waits properly.
  • Add a code quality tool to review your automation code (eg. ESlint, Code Inspector)

 

Resources:

https://en.wikipedia.org/wiki/Code_smell

https://pureadmin.qub.ac.uk/ws/portalfiles/portal/178889150/MLR_Smells_in_test_code_Dec_9.pdf

https://www.sealights.io/code-quality/the-problem-of-code-smell-and-secrets-to-effective-refactoring/

https://slides.com/angiejones/automation-code-smells-45

https://medium.com/ingeniouslysimple/should-you-refactor-test-code-b9508682816

Automation challenges

Hello guys !!

I thought about sharing something that we all have gone through at some point of our career. Some of these challenges are related to lack of standards, knowledge and processes, but some others are related to company’s culture and people’s mindset (this is the biggest challenge and most difficult to change).

I have posted about the difference between a growth mindset and a fixed mindset some years ago, after joining  a workshop where Joanna Chwastowska (Engineering Lead at Google) explains how she learned to have a growth mindset, this picture below summarizes it:

 

Basically, we are always learning and you shouldn’t feel ashamed to admit that 🙂

 

Scope

What scenarios should I have for this layer of tests ? What type of tests should I be responsible for ? These should be the questions that you first ask when deciding the scope of your tests. For each layer you need to have a different scope otherwise you are not only going to duplicate scenarios, but also have an extensive pipeline that is hard/impossible to maintain.  

Decide what are the layers that you are going to cover: Unit, Integration, E2E Tests… then decide what scenarios for each layer. So, for the Unit tests you need to have an extensive number of scenarios covering edge cases (special characters, etc), then integration between components and finally e2e tests on top with a reduced scope and no mocks. You should also consider UI tests comparing snapshots if you have a frontend as well.  

 

Team collaboration

Make sure you have a team that is working towards the same goal, improve the communication as much as possible. Do workshops, demos, pair-programming, code reviews, get feedback so you can continue to improve until you find the best way to work and at the same time make the business happy.

Understand the expectations and align them, there is nothing better than having a nice bunch of people to spend ~8 hours together for 5 days a week and achieving something together.

 

Time constraints

How many releases do you have per day ? How many projects are you allocated in ? How many developers do you have in your team ? You might have noticed that you usually have more than one developer for each QA in a team, and this is okay, as long as you manage what you are going to test, you can’t save the world, don’t test everything, focus on the end-user flow above everything else as this is the front door of the product.

Something you need to take into account is the scope of the regression pack, you probably want to have automation for that, right ? I am completely against having manual tests for the regression pack, unless there is a strong reason why.

 

Finding elements

Do you remember when you couldn’t click on that element because there was a div on top of it ? This is one of the problems you might have faced already, or maybe was a bad structured xpath ? or too many elements with the same css-selector ?

If you are testing react apps, here is something that helped when doing the tests, I’ve asked the developers to add one a data-test attribute: data-test, data-test-id, data-testid or data-cy to the elements. Adding this kind of attribute is considered a best practice since makes the automation resilient to change and it is dedicated for tests.

 

Flaky tests

Yeah… we all know the struggle is REAL !

I’ve faced this issue recently when doing tests with Espresso on android apps, instead of using waits remember to use idling resources which synchronizes the subsequent operations in a UI test.

For the react apps you can use frameworks like Cypress, Testcafe and Detox that runs in the same run-loop as the application being tested and can automatically wait for assertions and commands before moving ahead.

These are some of the reasons you can have flaky tests, but there are some other reasons like:

  • Environment/server is not stable
  • 3rd party system integration is not stable
  • Concurrency tests
  • Caching
  • Setup/teardown tests
  • Dynamic content

Identify the reason first to be able to take the correct action, but definitely tag this test as soon as possible since you won’t be able to trust on it until it has been fixed.

 

#BeSafe

Docker Quick Clean Script

Hey guys, just sharing a quick script that I saved here years ago about cleaning all docker images/containers/volumes:

 

#!/bin/bash

# Kill all running containers.
docker kill $(docker ps -q)

# Delete all stopped containers.
printf "\n>>> Deleting stopped containers\n\n" && docker rm $(docker ps -a -q)

# Delete all untagged images.
printf "\n>>> Deleting untagged images\n\n" && docker rmi $(docker images -q -f dangling=true)

# Delete all images.
printf "\n>>> Deleting untagged images\n\n" && docker rmi $(docker images -q)

# Delete all volumes.
printf "\n>>> Deleting volumes\n\n" && docker volume rm $(docker volume ls -q)

# Remove all unused containers, networks, images (both dangling and unreferenced), and optionally, volumes.
printf "\n>>> Deleting unused containers, networks, images and volumes\n\n" && docker system prune -a --volumes

 

Gist link:

https://gist.github.com/rafaelaazevedo/bec6cf339888bbac60336d01193ae923

TestProject Cloud Integrations

Test Clouds are a great solution to have multiple devices and browsers running your tests in parallalel. It is really cost effective since you don’t need to have real devices and machines to be able to get it running, there are some cons as well, like you can have some bandwidth issues.

Many frameworks are already able to run your tests in the cloud, it is really easy to setup as you just need to know the command to pass like you would do on Jenkins or any other CI tool. Currently TestProject is able to run tests in the SauceLabs and BrowserStack clouds and you can setup any of them quite easily following the documents for SauceLabs here and BrowserStack here.

 

Pros vs Cons having your tests running in the cloud

 

Pros Cons
Dynamic test environment easy to setup Possible bandwidth issues
Faster than having real devices Loss of autonomy
Scalable Small security risk
Environment customizable No free tools
Cost-effective
You can access any time 24/7
Improve team collaboration

 

Resources:

https://link.testproject.io/wpq

https://docs.testproject.io/testproject-integrations/browserstack-integration

https://www.lambdatest.com/blog/benefits-of-website-testing-on-cloud/