Load tests: Jmeter vs K6

Hello all,

Today it’s the turn of Jmeter and K6 ! As always, remember to check your other options and see what better fits for your project.

Jmeter is a great and powerful tool, but depending on what you really need (something more lighter) then Jmeter might become an over complex, slow, hard to maintain tool.

Jmeter K6
In-built Protocols Support
  • HTTP
  • FTP
  • JDBC
  • SOAP
  • LDAP
  • TCP
  • JMS
  • SMTP
  • POP3
  • IMAP
  • HTTP 1.1
  • HTTP 2
  • WebSockets
Speed to write tests
  • Slow
  • Fast
Support of “Test as Code”
  • GUI oriented
  • Possibility to create scripts, but too complex and lack of documentation
  • Weak (Java)
  • Hard to maintain
  • Scripts oriented
  • JavaScript
  • Easier to maintain
Ramp-up Flexibility
  • Plugins available to be able to configure flexible load
  • Supports ramp-up phases and flexible load
Test Results Analyzing
  • Yes
  • Yes
Resources Consumption
  • Heavy to run tests with multiple users on a single machine, more memory consumption
  • Lightweight and doesn’t take up so much memory of your machine

Screenshot 2020-07-06 at 23.34.47

Easy to use with Version Control Systems
  • No
  • Yes
Number of Concurrent Users
  • Thousands, under restrictions
  • Thousands
Recording Functionality
  • Yes
  • No, but it allows to auto-generate a k6 script via an HAR file
Distributed Execution
  • Yes
  • Yes
Load Tests Monitoring
  • Add listeners, but consume more memory

Screenshot 2020-07-06 at 23.35.02

Jmeter is most used when:

  • You need to perform a complex load including different protocols
  • You can record scenarios
  • Robust support and training ecosystem
  • Require that a full scenario be written for every test
  • If you need to simulate specific load with some custom ramp-up patterns
  • If you just prefer UI desktop app for scripts creation, or you just do not know Javascript/YAML/JSON well enough

 

K6 solves some specific problems:

  • CLI tool with developer-friendly APIs.
  • You can use HAR files to generate record sessions
  • Checks and Thresholds – for goal-oriented, automation-friendly load testing
  • Open source, great support and documentation
  • Lightweight uses Javascript
  • Does not run in NodeJS and doesn’t run in a browser

 

Resources:

https://k6.io/

Load Tests: Locust vs Jmeter

Hello guys,

Today I am going to post a comparison of these two different load tests framework. I know most people use Jmeter and it has been longer in the market, but I have recently used Locust and also Artillery (which will post a comparison later) and the results are great, the team was able to improve the creation of the tests and also the maintainability.

In the end of the day you need to use the right tool for your needs, Jmeter is a great and powerful tool, but depending on what you really need (something more lighter) then Jmeter might become an overcomplex, slow, hard to maintain tool.

Jmeter Locust
In-built Protocols Support
  • HTTP
  • FTP
  • JDBC
  • SOAP
  • LDAP
  • TCP
  • JMS
  • SMTP
  • POP3
  • IMAP
  • HTTP, but ability to add a customise protocol
Speed to write tests
  • Slow
  • Fast
Support of “Test as Code”
  • GUI oriented
  • Possibility to create scripts, but too complex and lack of documentation
  • Weak (Java)
  • Hard to maintain
  • Scripts oriented
  • Strong (Python)
  • Easier to maintain
Ramp-up Flexibility
  • Plugins available to be able to configure flexible load
  • Ability to specify a linear load only (so no complex scenarios)
Test Results Analyzing
  • Yes
  • Yes
Resources Consumption
  • Heavy to run tests with multiple users on a single machine, more memory consumption
  • Light to run tests with multiple users on a single machine, less memory consumption
  • Doesn’t take up so many of your machines’ resources.
Easy to use with Version Control Systems
  • No
  • Yes
Number of Concurrent Users
  • Thousands, under restrictions
  • Thousands
Recording Functionality
  • Yes
  • No
Distributed Execution
  • Yes
  • Yes
Load Tests Monitoring
  • Add listeners, but consume more memory
  • Ability to monitor a basic load

 

Jmeter is most used when:

  • You need to perform a complex load including different protocols
  • If you need the script recording functionality
  • If you need to simulate specific load with some custom ramp-up patterns
  • If you just prefer UI desktop app for scripts creation, or you just do not know Python well enough

 

Locust solves some specific problems:

  • You can write performance scripts pretty fast
  • Push to your VCS and easily maintain the scripts
  • Spend minimum time on maintenance without additional GUI applications
  • Simulate thousands of test users on local machine without the need to have multiple slaves

 

Reducing the Scope of Load Tests with Machine Learning

Hello hello,

Today I am going to share this really interesting webinar of my friend Julio de Lima, he is one of the top QA influencers in Brazil, and in this video he is talking about how to reduce the scope of load tests using Machine Learning.

Load testing execution produces a huge amount of data. Investigation and analysis are time-consuming, and numbers tend to hide important information about issues and trends. Using machine learning is a good way to solve data issues by giving meaningful insights about what happened during test execution.

Julio de Lima will show you how to use K-means clustering, a machine learning algorithm, to reduce almost 300,000 records to fewer than 1,000 and still get good insights into load testing results. He will explain K-means clustering, detail what use cases and applications this method can be used in, and give the steps to help you reproduce a K-means clustering experiment in your own projects. You’ll learn how to use this machine learning algorithm to reduce the scope of your load testing and getting meaningful analysis from your data faster.

 

Thank you Julio 🙂

Webinar Jmeter Pipeline – Improving the performance

Hi guys, just sharing a cool webinar that I watched this week about performance tests, but what I thought really interesting is the demo with Taurus and Jmeter.

The performance tests with Taurus look way more readable and simple than when creating on Jmeter.

How to set a list of jmeter properties with a list of jmeter variables

Hey guys, today I will post about a quick code to set properties in runtime from jmeter variables.
Where you could use this ?
If you have a list after extract with json path extractor and you want to set a property to use in another thread group, this could be useful.
For example, I will get the serials of a list extracted with json path extractor.
– Create a beanshell script with:
//Get the count of the serials
String countSerials = vars.get("serials_matchNr");

// create a for to create a property for each serial variable
for(int i=1;i<Integer.parseInt(countSerials);i++){

   //set each property + index with the variable of each serial in the list
   ${__setProperty("serials_"+i,String.valueOf(vars.get("serials_"+i)))};

}
Thank you guys ! See you next week 🙂

How to get a random number from a list on Jmeter

Hey folks,

So for today I will post the code I am using on Jmeter to get a random index from a list. So, if you have a json with several ids, for example, and you want to choose one of them to use on the next request, you need:

1 – Create a post processor to get the id node from your json response request. As you may know when you use this plugin, jmeter creates automatically the list of variables with the prefix you have chosen and the suffix “__index”. You can see how to use this plugin on my previous post.

2 – Create a Beanshell Sample

3 – Get a random index from this list of ids or you could use this method to get the random id directly. I found easiest get the index r at least it was the first thing which came into my mind.

import java.util.Random;
String idSize = vars.get("ids_matchNr");
int max = Integer.parseInt(idSize);
int min = 1;
int idx =  min + (int) (Math.random() * ((max - min) + 1));
String id = vars.get("ids_" + Integer.toString(idx));
Thank you guys ! See you next week or before 🙂

How to get specific value/node from a json with Jmeter

Hi guys, today I will post about how to get a value from a json response with Jmeter.

First we need Jmeter with all plugins installed:

brew install jmeter –with-plugins

 

Let’s use this response for instance. Look that we have a token which is dynamically generated and we need get this value to use in the next request.

Response:

{
   "status": "success",
   "message": "success registering this user",
   "data":    {
      "date": "2015-04-06",
      "id": "1",
      "user": "Rafa",
      "token": "30154dbe350991cf316ec52b8743137b"
   }
}

 

You can use this site to find the json path and get the expression to use on jmeter:
https://jsonpath.curiousconcept.com

So, we got the expression: $.data.token and now we can create the json path extractor with a default value (when jmeter doesn’t find the expression), jsonpath and the variable (which will contain the value of the expression). See:

Screen Shot 2016-03-02 at 22.38.38

Now, we can use the variable on a header or another request:

Screen Shot 2016-03-02 at 22.44.46

 

 

As always, feel free to add any comments !

Cheers. See you next week 🙂

Get json path from an authentication request

 

This example I am getting json path from an authentication request and use it in another group thread.
Remember install the json plugin for jmeter (If you don’t have), you can do that with homebrew command :

brew install jmeter --with-plugins

 

  • Create a test plan and set the web server and the port:

 

Screen Shot 2015-11-11 at 20.08.27

 

  • Create a Thread Group:

 

Screen Shot 2015-11-11 at 20.08.44

  • Create HTTP Request, use the variables for the server and the password. Put the path of the authentication page, username and the password.

 

Screen Shot 2015-11-11 at 20.09.09

  • Create Json Path Extractor and put the path Expression and the variable that you will use. You can test if your path is correct here.

 

Screen Shot 2015-11-11 at 20.10.14

 

  • Create a beanshell Assertion or Post processor and set the property:

 

${__setProperty(access_token,${access_token})};

Screen Shot 2015-11-11 at 20.10.27

 

  • Create a new Thread Group and a HTTP Header Manager and use the same variable you used before:

 

${__property(access_token)}

Screen Shot 2015-11-11 at 20.42.41

 

  • Create the Listener > View Result Tree and it’s done, you can run the jmeter and see if it’s getting the token and using in the next thread group

 

See you guys 🙂

Using Dynamic data from a CSV file with JMeter

Hi guys !

Today I will post this tutorial of Jmeter about how use dynamic data from a CSV (Excel or other) file, because a follower asked for some Jmeter’s tutorial.

I did the same here and didn’t have any problem. So, if you have any question/suggestion, just write below !

Thank you 🙂