Load Tests: Jmeter vs Artillery

Just to remind again:

	Jmeter	Artillery
In-built Protocols Support	HTTP FTP JDBC SOAP LDAP TCP JMS SMTP POP3 IMAP	HTTP Socket.io WebSocket
Speed to write tests	Slow	Fast
Support of “Test as Code”	GUI oriented Possibility to create scripts, but too complex and lack of documentation Weak (Java) Hard to maintain	Scripts oriented Strong (JSON/YAML – YAML is the recommended format since it allows comments) Easier to maintain
Ramp-up Flexibility	Plugins available to be able to configure flexible load	Supports ramp-up phases and flexible load
Test Results Analyzing	Yes	Yes
Resources Consumption	Heavy to run tests with multiple users on a single machine, more memory consumption	Light to run tests with multiple users on a single machine, less memory consumption Doesn’t take up so many of your machines’ resources Multicore support
Easy to use with Version Control Systems	No	Yes
Number of Concurrent Users	Thousands, under restrictions	Thousands
Recording Functionality	Yes	No
Distributed Execution	Yes	Yes
Load Tests Monitoring	Add listeners, but consume more memory	No. Reports are only created at the end or you can check the terminal logs.

Jmeter is most used when:

• You need to perform a complex load including different protocols
• If you need the script recording functionality
• Require that a full scenario be written for every test
• If you need to simulate specific load with some custom ramp-up patterns
• If you just prefer UI desktop app for scripts creation, or you just do not know Javascript/YAML/JSON well enough

Artillery solves some specific problems:

• You can write performance scripts pretty fast, there is even a “quick” mode (where you don’t need to create any script)
• Push to your VCS and easily maintain the scripts
• Artillery has WebSocket support out of the box and native support for Socket.io
• Spend minimum time on maintenance without additional GUI applications
• Simulate thousands of test users on local machine without the need to have multiple slaves as it uses Node.js is easier to install and lightweight

Resources:

https://artillery.io/faq.html