Parallel tests with Maven – Junit

Hello guys,

Today I will post about multithreading with Maven. We have some plugins which are able to do the thread safe in your tests:

• Surefire offers a variety of options to execute tests in parallel, allowing you to make best use of the hardware at your disposal.

 

• Forking in particular can also help keeping the memory requirements low.

How can you configure the test execution ?

 
One can impose thread-count limitations on suites, classes or methods using one or more of the parameters threadCountSuites, threadCountClasses and threadCountMethods. If only threadCount is specified, Surefire attempts to estimate the thread counts for suites, classes and methods and reuses the threads in favor of a leaf, e.g. parallel methods (possibly increasing concurrent methods).
 
In the next post I will do some tests with junit and testng, so I can talk more about this plugin with each framework, but for now you can see the example pages: JUnit and TestNG.

 
As an example with an unlimited number of threads, there is maximum of three concurrent threads to execute suites: parallel = all, useUnlimitedThreads = true, threadCountSuites = 3. In the second example the thread-counts represent a ratio, e.g. for parallel = all, threadCount = 16, threadCountSuites = 2, threadCountClasses = 3, threadCountMethods = 5. Thus the concurrent suites will be 20%, concurrent classes 30%, and concurrent methods 50%.
 
Finally, the threadCount and useUnlimitedThreads may not be necessarily configured if the equivalent thread-counts are specified for the value in parallel.
 
The parameters parallelTestsTimeoutInSeconds and parallelTestsTimeoutForcedInSeconds are used to specify an optional timeout in parallel execution. If the timeout is elapsed, the plugin prints the summary log with ERROR lines: “These tests were executed in prior to the shutdown operation”, and “These tests are incomplete” if the running Threads were interrupted.
 
The important thing to remember with the parallel option is: the concurrency happens within the same JVM process. That is efficient in terms of memory and execution time, but you may be more vulnerable towards race conditions or other unexpected and hard to reproduce behavior.
 
 

Parallel Test Execution and Single Thread Execution

 
As mentioned above the parallel test execution is used with specific thread count. Since of Surefire 2.18, you can apply the JCIP annotation @net.jcip.annotations.NotThreadSafe on the Java class of JUnit test (test class, Suite, Parameterized, etc.) in order to execute it in single Thread instance. The Thread has name “maven-surefire-plugin@NotThreadSafe“.
 
This way the parallel execution of tests classes annotated with @NotThreadSafe are forked in single thread instance (don’t mean forked JVM process).
 
If the Suite or Parameterized is annotated with @NotThreadSafe, the suite classes are executed in single thread. You can also annotate individual test class referenced by Suite, and the other unannotated test classes in the Suite can be subject to run in parallel.

Forked Test Execution

 
The parameter reuseForks: to reuse the processes to execute the next tests (reuseForks=true/false). The default setting is forkCount=1/reuseForks=true, which means that Surefire creates one new JVM process to execute all tests in one maven module. forkCount=1/reuseForks=false executes each test class in its own JVM process, one after another. You can use the place holder ${surefire.forkNumber} within argLine, or within the system properties (both those specified via mvn test -D... and via systemPropertyVariables).

<plugins>
[...]
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.18.1</version>
    <configuration>
        <forkCount>3</forkCount>
        <reuseForks>true</reuseForks>
        <argLine>-Xmx1024m -XX:MaxPermSize=256m</argLine>
        <systemPropertyVariables>
            <databaseSchema>MY_TEST_SCHEMA_${surefire.forkNumber}</databaseSchema>
        </systemPropertyVariables>
    </configuration>
  </plugin>
[...]
</plugins>

 
 
In case of a multi module project with tests in different modules, you could also use, say, mvn -T 2 ... to start the build, yielding values for ${surefire.forkNumber} ranging from 1 to 6.
 
By setting reuseForks=true, you can reuse the same context for consecutive tests. And as many tests tend to use and access the same test data, you can avoid database locks during the concurrent execution by using distinct but uniform database schemas.
 
As reuseForks=false creates a new JVM process for each test class, using parallel=classes would have no effect. You can still useparallel=methods, though.
 
When using reuseForks=true and a forkCount value larger than one, test classes are handed over to the forked process one-by-one. Thus, parallel=classes would not change anything. However, you can use parallel=methods: classes are executed in forkCountconcurrent processes, each of the processes can then use threadCount threads to execute the methods of one class in parallel.
 
 

Examples:

 
In your pom.xml file:

Add ths surefire plugin.

<plugins>
[...]
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.18.1</version>
    <dependencies>
      <dependency>
        <groupId>org.apache.maven.surefire</groupId>
        <artifactId>surefire-junit47</artifactId>
        <version>2.18.1</version>
      </dependency>
    </dependencies>
  </plugin>
[...]
</plugins>

After, you must set the parallel parameter, and may change thethreadCount or useUnlimitedThreads attribute.
 

<plugins>
    [...]
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.18.1</version>
        <configuration>
          <parallel>methods</parallel>
          <threadCount>10</threadCount>
        </configuration>
      </plugin>
    [...]
</plugins>

 
As of Surefire 2.7, no additional dependencies are needed to use the full set of options with parallel. As of Surefire 2.16, new thread-count attributes are introduced, namely threadCountSuites, threadCountClasses and threadCountMethods. Additionally, the new attributes parallelTestsTimeoutInSeconds and parallelTestsTimeoutForcedInSeconds are used to shut down the parallel execution after an elapsed timeout, and the attribute parallel specifies new values.

Thank you guys ! Hope you can enjoy and search even more about this. I’ve just summarised what I think it’s more important to know about junit, maven and parallel tests 🙂

As always if you have some question, suggestion feel free to comment below. I will reply as soon as possible and if I don’t know I will find someone to help me reply to you. Have an excellent weekend everyone ! See you next week !
 
 
Resources:

https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html

http://maven.apache.org/surefire/maven-surefire-plugin/examples/junit.html