Automated tests in a CD/CI pipeline

Good pipelines are stable and can support frequent and small releases. When building the pipeline you need to include not only the build and unit tests part, but also the e2e tests and even the smoke tests and deploy to all the environments, so you have as minimun as human interation as possible, avoiding releases to be deployed by mistake.

CD pipeline

Below you have an image of a simple pipeline where you have the Build, Deploy, Test and Promote/Release stages. Imagine a developer merging his changes to the master branch, triggering a new build on this pipeline. The CI job builds the application, deploy to the specific environment, perform some tests and if everything is ok, the changes are promoted to prod environment if you have a CD environment. If you have any issue in any of these stages, the CI job is going to be stopped and you will have a red pipeline in the history of this job. These basic stages could be followed like:

Build – The stage where you need to build your application, usually I run the unit tests here already.

Deploy – The stage where you are going to deploy your application, so this could be your dev environment or dev and qa environment at the same time.

Test – Depending of which environment you have deployed your application, you will decide what type of tests you are going to run, for instance: smoke tests for the Staging or the e2e tests for the QA environment.

Promote – This stage is where you are going to add the script to deploy your application to the production environment in case the tests passed.


Continuous Integration (CI), Continuous Deployment (CD) and DevOps have a common goal: small, frequent releases of high-quality software. Sometimes of course things happen in the middle of the way and you discover a bug after the deployment to prod, but as you have this pipeline with a CI and CD running, the fix will be deployed to prod as soon as it is fixed and there is no other new bugs.  For this reason it doesn’t matter how your release cycle is, integrated automated tests is one of the keys to have a CI/CD job running and doing the job.

Having a continuous integration pipeline doing tests on all the environments after the deployment helps to get a fast feedback, reduce the volume of the merge conflicts, everyone has a clear view of the status of the build and a current “good build” is always available for demos/release.


Some test/general recommendations

Separate type of tests for each stage – If you have different type of tests running in your CI pipeline (performance, UI, security tests ), better to separate each of them in a different stage so you have a clear vision when one of the stages fail.

Develop your pipeline as code – This is a general advice as it has been a long time that I don’t see teams developing their pipeline through the UI on jenkins. Always create a .jobdsl containing the pipelines that your project is going to have and create/update them automatically.

Wrap your inputs in a timeout – In case you have a pipeline that demands user interaction at some point, you should always wrap your input in a timeout since you don’t want this job hanging there for weeks. I usually have these inputs on the last release stage (deploy to production)

timeout(time:5, unit:'DAYS') {
    input message:'Approve deployment?', submitter: 'it-ops'

Refactor your automated tests – As any development code you need to improve your automation after working in the project and obtaining more knowledge about the product/project. So, to keep your testing efficient, you need to regular look for redundancies that can be eliminated, such as multiple tests that cover the same feature or data-driven tests that use repetitive data values. Techniques such as boundary value analysis can help you to reduce the scope to just the essential cases.

Keep your build fast – Nobody wants a release pipeline that takes hours to finish, it is painful and pointless. Try to trigger the minimum automated tests required to validate your build, keep it simple. Due to their more complex nature, integration tests are usually slower than unit tests. If you have a CD pipeline run a full regression on QA, but if you have a CI and a separate pipeline for the releases run only smoke tests on QA first and then the full regression on the deployment to prod pipeline.

Test in the right environment – You should have an isolated QA platform that is dedicated solely to testing. Your test environment should also be as identical as possible to the production environment, but this can be challenging. Honestly you will probably need to mock certain dependencies such as third-party applications. In complex environments, a virtualization platform or solution such as Docker containers may be an efficient approach to replicate the production environment.

Test in parallel – As speed is essential in a CI/CD environment, save time by distributing your automated tests on multiple stages. As mentioned earlier in this series, keep your automated tests as modular and independent from each other as possible so that you can test in parallel.

parallel {
                stage('Branch A') {
                    agent {
                        label "for-branch-a"
                    steps {
                        echo "On Branch A"
                stage('Branch B') {
                    agent {
                        label "for-branch-b"
                    steps {
                        echo "On Branch B"

Include non-functional tests – Don’t think that regression tests is only about testing functional end-to-end tests, it takes a combination of automated testing approaches to confirm that your application is ready to release. Make sure you have a bit of performance tests running a happy path with concurrent users, also some security tests and of course the e2e functional tests. Exploratory testing can uncover defects that automated tests miss, but then this should have been done before during the feature tests not the release pipeline

Don’t rely only on unit tests – Unit testing doesn’t tell you enough about how that code will work once it is introduced to the production application. Integration of new or revised code may cause a build to fail for several reasons. Therefore, it’s important to run integration tests, regression tests and high-priority functional UI tests as part of the build verification process.


Webinar: Best Practices for Mobile CI – Sauce Labs

Hey guys, today I will post a webinar that I joined last week about Mobile CI.

It is very interesting, so you will see briefly when you need to use, why, how it works, etc. It lasts 43 minutes so, make sure that you have time to watch. Jump to 00:27:35 to watch the demo of Sauce Labs.

To download or watch you need follow the instructions in the link below:


What is CI for mobile ?

Screen Shot 2015-05-28 at 10.45.39

Automatically – every commit

How does it works?

Screen Shot 2015-05-28 at 10.46.57

Why ?

– Reduce the need for human-based testing (Save money and time)

– Faster Feedback

– Automate Everything (Very optimistic point of view)

– No touch configuration

– Automated OTA Distribution

– Code Validation: Automated Builds and Tests


How choose the right process ?

Screen Shot 2015-05-28 at 10.53.00



Depend the size of the




Mobile is harder

– Infrastructure (If you are building in iOS – You will need XCode, a MacMini, etc.)

– Compilation/Code Signing (You have to configure your build machine, so you will need your key store file, mobile provisioning…)

– Testing (What is the right time to use, challenge to choose what is the better for the project- real devices, simulators…)

– Deployment


Simulators x Emulators x Devices


– Used by iOS, included w/ XCode

– Execute i386 instruction set, not ARM

Upside: Very fast compared to Emulators

Downside : You don’t have access hardware functions (GPS,Bluetooth Radio, Accelerometers,etc)


– Used by Android, included with Android sdks

– Execute ARM (or native device instruction set)

Upside: You can find free tools

– Downside: Very slow compared to Simulators

– Do not have access to certain hardware functions (GPS, Bluetooth Radio, Accelerometers, etc)


Upside: Reproduces the actual performance experienced by your users

– Upside: The only way to catch manufacturer/OEM bugs

Downside: Very expensive to configure and maintain compared

– Upside: Full access to hardware functions (GPS, Bluetooth Radio, Accelerometers, etc)


When to use devices vs simulators/emulators ?

– Emulators and Simulators are an excellent solution for running automated tests during development. They are inexpensive and will reliably catch most problems.

– Physical Devices can be used on a lower frequency (i.e. pre-release, weekly, daily, etc.). They are the only way to catch performance problems, test certain hardware features and find OEM issues. In the least, devices should be used before every release.


Unit Testing

– 77% of developers said app quality is “very important or mission critical”

– 80% of mobile developers test their apps manually

– Only 13% of mobile developers use automated testing


Unit vs Functional

Unit: Testing small pieces of code

Functional: Testing button clicks and UI interaction


– Instant Gratification !

– Repeatable

– Can automatically test every commit


– Unit Tests are not users

– Lot of work to write and maintain them

Which framework to use ?

– You need to remember:

– What is the language/framework do your developers know

– Open Source/Community Support

– 3rd party framework requirements


CI Tools/Services for mobile


Open Source, Lots of plugins, iOS and Android 

 Self-Hosted, DYI solution (Setup all the environment, it isn’t easy this part)

Travis CI:

Hosted Solution, OS X support, lots of plugins, iOS and Android 

Tedious setup process

Hosted solution, Designed specifically for mobile, Easy setup, iOS and Android 

Less flexible than other solutions

Xcode CI:

Integrated w/ Xcode, Apple-Supported

Self-hosted, iOS Only

Thank you guys, thank you Sauce Labs ! Hope you like it !

%d bloggers like this: