Good pipelines are stable and can support frequent and small releases. When building the pipeline you need to include not only the build and unit tests part, but also the e2e tests and even the smoke tests and deploy to all the environments, so you have as minimun as human interation as possible, avoiding releases to be deployed by mistake.
CD pipeline
Below you have an image of a simple pipeline where you have the Build, Deploy, Test and Promote/Release stages. Imagine a developer merging his changes to the master branch, triggering a new build on this pipeline. The CI job builds the application, deploy to the specific environment, perform some tests and if everything is ok, the changes are promoted to prod environment if you have a CD environment. If you have any issue in any of these stages, the CI job is going to be stopped and you will have a red pipeline in the history of this job. These basic stages could be followed like:
Build – The stage where you need to build your application, usually I run the unit tests here already.
Deploy – The stage where you are going to deploy your application, so this could be your dev environment or dev and qa environment at the same time.
Test – Depending of which environment you have deployed your application, you will decide what type of tests you are going to run, for instance: smoke tests for the Staging or the e2e tests for the QA environment.
Promote – This stage is where you are going to add the script to deploy your application to the production environment in case the tests passed.
Continuous Integration (CI), Continuous Deployment (CD) and DevOps have a common goal: small, frequent releases of high-quality software. Sometimes of course things happen in the middle of the way and you discover a bug after the deployment to prod, but as you have this pipeline with a CI and CD running, the fix will be deployed to prod as soon as it is fixed and there is no other new bugs. For this reason it doesn’t matter how your release cycle is, integrated automated tests is one of the keys to have a CI/CD job running and doing the job.
Having a continuous integration pipeline doing tests on all the environments after the deployment helps to get a fast feedback, reduce the volume of the merge conflicts, everyone has a clear view of the status of the build and a current “good build” is always available for demos/release.
Some test/general recommendations
Separate type of tests for each stage – If you have different type of tests running in your CI pipeline (performance, UI, security tests ), better to separate each of them in a different stage so you have a clear vision when one of the stages fail.
Develop your pipeline as code – This is a general advice as it has been a long time that I don’t see teams developing their pipeline through the UI on jenkins. Always create a .jobdsl containing the pipelines that your project is going to have and create/update them automatically.
Wrap your inputs in a timeout – In case you have a pipeline that demands user interaction at some point, you should always wrap your input in a timeout since you don’t want this job hanging there for weeks. I usually have these inputs on the last release stage (deploy to production)
timeout(time:5, unit:'DAYS') {
input message:'Approve deployment?', submitter: 'it-ops'
}
Refactor your automated tests – As any development code you need to improve your automation after working in the project and obtaining more knowledge about the product/project. So, to keep your testing efficient, you need to regular look for redundancies that can be eliminated, such as multiple tests that cover the same feature or data-driven tests that use repetitive data values. Techniques such as boundary value analysis can help you to reduce the scope to just the essential cases.
Keep your build fast – Nobody wants a release pipeline that takes hours to finish, it is painful and pointless. Try to trigger the minimum automated tests required to validate your build, keep it simple. Due to their more complex nature, integration tests are usually slower than unit tests. If you have a CD pipeline run a full regression on QA, but if you have a CI and a separate pipeline for the releases run only smoke tests on QA first and then the full regression on the deployment to prod pipeline.
Test in the right environment – You should have an isolated QA platform that is dedicated solely to testing. Your test environment should also be as identical as possible to the production environment, but this can be challenging. Honestly you will probably need to mock certain dependencies such as third-party applications. In complex environments, a virtualization platform or solution such as Docker containers may be an efficient approach to replicate the production environment.
Test in parallel – As speed is essential in a CI/CD environment, save time by distributing your automated tests on multiple stages. As mentioned earlier in this series, keep your automated tests as modular and independent from each other as possible so that you can test in parallel.
parallel {
stage('Branch A') {
agent {
label "for-branch-a"
}
steps {
echo "On Branch A"
}
}
stage('Branch B') {
agent {
label "for-branch-b"
}
steps {
echo "On Branch B"
}
}
}Include non-functional tests – Don’t think that regression tests is only about testing functional end-to-end tests, it takes a combination of automated testing approaches to confirm that your application is ready to release. Make sure you have a bit of performance tests running a happy path with concurrent users, also some security tests and of course the e2e functional tests. Exploratory testing can uncover defects that automated tests miss, but then this should have been done before during the feature tests not the release pipeline
Don’t rely only on unit tests – Unit testing doesn’t tell you enough about how that code will work once it is introduced to the production application. Integration of new or revised code may cause a build to fail for several reasons. Therefore, it’s important to run integration tests, regression tests and high-priority functional UI tests as part of the build verification process.
https://www.cloudbees.com/blog/top-10-best-practices-jenkins-pipeline-plugin
Rafaela Azevedo 
One thought on “Automated tests in a CD/CI pipeline”