How to get your test reports from a pod – Kubernetes

Hello guys, today I am going to post a workaround that I am doing to get test reports from a kubernetes pod that has terminated. If you run your tests in a kubernetes pod, you must know by now that you can’t copy your files containing your test reports from a terminated pod. Many people were complaining about and you can see there is an open issue here.

I am going to show an example of api tests with postman and newman where I run the tests in a kubernetes pod and in the same microservices cluster since there is no public api to access them.

Create a Job

  • First you need to create a job and save it in a .yaml file, like the example below:
apiVersion: batch/v1
kind: Job
metadata:
  name: api-tests
  namespace: default
spec:
  parallelism: 1
  template:
    metadata:
      name: api-tests
    spec:
      containers:
      - name: api-tests
        image: postman/newman:alpine
        command: ["run"]
        args: ["/etc/newman/test.postman_collection.json","--reporters","cli","--reporter-cli-no-failures"]
      restartPolicy: Never
  • Then run on your terminal or add it to your jenkins file:

kubectl apply -f job.yaml

Or you can also create a pod

  • You just need to add this command to your jenkins file:
 sh "kubectl run api-tests -i --rm --namespace=${ENVIRONMENT_NAMESPACE} --restart=Never --image=${YOUR_IMAGE}:latest --image-pull-policy=Always -- run /etc/newman/${YOUR_COLLECTION_PATH}.postman_collection.json -e /etc/newman/${YOUR_ENVIRONMENT_CONFIG_PATH}.postman_environment.json --reporters cli --reporter-cli-no-failures"

Why not a Deployment ?

In the beggining of the implementation I first tried to create a deployment, but deployments don’t support the policy to never restart the pod, which means the automation would never stop running and you wouldn’t be able to copy the reports from the container.

 

Copy the reports

So, now that your tests have finished, you can see the logs on jenkins showing they passed (or failed), but you want to extract the report from the logs and have them in a html/json/any file, so you can archive or publish them. Doing this you would be able to see clearer what is the issue and keep easy to access the reports for each pipeline.

Well,kubectl cp doesn’t work like docker cp unfortunatelly. Once your pod is terminated, you are not able to access the reports or anything inside the pod. So, for this reason there is an issue opened on the kubectl github repository that is exactly about that, you can check the progress of the issue here.

Now how can you copy the reports from the container if you can’t access it after the tests are finished ? Well, there is not a perfect way, some people send the reports to S3, some people send the reports to their emails, but I did find better to save the report copying the html code from the logs and saving it in a file.

On your jenkins file you will have the command to run the pod with the tests and after you need to cat the html report generated to be able to get everything inside the html tag and saving it in a file:

      sh "kubectl run api-tests -i --rm --namespace=${ENVIRONMENT_NAMESPACE} --restart=Never --image=${YOUR_IMAGE}:latest --image-pull-policy=Always -- run /etc/newman/${YOUR_COLLECTION_PATH}.postman_collection.json -e /etc/newman/${YOUR_ENVIRONMENT_CONFIG_PATH}.postman_environment.json --reporters cli,html --reporter-html-export api-tests.html --reporter-cli-no-failures ; cat api-tests.html | tee report"
      def report = readFile "report"
      def update = report.substring(report.indexOf('<html>'), report.indexOf('</html>'))
      writeFile file: "${workspace}/api-tests.html", text: update
      sh "rm report"

First, you need to cat the html report that your tests generated (remember to have this script to run the tests in your docker image or the package.json if you use NodeJs, this is just an example). You can see that you will need to grab everything between the html tags. You can do this using substring or awk, whatever is your preference. I am using substring on this example, but if you want to filter out using awk, the code should be something like this awk '/<html>/','/<\\html>/'.

After grabbing the html report, I am saving it in a file and deleting the previous report file that contained the whole logs from the kubernetes pod.

Not perfect, not happy about doing these kind of workarounds, but this is a way to avoid sending the files to S3 or any other place. Hope it helps !

Improve the architecture of your cross-platform automation

Hi guys, today I am going to talk about how to better architect your test code and obtain better cross-platform code reuse.

The idea is that you provide an implementation of page objects for each platform you need to support (e.g. iPhone, iPad, Android phone, Android tablet, mobile web, desktop web,…).

 

Use Page Objects

While originating in web testing, the ideas of POP apply equally well to native mobile. The most obvious benefit of this is abstraction and reuse. If you have several steps needing to navigate to details, the code is reused. Also, callers of this method need not worry about the details (e.g. query and touch) or navigating to this screen.

 

Don’t specify fields in your feature files

Working this way gets you complete reuse of Cucumber features as well as step definitions: the details of interacting with the screen is pushed to page object implementations.

Scenarios should be written like a user would describe them. Beware of scenarios that only describe clicking links and filling in form fields, or of steps that contain code or CSS selectors. This is just another variant of programming, but certainly not a feature description.

Declarative features are vivid, concise and contain highly maintainable steps.

For example:

Scenario: Valid login credentials
  Given I am on login page
  When I enter valid credentials
  Then I should be redirected to the management page

 

Define a page object for each screen

For each screen you want to automate, decide on an interface for a page object class (e.g. like LoginScreen). This means that the calling code, which is usually in a step definition, is independent of platform – hence it can be reused across platforms. For each supported platform, define a class with implementations of the page-object methods.

 

Don’t expose the framework in your step definitions

Use only custom steps, and in each step definition only use page objects and their methods (no direct calls to the framework (Calabash, Selenium, Appium, Robotium).

 

Reuse step definitions

This comes in handy when a step extends another step’s behaviour or defines a superior behaviour that consists of multiple steps. You should try to reuse steps as often as possible.This will improve the maintainability of your app: If you need to change a certain behaviour, you just need to change a single step definition.

For example (Using javascript and protractor):

 this.Given(/^I am on login page$/, function() {
    loginPage.openLoginPage();
 });

 

Thank you guys ! See you next week !

 

Resources:

https://developer.xamarin.com/guides/testcloud/calabash/xplat-best-practices/

https://blog.codeship.com/cucumber-best-practices/