Choose the right test framework for mobile


Today I will post more about my experiences in developing the automation project from scratch. I developed 2 test mobile automation from scratch until now: one was with Calabash and Cucumber and the other one was with Robotium and Cucumber.

  • Support – Be sure that you will have a lot of support from the framework team. If it’s an open source you could have fast support or not, depends of how the developers are busy. So, make sure that you will have a good support when you start to use the framework, go to the github of the project and look for the open issues and what is the frequency they reply and when it was the last bug fixed. Pay attention the frequency of the answers in forums on google groups/linkedin groups/stackoverflow, etc.


  • Stability – Is stable enough ? How long this framework is in the market ? Really, you don’t want to start your automation with a lot of problems because the framework that you are using is still in beta phase or is still improving a lot of things. You need to be sure that your choice will depend more in you and your code than the framework you’ve chosen.


  • Your app – Yes, you need to know first if your app is stable, with good performance and what are the objectives of your automation. Some frameworks work very well with stable apps, but when you need to test an app with memory leaks or performance problems you won’t be able to even start a scenario. I worked with Calabash most of my mobile automation experience and to be honest I didn’t have any problems to test some unstable apps, but when I did a POC with Espresso, the first simple scenario couldn’t even go further the first step, just because the app was not stable enough (and this it wasn’t the first priority of the automation – of course if you know that you have performance issues and they are not relevant enough you should be able to carry on the automation).


  • Developers – Again, I will use the experience that I had with Espresso. The developers are from Google, which is a famous company. You could think, of course I will choose this one, because google is taking care of it. No, to be honest, I don’t really care about the company, I may consider the fact of the framework being developed from a good company, but in first place I see all the priorities above. In this example: Espresso is relative new if you compare with robotium or calabash, it takes serious about the performance of the app, so it won’t go further the automation if you have performance problems, the support is really fast and you can find a lot of people who already started use it.


  • Pressure – You need to consider this, probably you will have a developer who will need to push you to use the framework he thinks it is really good (OMG Espresso is being developed by Google, we need to use it, because reasons and stuffs). I think all of us already worked with people like this, I am not saying that you need to ignore them, but just pay attention about what is more important and WHAT IT WILL WORK FOR YOUR COMPANY/APP. Please, not all the apps or companies work in the same way, you don’t need to follow the crowd, just follow a single tip to figure out which framework is better: POC.


  • POC (Proof of concept) – The last tip and probably the most important one. If you want to know which framework will work better with your app it’s easy, take one scenario, a basic one, and automate it for the frameworks that you are in doubt 🙂


I hope this helps someone as well. If you have any suggestions/questions, please fell free to comment below. See you next week 🙂

Environment variables, BuildConfig, and Android Studio

Hello guys,

Today I will post some simple steps to have some environment variables in your android studio project. So, if you have some confidential data to use in your android tests, you can hide the password, username or if you just want to have some environment variables separated from your project, just follow these steps:

So, let’s start:

Screen Shot 2015-09-17 at 21.58.28

  • In your create the environment variables (You can find this file inside of your Android Studio project or in your /Users/Your Username/.gradle)
  • In your build.gradle file (Remember, you can have many build.gradle file, so first try to find the one that you want to share with all projects or the one which you will use only in your project > buildTypes > debug, as it is showed in the structure below). The “USERNAME” and “PASSWORD” will be the variables in your code and the testUsername and testPassword should be the same variable which you are using in your file (above).
android {
buildTypes {
  debug {
    buildConfigField "String", "USERNAME", testUsername
    buildConfigField "String", "PASSWORD", testPassword

  release {
  • After this, you need to sync your gradle.
  • And in your code, just call the variables like this:

I am using gradle 2.4 and androidStudio 1.3.2.

Thank you, see you next week 🙂


How to install Espresso and Cucumber in Android Studio

You can download a sample project that is already configured and try first:

  • The structure of your project should be like this:

On Android View:

Screen Shot 2015-09-12 at 15.14.59

On Project View:

Screen Shot 2015-09-12 at 15.14.46


  • Now, open the build.gradle of your app and write these dependencies:
dependencies {
 androidTestCompile ''
 androidTestCompile ''
 androidTestCompile 'info.cukes:cucumber-android:1.2.4'
 androidTestCompile 'info.cukes:cucumber-picocontainer:1.2.4'
  • I had some problems with the version of java, if you have the same problem just update your java or downgrade/upgrade the version of the plugin which is incompatible.
  • In your build.gradle you will need to write more these configs. Change the name of your application and the package of the runner, following the structure of your project and sync your build.gradle file.
android {
 defaultConfig {
 testApplicationId "com.example.azevedorafaela.myapplication"
 testInstrumentationRunner "com.example.azevedorafaela.myapplication.test.
   sourceSets {
     androidTest {
        assets.srcDirs = ['src/androidTest/assets']
  • In your Feature file you an write your scenario like this:
Feature: Test

Scenario: Espresso with cucumber test
Given I have my app configured
When something happens
Then I should see xx on the display
  • In your Instrumentation class:
package com.example.azevedorafaela.myapplication.test;
import android.os.Bundle;
public class Instrumentation extends MonitoringInstrumentation {
private final CucumberInstrumentationCore instrumentationCore = new 
 public void onCreate(final Bundle bundle) {
 public void onStart() {
  • In your Steps file:
package com.example.azevedorafaela.myapplication.test;

import android.test.ActivityInstrumentationTestCase2;

import com.example.azevedorafaela.myapplication.MainActivity;

import cucumber.api.CucumberOptions;
import com.example.azevedorafaela.myapplication.R;
import static;
import static;
import static
import static;
import static;

@CucumberOptions(features = "features")
public class MainActivitySteps extends ActivityInstrumentationTestCase2
<MainActivity> {

    public MainActivitySteps(){

    @Given("^I have my app configured$")
    public void I_have_my_app_configured() {

    @When("^something happens$")
    public void something_happens(final char op) {


    @Then("^I should see xx on the display$")
    public void I_should_see_xx_on_the_display(final String s) {


Now you can start write your espresso code inside of each step. To run your test you need to:

  • In your terminal, open your project folder and run:
gradle --parallel :app:assembleDebugTest
  •  Now you need istall the apk in your device/simulator:
adb install -r app/build/outputs/apk/app-debug.apk
  • Check if your app is installed. Should display the instrumentation of your app
adb shell pm list instrumentation
  • To run the tests with gradle:
gradle connectedCheck
  • To run the tests with adb:
adb shell am instrument -w com.example.azevedorafaela.myapp
  • To run your tests via android configurations:
    1. Open your run configurations
    2. Create an androidTests
    3. Give a name to this config
    4. Select your module
    5. Don’t write anything on the instrumentation field. (We already configured this in our build.gradle)
    6. Ok.

Screen Shot 2015-09-12 at 15.39.22


Now you can run your cucumber with espresso tests. I hope this “tutorial” helps you as helped me to install everything on my project.

Thank you guys ! See you next week 🙂

10 Steps to setting up the QA Area

Hello guys, today I will post about some steps that you can follow to create the QA area from the scratch.

1 – Make questions like:

  • Do you have any written scenarios ?
  • Who is writing the scenarios and in what phase ?
  • QA team will need create its own scenarios, what phase may the team do that ?
  • Who will need to look the scenarios, just QA and dev area ?
  • Who will run the tests DEV and QA only ?
  • How the application is working ?
  • What are the critical scenarios ? Like, what are the scenarios which will crash the functions/app/process ?
  • What are the most used scenarios (Like create an account…) ?
  • What are the scenarios which are more unstable, like if you change a simple thing you will need to test this scenario every time ?
    • What are the most repetitive tests ?
  • Do you need run the regression everyday ? before every release ? (regression tests, exploratory tests, monkey tests, stress tests, performance tests) every time after a push ? (smoke tests)
  • Do you have a QA environment ?
  • If you are in a mobile/web project, remember to make these questions:
    • What are the most used devices/browsers/OS ?
    • What are the most unstable devices/browsers/OS ?
    • What are the most used versions on these browsers/OS ?
    • What are the most unstable versions on these browsers/OS ?
  • If you are in a mobile project:
    • The app is hybrid, native or web app ?
  • If you are in a web project:
    • Is this site responsive ?
  • Do you have a process of QA, like SCRUM, Kanban ?
  • If you find a bug in a task, do you return the task or create the bug to be fix separately ?

2 – Understand what will be first priority and until where you will reach (Like performance tests, integration ui tests, etc…), so you can have a big picture of the project

3 – Remember that if you are using BDD, it is just 3 layers (Don’t expose your code)

4 – Decide what are the scenarios that must be in the regression (Priority).  What are the most repetitive tests ?

5 – Use tags for all the type of scenarios, like smoke, regression, iOS, android, manual. Even manual tests should be in the features inside the automation project

6 – Don’t use too many complex steps in your scenarios

7 – Try to re-use steps so you don’t need waste time

8 – Use examples

9 – Regression tests – Automated tests

  • Choose the tool, do a POC with different frameworks
    • On this POC, choose the most simple and important scenario (E.g. Create an account)
  • Considerate the language which the developers are using (You never know when you will need help)
  • Distribute the right level of tests between unit (70%), integration (20%) and manual tests (10%) – new functions
  • Include the Automation in your development
  • Decide if you need to create the automation project in the same or separated project
  • Use CI since the beginning
  • Decide what must to have in the report
  • Choose a good report from the beginning
  • Structure of Scripts. Create or follow a Code Style Guideline

10 – Form a team of automation and equality the level of knowledge. Make some meetings to prepare people in your team to use the tool with wisdom.

I think it’s pretty much this. Sorry guys if I forgot something… If I remember anything else I will update here. Thank you ! Feel free for comments and your opinion always ! See you next week !