Automation Plan and Strategy

Hey guys, in this article I will not talk about automation tools or automation frameworks, but I will talk about how to set the strategy and choose the key scenarios and priorities to build your automation strategy.

 

Who is most interested to have automation tests running ? Quality Assurance team.

 

Why ? Because the regression tests that are always performed by the QA engineers will now be performed by automated scripts. They use the automation to run detailed, repetitive, and data-intensive tests automatically.

It helps to improve software quality and make the most of their always-limited testing resources. Automated testing will shorten your development cycles, avoid repetitive tasks and help improve software quality.

 

Why should we implement an automation test project ?

  • It helps to test faster (Failures will be reported ASAP to be fixed)
  • Allows them to test substantially more code (Able to build deeper tests)
  • Improves test accuracy (Avoid human mistakes)
  • Frees up QA engineers so they can focus on tests that require manual attention and their unique human skills (Exploratory tests)

 

For a manual testing project the cost consuming factors are:

  1. People
  2. Tools – Test/defect management
  3. Infrastructure – environment
  4. Time
  5. Training

 

For an automation project, in addition to the above items it needs also:

  1. Automation tools
  2. Add-in for test management tool integration
  3. Add-in to support AUT (like SAP, oracle etc)
  4. Framework set up
  5. Tool specific training

 

What determines the success of your automation ?

“Are you able to generate a better ROI (Return on Investment) in comparison to the manual route”? – If not immediately, eventually.

 

Test Plan

 

  1. Decide what Test Cases to Automate
  2. Setup a Continuous Integration Box
  3. Select the Right Automated Testing Tool
  4. Divide your Automated Testing Efforts
  5. Create Good, Quality Test Data
  6. Create Automated Tests that are Resistant to Changes in the UI

 

Scope

 

  • Add repetitive tests that run for multiple builds.
  • Add tests that tend to cause human error.
  • Add tests that require multiple data sets.
  • Add frequently used functionality that introduces high risk conditions.
  • Add tests that are impossible to perform manually.
  • Add tests that run on several different hardware or software platforms and configurations.
  • Add tests that take a lot of effort and time when manual testing.
  • Divide your projects in phases and prioritise them.

 

Strategy

 

  • Name convention
  • Reports
  • Select the right automation tool
  • Browsers and its versions, desktop or mobile size (Layout responsive)
  • Platforms (Windows, Mac, Linux)
  • Multiple mobile devices (Portrait and landscape)
  • Continuous integration (Frequency which you will run the tests)
  • Locations for test assets to be stored

 

Resources:

https://smartbear.com/learn/automated-testing/best-practices-for-automation/

http://www.softwaretestinghelp.com/automation-test-palnning/

Headless browsers vs Real browsers

Hi guys, I have noticed that many developers doesn’t know the purpose of the headless browsers and when you can consider to use them. Everything depends of your project and what is the purpose of your tests.

Headless browser is like any other regular browser, but without the GUI (Graphical User Interface). The networking component, javascript interpreter, rendering and layout engines are still present. They are particularly useful for testing web pages as they are able to render and understand HTML the same way a browser would, including styling elements such as page layout, colour, font selection and execution of JavaScript and AJAX which are usually not available when using other testing methods.

 

You can use headless browser for:

  • Test automation in web applications (Best performance as you don’t have a GUI).
  • Taking screenshots of web pages.
  • Running automated tests for JavaScript libraries.
  • Scraping web sites for data.
  • Automating interaction of web pages.
  • Perform DDOS attacks on web sites.
  • Increase advertisement impressions.
  • Automate web sites in unintended ways e.g. for Credential stuffing.

 

Headless browsers became a known technology back in 2009 when Google revealed it was opened to using one to parse and index AJAX-based websites.

Headless browsers have been known to be used in DDOS attacks, brute-force attacks, and also for falsely increasing ad revenue by faking page loads and user interactions.

 

Why not to use them:

  • The gain in the speed is not that big compared with real browsers, depending of how many tests you have.
  • Hard to debug.
  • It is not a real browser, so in the end, it is not the same experience your users have.
  • Headless browser can have issues of its own that do not show up in other browsers (a bug found using a headless browser like PhantomJS may actually not be a bug).

 

Tradeoffs:

  • Speed vs Consistency.
  • More specific code vs more general code.

 

Hope I have helped you guys to understand when it is possible to use them and when it is not, since it depends of your project and your requirements. You can judge what is the best option for your project in the moment.

Thank you guys, see you later !

 

Resources:

http://news.softpedia.com/news/what-is-a-headless-browser-and-what-s-it-good-for-485162.shtml#ixzz4Ntp2UsDK

http://blog.arhg.net/2009/10/what-is-headless-browser.html

http://phantomjs.org/

http://www.ecommercetimes.com/story/80194.html

http://www.business2community.com/tech-gadgets/headless-browser-botnet-used-150-hour-ddos-attack-0688767#XhLlg0ZU610scl9l.97

http://www.itproportal.com/2014/04/01/headless-browsers-legitimate-software-enables-attack/

https://webmasters.googleblog.com/2009/10/proposal-for-making-ajax-crawlable.html

Real vs Headless Browsers for Automated Acceptance Tests

 

Hooks vs Backgrounds (Cucumber)

Sometimes you need some pre conditions to run your scenario or a group of scenarios sharing the same steps repeatedly. You can use background or hooks to setup these conditions. How to know what is the best to use ? Well, depends of the case. So today, I will give some examples with best practices when you should use background and when you should use hooks.

Step definition files have a corresponding method available in the before(condition) do . . .method, which has however a matching after(condition) do . . . method as well.

 

Use Background only to setup a pre-condition that the user needs to know

 

If it is not a trivial information to the user, set it up in the implementation (hooks), not in the test steps. Remember feature files should focus on What, and not How. These should be high level steps. We want to keep this simple. So, for this reason you avoid give too many details like this type of steps: “When I press the button”.

 

When using hooks

 

You can use hooks to run before/after each scenario, a group of scenarios according to the tags, all the scenarios in a feature or all the scenarios of your project. They will run before the first step of your scenario, like the background, but it won’t need any step in your feature file.

 

@Before and @After each scenario

Similar to JUnit @Before and @After tagging a method with either of these will cause the method to run before or after each scenario runs. Common functionality like starting or stop browsers are nice to place in these hooks. They reduce the number of common test steps in each scenario. Before hooks will be run before the first step of each scenario. They will run in the same order of which they are registered. After hooks will be run after the last step of each scenario, even when there are failing, undefined, pending or skipped steps.

You may want to finish the tests after the first failure (could be useful in some cases like Continuous Integration when fast feedback is important), for those cases add the command (ruby) in your hook:

  Cucumber.wants_to_quit = true if s.failed?

 

@AfterStep

You have also the possibility to create an after step hook and add for example a take screenshot action. This hook will run after each step of you scenario and you can also filter for certain scenarios using tags. This is only available for Ruby language at the moment, not for Java.

 

@Around

Around hooks will run “around” a scenario. This can be used to wrap the execution of a scenario in a block. The Around hook receives a scenario object and a block (Proc) object. The scenario will be executed when you invoke block.call.

The following example (ruby) will cause scenarios tagged with @fast to fail if the execution takes longer than 0.5 seconds:

Around('@fast') do |scenario, block|
  Timeout.timeout(0.5) do
    block.call
  end
end

 

Tagged hooks

You can filter what are the scenarios that will run this hook every time before start the scenario or after the scenario ends. The condition which enables the before/after block is the tag (false or nil). Tagged hooks can have multiple tags, and follow similar tagging AND/OR rules that the runner does. You can OR and AND tags in much the same way as you can when running Cucumber from the command line.
e.x. @Before(‘@mobile’, ‘Ëś@login’) for tests needing a mobile browser launched and are not tagged as login

e.x. @Before(‘@mobile, Ëś@login’) for tests needing a mobile browser launched or are not tagged as login

 

@Before all scenarios

If you have a hook you only want to run once before all the scenarios, use a global variable. Example (ruby):

Before do 
  $dunit ||= false # define a variable before we can reference its value
  return $dunit if $dunit                  # bail if $dunit TRUE
  step "run the really slow log in method" # otherwise do it.
  $dunit = true                            # don't do it again.
end

 

@AfterConfiguration

You may also provide an AfterConfiguration hook that will be run after Cucumber has been configured. This hook will run only once; after support has been loaded but before features are loaded. You can use this hook to extend Cucumber, for example you could affect how features are loaded or register custom formatters programatically.

 

 

When using background

 

Short Backgrounds

When using background keep it as short as possible. These steps won’t be written out each time the user reads the scenario, so it’s best to have something simple that the user can remember while reading through your feature file.

 

Vigorous Backgrounds

Similar to the above, since these steps won’t be listed with each scenario, the more vivid, the test step is, the easier time the user will have remembering it.

 

Short Feature files

It’s best to keep these feature files smaller, so that the Background information is more readily available. The general rule of thumb is to keep the file small enough to still see the Background test steps at the top of page when reading any scenario.

 

These two methods are powerful tools, but be aware to not use them excessively.

 

Resources:

https://github.com/cucumber/cucumber/wiki/Hooks

https://github.com/cucumber/cucumber/wiki/Cucumber-Backgrounder

https://seleniumbycharan.wordpress.com/2015/08/25/use-of-background-hooks-tags-in-cucumber-jvm/

Open different browsers with C# and Selenium

Hey guys, I am going to post some snippets to run a test in different browsers with C# and selenium.

Import:

using System;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.IE;
using OpenQA.Selenium.PhantomJS;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using NUnit.Framework;
using OpenQA.Selenium.Support.PageObjects;
using Assert = NUnit.Framework.Assert;

 

Driver Utils class: 

Just the beginning of the class declaring the WebDriver and the TxtSearch field, which will be used for all the below functions to open a specific browser.

namespace Helpers
{
 [TestClass]
 public class DriverUtils
 {
 IWebDriver webDriver;

 //Declaring search element
 [FindsBy(How = How.Name, Using = "q")]
 public IWebElement TxtSearch { get; set; }

 

Open Chrome:

 [Test]
 public void Chrome_Browser()
 {
 webDriver = new ChromeDriver();
 AssertSearchElement();
 }

 

Open Chrome Mobile Emulator:

 [Test]
 public void Chrome_Mobile_Emulator_Browser()
 {
 ChromeOptions chromeCapabilities = new ChromeOptions();
 chromeCapabilities.EnableMobileEmulation("Google Nexus 5");
 webDriver = new ChromeDriver(chromeCapabilities);
 AssertSearchElement();
 }

 

Open Chrome Capabilities starts Maximized:

 [Test]
 public void Chrome_Capabiblities_Browser()
 {
 ChromeOptions chromeCapabilities = new ChromeOptions();
 chromeCapabilities.AddArgument("start-maximized");
 webDriver = new ChromeDriver(chromeCapabilities);
 AssertSearchElement();
}

 

Open Firefox:

[Test]
 public void Firefox_Browser()
 {
 webDriver = new FirefoxDriver();
 AssertSearchElement();
}

 

Open Firefox with Profile:

[Test]
 public void Firefox_Profile_Browser()
 {
 var profile = new FirefoxProfile();
 profile.SetPreference("browser.startup.homepage", 
"https://www.azevedorafaela.wordpress.com/");
 webDriver = new FirefoxDriver(profile);
 AssertSearchElement();
}

 

Open PhantomJS:

[Test]
 public void PhantomJS_Browser()
 {
 webDriver = new PhantomJSDriver();
 AssertSearchElement();
}

 

Open PhantomJS with Capabilities:

[Test]
 public void PhantomJS_Capabilities_Browser()
 {
 var options = new PhantomJSOptions();
 options.AddAdditionalCapability("phantomjs.page.settings.userAgent", 
"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/40.0.2214.94 Safari/537.36");
 webDriver = new PhantomJSDriver(options);
 AssertSearchElement();
}

 

Open Internet Explorer:

 [Test]
 public void IE_Browser()
 {
 webDriver = new InternetExplorerDriver();
 AssertSearchElement();
}

 

Open Internet Explorer with Options:

[Test]
 public void IE_Options_Browser()
 {
 var options = new InternetExplorerOptions
 {
 IgnoreZoomLevel = true 
 };

 webDriver = new InternetExplorerDriver(options);
 AssertSearchElement();
}

 

Assert Search Element is Displayed:

public void AssertSearchElement()
 {
 webDriver.Navigate().GoToUrl("https://www.google.com");

 PageFactory.InitElements(webDriver, this);
 Assert.IsTrue(TxtSearch.Displayed);
}

 

Close/Quit Browser and driver function:

[TearDown]
public void CloseDriver(IWebDriver driver)
  {
  driver.Close();
  driver.Quit();
  }
 }
}

 

See you again in the end of this week, hopefully I will have the complete flow for a BDD scenario with C# and Selenium.

 

Appium, how to install and inspect elements on Android ?

  • Install brew:
    ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/
    install/master/install)"
    

 

  • Install Node.js: brew install node

 

  • Install XCode

 

  • Install XCode Command line

 

  • Install appium. app

 

  • Install android jdk

 

  • Install java

 

  • Export $HOME

 

  • Export $JAVA_HOME – If you are using Mac, your java will be at: /Library/Java/JavaVirtualMachines/jdkx.x.x.jdk/Contents/Home

 

  • Export $ANDROID_HOME – If you have android studio (and on Mac), your jdk will be point to /Library/Android/sdk

 

  • Export $PATH: $PATH:$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools

 

  • Create an emulator with adb

 

 

Be sure you have an android emulator created, I normally use genymotion.

 

Inspect this app with appium

 

screen-shot-2016-09-24-at-13-04-41

 

  • Run Appium Doctor and be sure everything is setup:

screen-shot-2016-09-24-at-13-15-01

  • Click on General Settings button and fill the form like this example:

screen-shot-2016-09-24-at-13-04-55

  • Click on Android Settings buttons and fill the form like this example:


screen-shot-2016-09-24-at-13-04-26

  • Open the emulator with adb or genymotion and click to Launch:

screen-shot-2016-09-24-at-13-23-20

  • After this you can click on Inspector button, which will open the app and let you inspect the elements

screen-shot-2016-09-24-at-13-22-40

 

Easy peasy 🙂

Thank you guys, see you next weekend !

Config Cucumber and Selenium with Maven on Eclipse

Hello guys,

Today I will post the link for a project on github that contains a basic setup to run your automation on Selenium and Cucumber with Maven on Eclipse.

https://github.com/rafaelaazevedo/cucumber-maven-selenium

Don’t forget to change the path of the chrome driver, features folder and glue package.

The project is on github, so feel free to clone and use as you want. We usually forget about how to do this first configuration as we just need to do in the beginning of the project. Hope this makes you save some time xD

Risk Analysis when descoping scenarios

Hey hey hey guys, today I will post about how to manage a risk analysis and what type of questions you should ask when descoping scenarios from your regression pack.

Regression testing must cover certain conditions in order to be effective.

Regression coverage isn’t a question of the number of cases as much as covering the conditions that

  1. Ensure functionality works
  2. Bugs found have been resolved
  3. A functional area can handle some amount of negative or destructive behaviour

Regression testing that addresses all three aspects should guide your testing efforts more than focusing on the number of cases.

Assessing coverage by the number of test cases is difficult — one case can cover many conditions or one case could provide coverage of only a single condition. If I provided a response by a metric of one case will cover what is needed, you might be underestimating regression testing that such a response would be potentially dangerous or misleading. So addressing the question how I would plan regression testing might be more practical. The questions you should keep in mind are:

  • Is this feature stable ?
  • Is this feature dependent on other less-stable features ?
  • What will happen if this feature fails in a subtle/medium/catastrophic manner ?

 

You can find some techniques to develop a risk analysis:

  • A close reading of the requirements specification, design specifications, user documentation and other items.
  • Pair-wise, you can use a tool to mix the scenarios with different combinations and reduce the scope of your regression pack.
  • A sequence of one-on-one or small-group sessions with the business and technology experts in the company.
  • A tool built from Six Sigma DMAIC model. You can simply write down each function and it’s permutations in any manner that works for your business. The whole development team provides input, while the QA professional manages the document and detail of the information and input gathered.

If you are in a small company, like a startup, my opinion is: You can write the scenarios for you regression pack and ask for some developer to review it. So, you will have a regression pack built with their product code knowledge and you with your global view. QA ends up with factual data on what areas and functions of the application are most critical.

Finally, you put together a regression pack that focuses the higher priority functions while exercising the lesser functions less in depth. The key is to test the higher risk areas first and frequently, while still testing the lesser risk functions more superficially or via rotating test suites based on priority or risk.

It seems too simple, but it works to improve communication between team members while also documenting application functionality.

Stay updated with the last features, your regression pack needs to be fresh with the scenarios. Each new version will contain some new feature as it also can remove an old one. It’s important to try to execute that final test cycle with the freshest view you can.

 

Resources:

http://istqbexamcertification.com/what-is-risk-analysis/

http://www.growingagile.co.za/2012/12/breaking-down-user-stories/

http://searchsoftwarequality.techtarget.com/answer/Regression-testing-How-to-select-test-cases

http://searchsoftwarequality.techtarget.com/answer/Using-risk-analysis-for-regressive-testing-planning

Web automation strategy

Hey guys, today I will post about one of the approaches for web automation. This is a general strategy, which I always try to follow, but depending of the situation you need to adjust and change some priorities.

 

What should come first ?

 

  1. Critical scenarios

    Critical or basic scenarios should come first, so before you even start to think about if the validation message when you type negative time is working as expected, you should check, for instance, if the login is working properly.

    So, choose the basic/critical scenario which is the MVP first and implement it. After all the basic scenarios are implemented you can start to think about the others.Doing this, it will save time when doing the regression tests. Separate your automation in phases like, first phase, basic scenarios, second phase, second priority scenarios and so on. So, you don’t get annoyed by taking so much time trying to implement one complete feature. You do the basic things, let the automation working and move to the second phase.

  2. Maintainability

    How much effort does it take to adjust test scripts to the changed requirements, interfaces, data structure, accounts, platforms etc ? You should worry about this since the start, but we know that not everybody thinks the same and even that you have aligned this in the beginning it is common that people forget.

    So, this is the time that you can go back and change/add little things to improve your automation. For example, if your scenarios are not independent, you can implement some checks and make them run by themselves. This will save time when you run a single scenario.

  3. Browsers compatibility

    This depends of what is the most used browser by your clients. You can start implement your automation for all browsers or you can focus on the most used browser and then you adjust the automation to run on all the other browsers. If you choose to run the automation on all browsers since the beginning you need to be aware of you will spend more time implementing the same scenario. Your automation will be done for all browsers, but all other scenarios will be behind schedule since you need to spend more time on one scenario.

    So, you need to be aware of what is more important, choose the most used and delivery the automation in a short time or take more time to delivery the automation and it will be ready for all browsers already.In my opinion, I prefer to focus on the most used browsers and then implement other browsers in the next phase. When you are implementing one scenario for all browsers, it could be hard to focus and find the issue, since every time you run, it will be running on different browser instances. Same for mobile browsers, this should be done in another step. But again, this depends of the priorities of your project.

  4. Portability

    Is it easy to test your automation on another environment ? This is one of the things you need to be aware of. It could be an improvement task, since you will already have the automation and this can be implemented in the next phase of your automation.

 

Thank you guys ! See you next week !

Resources:

https://www.atlantbh.com/five-important-aspects-of-successful-test-automation-approach-in-agile/

Improve the architecture of your cross-platform automation

Hi guys, today I am going to talk about how to better architect your test code and obtain better cross-platform code reuse.

The idea is that you provide an implementation of page objects for each platform you need to support (e.g. iPhone, iPad, Android phone, Android tablet, mobile web, desktop web,…).

 

Use Page Objects

While originating in web testing, the ideas of POP apply equally well to native mobile. The most obvious benefit of this is abstraction and reuse. If you have several steps needing to navigate to details, the code is reused. Also, callers of this method need not worry about the details (e.g. query and touch) or navigating to this screen.

 

Don’t specify fields in your feature files

Working this way gets you complete reuse of Cucumber features as well as step definitions: the details of interacting with the screen is pushed to page object implementations.

Scenarios should be written like a user would describe them. Beware of scenarios that only describe clicking links and filling in form fields, or of steps that contain code or CSS selectors. This is just another variant of programming, but certainly not a feature description.

Declarative features are vivid, concise and contain highly maintainable steps.

For example:

Scenario: Valid login credentials
  Given I am on login page
  When I enter valid credentials
  Then I should be redirected to the management page

 

Define a page object for each screen

For each screen you want to automate, decide on an interface for a page object class (e.g. like LoginScreen). This means that the calling code, which is usually in a step definition, is independent of platform – hence it can be reused across platforms. For each supported platform, define a class with implementations of the page-object methods.

 

Don’t expose the framework in your step definitions

Use only custom steps, and in each step definition only use page objects and their methods (no direct calls to the framework (Calabash, Selenium, Appium, Robotium).

 

Reuse step definitions

This comes in handy when a step extends another step’s behaviour or defines a superior behaviour that consists of multiple steps. You should try to reuse steps as often as possible.This will improve the maintainability of your app: If you need to change a certain behaviour, you just need to change a single step definition.

For example (Using javascript and protractor):

 this.Given(/^I am on login page$/, function() {
    loginPage.openLoginPage();
 });

 

Thank you guys ! See you next week !

 

Resources:

https://developer.xamarin.com/guides/testcloud/calabash/xplat-best-practices/

https://blog.codeship.com/cucumber-best-practices/