Category: QA

Automation Plan and Strategy

Hey guys, in this article I will not talk about automation tools or automation frameworks, but I will talk about how to set the strategy and choose the key scenarios and priorities to build your automation strategy.

 

Who is most interested to have automation tests running ? Quality Assurance team.

 

Why ? Because the regression tests that are always performed by the QA engineers will now be performed by automated scripts. They use the automation to run detailed, repetitive, and data-intensive tests automatically.

It helps to improve software quality and make the most of their always-limited testing resources. Automated testing will shorten your development cycles, avoid repetitive tasks and help improve software quality.

 

Why should we implement an automation test project ?

  • It helps to test faster (Failures will be reported ASAP to be fixed)
  • Allows them to test substantially more code (Able to build deeper tests)
  • Improves test accuracy (Avoid human mistakes)
  • Frees up QA engineers so they can focus on tests that require manual attention and their unique human skills (Exploratory tests)

 

For a manual testing project the cost consuming factors are:

  1. People
  2. Tools – Test/defect management
  3. Infrastructure – environment
  4. Time
  5. Training

 

For an automation project, in addition to the above items it needs also:

  1. Automation tools
  2. Add-in for test management tool integration
  3. Add-in to support AUT (like SAP, oracle etc)
  4. Framework set up
  5. Tool specific training

 

What determines the success of your automation ?

“Are you able to generate a better ROI (Return on Investment) in comparison to the manual route”? – If not immediately, eventually.

 

Test Plan

 

  1. Decide what Test Cases to Automate
  2. Setup a Continuous Integration Box
  3. Select the Right Automated Testing Tool
  4. Divide your Automated Testing Efforts
  5. Create Good, Quality Test Data
  6. Create Automated Tests that are Resistant to Changes in the UI

 

Scope

 

  • Add repetitive tests that run for multiple builds.
  • Add tests that tend to cause human error.
  • Add tests that require multiple data sets.
  • Add frequently used functionality that introduces high risk conditions.
  • Add tests that are impossible to perform manually.
  • Add tests that run on several different hardware or software platforms and configurations.
  • Add tests that take a lot of effort and time when manual testing.
  • Divide your projects in phases and prioritise them.

 

Strategy

 

  • Name convention
  • Reports
  • Select the right automation tool
  • Browsers and its versions, desktop or mobile size (Layout responsive)
  • Platforms (Windows, Mac, Linux)
  • Multiple mobile devices (Portrait and landscape)
  • Continuous integration (Frequency which you will run the tests)
  • Locations for test assets to be stored

 

Resources:

https://smartbear.com/learn/automated-testing/best-practices-for-automation/

http://www.softwaretestinghelp.com/automation-test-palnning/

Config Cucumber and Selenium with Maven on Eclipse

Hello guys,

Today I will post the link for a project on github that contains a basic setup to run your automation on Selenium and Cucumber with Maven on Eclipse.

https://github.com/rafaelaazevedo/cucumber-maven-selenium

Don’t forget to change the path of the chrome driver, features folder and glue package.

The project is on github, so feel free to clone and use as you want. We usually forget about how to do this first configuration as we just need to do in the beginning of the project. Hope this makes you save some time xD

Risk Analysis when descoping scenarios

Hey hey hey guys, today I will post about how to manage a risk analysis and what type of questions you should ask when descoping scenarios from your regression pack.

Regression testing must cover certain conditions in order to be effective.

Regression coverage isn’t a question of the number of cases as much as covering the conditions that

  1. Ensure functionality works
  2. Bugs found have been resolved
  3. A functional area can handle some amount of negative or destructive behaviour

Regression testing that addresses all three aspects should guide your testing efforts more than focusing on the number of cases.

Assessing coverage by the number of test cases is difficult — one case can cover many conditions or one case could provide coverage of only a single condition. If I provided a response by a metric of one case will cover what is needed, you might be underestimating regression testing that such a response would be potentially dangerous or misleading. So addressing the question how I would plan regression testing might be more practical. The questions you should keep in mind are:

  • Is this feature stable ?
  • Is this feature dependent on other less-stable features ?
  • What will happen if this feature fails in a subtle/medium/catastrophic manner ?

 

You can find some techniques to develop a risk analysis:

  • A close reading of the requirements specification, design specifications, user documentation and other items.
  • Pair-wise, you can use a tool to mix the scenarios with different combinations and reduce the scope of your regression pack.
  • A sequence of one-on-one or small-group sessions with the business and technology experts in the company.
  • A tool built from Six Sigma DMAIC model. You can simply write down each function and it’s permutations in any manner that works for your business. The whole development team provides input, while the QA professional manages the document and detail of the information and input gathered.

If you are in a small company, like a startup, my opinion is: You can write the scenarios for you regression pack and ask for some developer to review it. So, you will have a regression pack built with their product code knowledge and you with your global view. QA ends up with factual data on what areas and functions of the application are most critical.

Finally, you put together a regression pack that focuses the higher priority functions while exercising the lesser functions less in depth. The key is to test the higher risk areas first and frequently, while still testing the lesser risk functions more superficially or via rotating test suites based on priority or risk.

It seems too simple, but it works to improve communication between team members while also documenting application functionality.

Stay updated with the last features, your regression pack needs to be fresh with the scenarios. Each new version will contain some new feature as it also can remove an old one. It’s important to try to execute that final test cycle with the freshest view you can.

 

Resources:

http://istqbexamcertification.com/what-is-risk-analysis/

http://www.growingagile.co.za/2012/12/breaking-down-user-stories/

http://searchsoftwarequality.techtarget.com/answer/Regression-testing-How-to-select-test-cases

http://searchsoftwarequality.techtarget.com/answer/Using-risk-analysis-for-regressive-testing-planning

Web automation strategy

Hey guys, today I will post about one of the approaches for web automation. This is a general strategy, which I always try to follow, but depending of the situation you need to adjust and change some priorities.

 

What should come first ?

 

  1. Critical scenarios

    Critical or basic scenarios should come first, so before you even start to think about if the validation message when you type negative time is working as expected, you should check, for instance, if the login is working properly.

    So, choose the basic/critical scenario which is the MVP first and implement it. After all the basic scenarios are implemented you can start to think about the others.Doing this, it will save time when doing the regression tests. Separate your automation in phases like, first phase, basic scenarios, second phase, second priority scenarios and so on. So, you don’t get annoyed by taking so much time trying to implement one complete feature. You do the basic things, let the automation working and move to the second phase.

  2. Maintainability

    How much effort does it take to adjust test scripts to the changed requirements, interfaces, data structure, accounts, platforms etc ? You should worry about this since the start, but we know that not everybody thinks the same and even that you have aligned this in the beginning it is common that people forget.

    So, this is the time that you can go back and change/add little things to improve your automation. For example, if your scenarios are not independent, you can implement some checks and make them run by themselves. This will save time when you run a single scenario.

  3. Browsers compatibility

    This depends of what is the most used browser by your clients. You can start implement your automation for all browsers or you can focus on the most used browser and then you adjust the automation to run on all the other browsers. If you choose to run the automation on all browsers since the beginning you need to be aware of you will spend more time implementing the same scenario. Your automation will be done for all browsers, but all other scenarios will be behind schedule since you need to spend more time on one scenario.

    So, you need to be aware of what is more important, choose the most used and delivery the automation in a short time or take more time to delivery the automation and it will be ready for all browsers already.In my opinion, I prefer to focus on the most used browsers and then implement other browsers in the next phase. When you are implementing one scenario for all browsers, it could be hard to focus and find the issue, since every time you run, it will be running on different browser instances. Same for mobile browsers, this should be done in another step. But again, this depends of the priorities of your project.

  4. Portability

    Is it easy to test your automation on another environment ? This is one of the things you need to be aware of. It could be an improvement task, since you will already have the automation and this can be implemented in the next phase of your automation.

 

Thank you guys ! See you next week !

Resources:

https://www.atlantbh.com/five-important-aspects-of-successful-test-automation-approach-in-agile/

Improve the architecture of your cross-platform automation

Hi guys, today I am going to talk about how to better architect your test code and obtain better cross-platform code reuse.

The idea is that you provide an implementation of page objects for each platform you need to support (e.g. iPhone, iPad, Android phone, Android tablet, mobile web, desktop web,…).

 

Use Page Objects

While originating in web testing, the ideas of POP apply equally well to native mobile. The most obvious benefit of this is abstraction and reuse. If you have several steps needing to navigate to details, the code is reused. Also, callers of this method need not worry about the details (e.g. query and touch) or navigating to this screen.

 

Don’t specify fields in your feature files

Working this way gets you complete reuse of Cucumber features as well as step definitions: the details of interacting with the screen is pushed to page object implementations.

Scenarios should be written like a user would describe them. Beware of scenarios that only describe clicking links and filling in form fields, or of steps that contain code or CSS selectors. This is just another variant of programming, but certainly not a feature description.

Declarative features are vivid, concise and contain highly maintainable steps.

For example:

Scenario: Valid login credentials
  Given I am on login page
  When I enter valid credentials
  Then I should be redirected to the management page

 

Define a page object for each screen

For each screen you want to automate, decide on an interface for a page object class (e.g. like LoginScreen). This means that the calling code, which is usually in a step definition, is independent of platform – hence it can be reused across platforms. For each supported platform, define a class with implementations of the page-object methods.

 

Don’t expose the framework in your step definitions

Use only custom steps, and in each step definition only use page objects and their methods (no direct calls to the framework (Calabash, Selenium, Appium, Robotium).

 

Reuse step definitions

This comes in handy when a step extends another step’s behaviour or defines a superior behaviour that consists of multiple steps. You should try to reuse steps as often as possible.This will improve the maintainability of your app: If you need to change a certain behaviour, you just need to change a single step definition.

For example (Using javascript and protractor):

 this.Given(/^I am on login page$/, function() {
    loginPage.openLoginPage();
 });

 

Thank you guys ! See you next week !

 

Resources:

https://developer.xamarin.com/guides/testcloud/calabash/xplat-best-practices/

https://blog.codeship.com/cucumber-best-practices/

Useful Adb commands for tests

  • Start and stop server to interact with the device:
adb start-server
adb kill-server
  • List of connected devices:
adb devices
  • Print the current log to the console:
adb logcat
  • Pull files from the device:
adb pull [device file location] [local file location]
  • Create a screenshot:
adb shell screencap -p /sdcard/screenshot.png
  • Create a video:
adb shell screenrecord /sdcard/Bug1234Video.mp4
  • Install/Uninstall apps:
adb install com.myAppPackage
adb uninstall com.myAppPackage
  • Send 10.000 random events on the real device. Use the parameter -s (seed) to execute the same commands over and over again:
adb shell monkey -p com.myAppPackage -v 10000 -s 100
  • Start an activity:
adb shell am start -a android.intent.action.VIEW
  • Open the shell:
adb shell
  • Remount the device with the right permissions:
adb remount
  • Remount, delete the app and reboot the tablet:
adb remount && adb shell rm /system/app/MyApp.apk && adb shell reboot

Thank you guys ! See you next week !

Resources: http://adventuresinqa.com/2016/04/19/powerful-adb-commands/

Research – QA process

Hey guys, I sent a form with some questions about the QA process in different corporations and I will post the result here, it’s interesting to know how the majority is working nowadays 🙂

  • Which moment do you create your scenarios (validation tests) ?

Screen Shot 2016-06-04 at 16.45.10

  • 80% – Create after “In conversation” when the ticket is Ready for development 
  • 20% – Create when the ticket is in development
  • 0% – Create when the ticket is Ready for QA

 

 

 

 

 

 

  • As a tester have you been involved in meetings about business rules of a ticket or just the PO ?

Screen Shot 2016-06-04 at 17.16.03

  • 20% – Just the PO is involved and I need to read the tickets to understand the business rules
  • 60% – I take part with devs and after we point the ticket
  • 20% – I take part since the ticket’s creation with the PO

 

 

 

 

  • Do you have a brief explanation or demo about what the ticket is about ?

Screen Shot 2016-06-04 at 17.20.21

  • 0% – I have a demo before test the ticket
  • 0% – PO explains to me what the ticket is about
  • 20% – Dev explains to me what the ticket is about
  • 60% – I have a meeting in the beginning of the sprint to know what the ticket is about
  • 20% – PO and dev explain to me the ticket

 

 

 

 

 

  • When you do the regression tests, do you test the integration between the platforms ?

Screen Shot 2016-06-04 at 17.27.49

  • 40% – I test the integration between the related platforms
  • 0% – Other
  • 60% – I test just the platform which will be released

 

 

 

 

  • Have you ever performed the regression tests before the tickets be resolved ?

Screen Shot 2016-06-04 at 17.31.01

  • 60% – Yes
  • 40% – No

 

 

 

 

 

 

 

  • Do you have a staging environment to test each single ticket ?

Screen Shot 2016-06-04 at 17.33.11

  • 40% – No
  • 60% – Yes

 

 

 

 

 

 

  • Do you have an UAT environment to test the group of tickets of each release ?

Screen Shot 2016-06-04 at 17.35.18

  • 40% – Yes
  • 60% – No

 

 

 

 

 

 

  • When you have tickets to be tested together…

Screen Shot 2016-06-04 at 17.36.57

  • 60% – You are notified by the developer
  • 0% – Other
  • 40% – It’s written on the ticket to be tested together with another(s) ticket(s)

 

 

 

 

Test Automation Obstacles On Demand

Hey guys, today I will post a webinar that I’ve watched last week. It’s about the challenges of test automation:

  • Collaboration
    • Daniel explains that is easy to ask developers to write test automation and for this reason, it’s better to choose the same language of the developers
    • Assign an owner for the digital test coverage but keep the conversation open to all teams
  • Tools
  • Reports
  • Flows
  • Time
  • False negatives
  • Stability
    • Object identification is the key for robust automation
    • Assure proper Error handling
  • Security
    • Networks
    • Orientation
    • Location
    • Notifications
    • Events
    • Device events

Here is the video, but you can have a look on the slides on the link below:

Source: http://info.perfectomobile.com/Overcoming-Test-Automation-Obstacles-On-Demand-Replay.html?mkt_tok=eyJpIjoiTkRKaVlUY3lZbVk0TTJaaiIsInQiOiJZV2w2VWtxMkk0a214d29ETmNvaFpodENQeHllTUcrMkYycU05bEV1WmhIYzQyUTBhS3NoRjA5TFFuTTN6S3NcL1ZPaDQ1YWtBNmRSZDRcLyt4eVdyVk5ES1FBS0MrWVwvSU5WUGZVZ0VwVGI1ST0ifQ%3D%3D