How to setup Protractor + Cucumber

Hello guys, I will post about how to install protractor and cucumber to test an AngularJS and how to import this project on your Eclipse.

You need JDK and Node.js installed:

– Install node in your mac

brew install node
node --version

– Check the version of you java

java -version

– Git clone this project as an example:

https://github.com/AndrewKeig/protractor-cucumber.git

– Install protractor and cucumber through npm, update and start the webdriver server

npm install -g protractor
npm install -g cucumber
protractor --version
webdriver-manager update
webdriver-manager start

– You will see a message like this on your console:

11:13:08.586 INFO - Selenium Server is up and running

– Run cucumber.js on the protractor-cucumber’s project folder

– If you have the error below, move the features folder to outside the examples folder and open the /features/step_definitions/steps.js and remove one of the “../”:

  return binding.lstat(pathModule._makeLong(path));

                 ^

Error: ENOENT: no such file or directory, lstat '/Users/Documents/workspace/protractor-cucumber/features'

    at Error (native)

    at Object.fs.lstatSync (fs.js:887:18)

    at Object.realpathSync (fs.js:1518:21)

    at Object.expandPathWithExtensions (/usr/local/lib/node_modules/cucumber/lib/cucumber/cli/path_expander.js:14:23)

    at /usr/local/lib/node_modules/cucumber/lib/cucumber/cli/path_expander.js:8:27

    at Array.map (native)

    at Object.expandPathsWithExtensions (/usr/local/lib/node_modules/cucumber/lib/cucumber/cli/path_expander.js:7:31)

    at Object.expandPaths (/usr/local/lib/node_modules/cucumber/lib/cucumber/cli/feature_path_expander.js:9:38)

    at Function.Configuration (/usr/local/lib/node_modules/cucumber/lib/cucumber/cli/configuration.js:21:63)

    at getConfiguration (/usr/local/lib/node_modules/cucumber/lib/cucumber/cli.js:46:38)

Rafaelas-MBP-2:protractor-cucumber rafaelasouza$ cucumber.js

module.js:341

If you want to run on Eclipse:
– Install nodeclipse, a Node.js plugin for Eclipse

npm install -g nodeclipse

nodeclipse -g

– Import this project into your workspace and install Enide on your eclipse.

Thank you guys ! See you tomorrow !

How to get a random number from a list on Jmeter

Hey folks,

So for today I will post the code I am using on Jmeter to get a random index from a list. So, if you have a json with several ids, for example, and you want to choose one of them to use on the next request, you need:

1 – Create a post processor to get the id node from your json response request. As you may know when you use this plugin, jmeter creates automatically the list of variables with the prefix you have chosen and the suffix “__index”. You can see how to use this plugin on my previous post.

2 – Create a Beanshell Sample

3 – Get a random index from this list of ids or you could use this method to get the random id directly. I found easiest get the index r at least it was the first thing which came into my mind.

import java.util.Random;
String idSize = vars.get("ids_matchNr");
int max = Integer.parseInt(idSize);
int min = 1;
int idx =  min + (int) (Math.random() * ((max - min) + 1));
String id = vars.get("ids_" + Integer.toString(idx));

Thank you guys ! See you next week or before 🙂

Setting Webdriver to run tests in a remote computer

Hey guys, today I will post about how can you setup your webdriver project to run in a remote machine.

So, first you need set a machine running the selenium server:

java -jar selenium-server-standalone-2.x.x.jar

You will see a message like this:

15:43:07.541 INFO - RemoteWebDriver instances should connect to: 
http://127.0.0.1:4444/wd/hub

Then in your code, point webdriver.chrome.driver to the location of chromedriver on the remote machine, like the chrome driver. Also, make sure to start up chromedriver there. If you want to run on your local machine point to: 127.0.0.1:4444/wd/hub

 String remoteURL = "http://localhost:9515"; 
 System.setProperty("webdriver.chrome.driver", "/Users/xxxxx/chromedriver");
 WebDriver driver = new RemoteWebDriver(new URL(remoteURL), DesiredCapabi
lities.chrome());
 driver.get("http://www.google.com");
 WebElement element = driver.findElement(By.id("lst-ib"));
 element.sendKeys("Azevedo Rafaela!");
 element.submit();
 driver.quit();

Summarising: You have to install a Selenium Server (a Hub), and register your remote WebDriver to it. Then, your client will talk to the Hub which will find a matching webdriver to execute your test.

Hope this helps, a long time that I don’t code using Selenium!

Resources:

http://stackoverflow.com/questions/8837875/setting-remote-webdriver-to-run-tests-in-a-remote-computer-using-java

http://stackoverflow.com/questions/9542020/using-selenium-2-remotewebdriver-with-chromedriver

http://selenium-python.readthedocs.org/getting-started.html

https://github.com/SeleniumHQ/selenium/wiki/RemoteWebDriver

Mock or don’t mock the server ?

Why mock the server

When you want isolate the system under test to ensure tests run reliably and only fail when there is a genuine error, this avoids tests failing due to irrelevant external changes such as network failure or a server being rebooted / redeployed.

Allows the full range of responses and scenarios to be tested without having to set up and manage a complex test infrastructure. For example increased response delay or dropped connections can increase as load increases on a dependant system. To simulate these types of performance related degradation can be extremely difficult without with out generating a large volume of traffic. If the dependent system is mocked the mock can control the exact response delay or any other characteristics of each response.

Each test can then independently encapsulate the data and logic used for mock services, ensuring each test runs independently. In addition such an approach also reduces the time for a suite of test to complete because tests can run in parallel and do not share data.

Allows development teams to isolated from an unstable, unreliable or volatile web service. This is particularly critical during the initial development phases when the APIs / services are changing frequently and cause development and testing to be blocked

Why not mock the server

Tests can be harder to understand. Instead of just a straightforward usage of your code (e.g. pass in some values to the method under test and check the return result), you need to include extra code to tell the mocks how to behave. Having this extra code detracts from the actual intent of what you’re trying to test, and very often this code is hard to understand if you’re not familiar with the implementation of the production code.

Tests can be harder to maintain. When you tell a mock how to behave, you’re leaking implementation details of your code into your test. When implementation details in your production code change, you’ll need to update your tests to reflect these changes. Tests should typically know little about the code’s implementation, and should focus on testing the code’s public interface.

Tests can provide less assurance that your code is working properly. When you tell a mock how to behave, the only assurance you get with your tests is that your code will work if your mocks behave exactly like your real implementations. This can be very hard to guarantee, and the problem gets worse as your code changes over time, as the behavior of the real implementations is likely to get out of sync with your mocks.

Anti-pattern

Record and replay real dependent service responses, these recordings are typically complex and shared between multiple tests. This, however, introduces unnecessary coupling between tests and breaks the Single responsibility principle, which, states that every context (class, function, variable, etc.) should define a single responsibility, and that responsibility should be entirely encapsulated by the context.

So, my advice is mock the server, you can mock the server and test the integration in the end of the project (End-to-End tests). Sometimes you can’t use a real dependency in a test (e.g. if it’s too slow or talks over the network), but there may better options than using mocks, such as a hermetic local server (e.g. a credit card server that you start up on your machine specifically for the test) or a fake implementation (e.g. an in-memory credit card server)

Thanks guys, see you next week 🙂

Resources:

http://googletesting.blogspot.co.uk/2013/05/testing-on-toilet-dont-overuse-mocks.html

https://blog.8thlight.com/uncle-bob/2014/05/10/WhenToMock.html

http://www.mock-server.com/#why-use-mockserver

https://en.wikipedia.org/wiki/MockServer

How to get specific value/node from a json with Jmeter

Hi guys, today I will post about how to get a value from a json response with Jmeter.

First we need Jmeter with all plugins installed:

brew install jmeter –with-plugins

Let’s use this response for instance. Look that we have a token which is dynamically generated and we need get this value to use in the next request.

Response:

{
   "status": "success",
   "message": "success registering this user",
   "data":    {
      "date": "2015-04-06",
      "id": "1",
      "user": "Rafa",
      "token": "30154dbe350991cf316ec52b8743137b"
   }
}

You can use this site to find the json path and get the expression to use on jmeter:
https://jsonpath.curiousconcept.com

So, we got the expression: $.data.token and now we can create the json path extractor with a default value (when jmeter doesn’t find the expression), jsonpath and the variable (which will contain the value of the expression). See:

Screen Shot 2016-03-02 at 22.38.38

Now, we can use the variable on a header or another request:

Screen Shot 2016-03-02 at 22.44.46

As always, feel free to add any comments !

Cheers. See you next week 🙂

QA Metrics

Hey guys, today I am going to post some metrics for the automation projects. So, let’s start with the percentage automatable, which means how many test cases you can automate and how many you need to test manually.

Percent automatable

PA (%) = ATC/TC

PA = Percent automatable
ATC = Number of test cases automatable
TC = Total number of test cases

As part of an AST effort, the project is either basing its automation on existing manual test procedures, or starting a new automation effort from scratch, some combination, or even just maintaining an AST effort. Whatever the case, a percent automatable metric or the automation index can be determined.

Automation Progress

AP (%) = AA/ATC

AP = Automation progress
AA = Number of test cases automated
ATC = Number of test cases automatable

Automation progress refers to the number of tests that have been automated as a percentage of all automatable test cases. Basically, how well are you doing against the goal of automated testing? The ultimate goal is to automate 100% of the “automatable” test cases. It is useful to track this metric during the various stages of automated testing development.

Test Progress (Manual or automated)

TP = TC/T

TP = Test progress
TC = Number of test cases executed
T = Total number of test cases

A common metric closely associated with the progress of automation, yet not exclusive to automation, is test progress. Test progress can simply be defined as the number of test cases (manual and automated) executed over time.

Percent of Automated Test Coverage

PTC (%) = AC/C

PTC = Percent of automated test coverage
AC = Automation coverage
C = Total coverage (i.e., requirements, units/components, or code coverage)

This metric determines what percentage of test coverage the automated testing is actually achieving. Various degrees of test coverage can be achieved, depending on the project and defined goals. Together with manual test coverage, this metric measures the completeness of the test coverage and can measure how much automation is being executed relative to the total number of tests. Percent of automated test coverage does not indicate anything about the effectiveness of the testing taking place; it is a metric that measures its dimension.

Defect Density

DD = D/SS

DD = Defect density
D = Number of known defects
SS = Size of software entity

Defect density is another well-known metric that can be used for determining an area to automate. If a component requires a lot of retesting because the defect density is very high, it might lend itself perfectly to automated testing. Defect density is a measure of the total known defects divided by the size of the software entity being measured. For example, if there is a high defect density in a specific functionality, it is important to conduct a causal analysis. Is this functionality very complex, and therefore is it to be expected that the defect density would be high? Is there a problem with the design or implementation of the functionality? Were the wrong (or not enough) resources assigned to the functionality, because an inaccurate risk had been assigned to it and the complexity was not understood?

Defect Trend Analysis

DTA = D/TPE

DTA = Defect trend analysis
D = Number of known defects
TPE = Number of test procedures executed over time

Another useful testing metric in general is defect trend analysis.

Defect Removal Efficiency

DRE (%) = DT/DT+DA

DRE = Defect removal efficiency
DT = Number of defects found during testing
DA = Number of defects found after delivery

DRE is used to determine the effectiveness of defect removal efforts. It is also an indirect measurement of product quality. The higher the percentage, the greater the potential positive impact on the quality of the product. This is because it represents the timely identification and removal of defects at any particular phase.

Automation Development
Number (or %) of test cases feasible to automate out of all selected test cases – You can even replace test cases by steps or expected results for a more granular analysis.
Number (or %) of test cases automated out of all test cases feasible to automate – As above, you can replace test cases by steps or expected results.
Average effort spent to automate one test case – You can create a trend of this average effort over the duration of the automation exercise.
% Defects discovered in unit testing/ reviews/ integration of all discovered defects in the automated test scripts

Automation Execution
Number (or %) of automated test scripts executed out of all automated test scripts
Number (or %) of automated test scripts that passed of all executed scripts
Average time to execute an automated test script – Alternately, you can map test cases to automated test scripts and use the Average time to execute one test case.
Average time to analyze automated testing results per script
Defects discovered by automated test execution – As common, you can divide this by severity/ priority/ component and so on.

References:

http://www.methodsandtools.com/archive/archive.php?id=94

2015 in review

The WordPress.com stats helper monkeys prepared a 2015 annual report for this blog.

Here’s an excerpt:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 33,000 times in 2015. If it were a concert at Sydney Opera House, it would take about 12 sold-out performances for that many people to see it.

Click here to see the complete report.

BDD Best practices

Here I am again talking about BDD haha 😀

I am trying to implement the best practices of BDD in my company, it’s quite hard because most of them are developers and they have their own way to think which is different for Business and QA professionals. Other day I was discussing with a guy who had read a post about BDD and suddenly he entitled himself as an expert ! Really, this is the most common attitude you will find in IT atmosphere.

57221765

So, the structure is exactly like this and some people like to write the Given step in the past, but this is the thing, you don’t really need to write the first step in the past, some cases you can write in the present tense.

Like:

Given I have xxx…
When I do xxx…
Then I should see xxxx…

The first step is a situation you have now like I am logged on the system, you don’t need write always in the past, this is not a rule. You need to write something is evident, this is the aim. So, if you want write in the past the Given, you will have something like:

Given I have logged xxx…
When I do xxx…
Then I should see xxx…

Except the Given that you can choose what is the best option for your scenarios, the When and Then you need to follow the right tenses, When in present Tense and Then in the future tense.

BDD structure:

The features must have .feature as the extension of the file and you need put the features in the correct path, if you don’t do that Cucumber won’t work. If you want to run with Jbehave you don’t need follow the right structure, you can change this on the pom file if you are using Maven.

Where should feature files be kept

Some people don’t like to keep them on git with the project, they would like to see the scenarios on JIRA, but you will need to update and maintain in both of the places, for this reason I always put on github.

How should we write feature files

There are generally two ways of writing feature files – Imperative and Declarative

Imperative style of writing a feature file, is very verbose, contains low level details and too much information.

Pros: person reading the feature file can follow the step-by-step

Cons: Because of too much detail, the reader can lose the point of the story and the tests. The feature file becomes too big, difficult to maintain and likely to fail due to UI updates.

Declarative style of writing a feature file is concise and to the point, contains only relevant information about the story.

Pros: The declarative style is more readable as it contains less steps in the scenario. The reader can easily understand the scope of the test and quickly identify if any key elements are missing.

Independence of Scenarios

Scenarios should be independent of one another. This means that each and every scenario is stand-alone and should not refer to or depend on functionalities/conditions in other scenarios.

Redundancy and Refactoring

Avoid repetitions! Repetitions or identical steps indicate the need forrefactoring. Refactoring denotes that a scenario should be split into multiple ones while the original meaning is preserved.
How many Givens, Whens and Thens (including Ands andButs) should you use? As a rule of thumb, there should be amaximum of 3 to 4 consecutive Givens and Thens. Otherwise this indicates a need for refactoring. Whens should be used very sparingly! Usually only include one When.
Generally, favour many simple scenarios over few complex ones.

Resources:

http://www.testingexcellence.com/bdd-guidelines-best-practices/

https://blog.grandcentrix.net/gherkin-guidelines-and-best-practices/

Why do we need to have a QA separated environment ?

If QA is using a dev environment, that environment is likely changing. It is hard to control the state of the environment with multiple people working against it.
Typically, QA environments mimic more closely production environments than do development environments. This helps ensure functionality is valid in an environment that is more production-like.
Developers tend to have lots of tools and things running in their environment that could affect QA validation.
You usually setup a separate QA environment, because you want to give the testers an isolated environment on which to test, so that developers and testers can work at the same time.
This allows reporting on a common revision so developers know whether particular issues found by testers has already been corrected in the development code.

Face the distinct possibility of releasing critical defects to customers because it’s not testing in a real-world environment.
The test team won’t see the issues when the environment is not the same because the playing field, so to speak, is not even.
Testing on a qa environment provides a more accurate measure of performance capacity and functional correctness
As Web applications become more mission-critical for customers, it becomes increasingly important to test on environments that exactly mimic production because it’s production where customers use your application

Resources: http://stackoverflow.com/questions/2777283/why-should-qa-have-their-own-qa-environment-what-are-the-pros-and-cons

http://searchsoftwarequality.techtarget.com/tip/A-good-QA-team-needs-a-proper-software-staging-environment-for-testing

Polymorphic Step Definitions with Cucumber-jvm

Just a quick snippet that I am using to create Polymorphic Step Definitions with cucumber-jvm.

First you need to import picocontainer library.

When you can use it ? It’s useful if you have for example Integration and UI scenarios in the same project. For example, for mobile tests I am using robotium, but for the server tests I am using HTTPClient Apache.

Example:
import cucumber.runtime.java.picocontainer.PicoFactory;

public class TestsFactory extends PicoFactory {
public TestsFactory() {
if ("integration".equals(System.getProperty("com.rsouza.androidTest.integration"))) {
addClass(SupportIntegrationSteps.class);
addClass(SupportIntegration.class);
}
else{
addClass(SupportUI.class);
}
}
}
You just need define Cucumber to use cucumber.api.java.ObjectFactory system property.

Create a cucumber.properties file on your classpath and if you are using Maven, this should be src/test/resources/cucumber.properties. If you are using gradle this should be src/androidTest/res/cucumber.properties
Add the following line: cucumber.api.java.ObjectFactory = my.features.CustomPicoFactory

Thank you guys ! See you next week 🙂

Resources: https://cucumber.io/blog/2015/07/08/polymorphic-step-definitions