Hi guys, I will post a research about the BDD engines, it is a bit old (2012), but despite this I found very interesting. I have summarised the relevant informations. The actual language to write tests with BDD is called Gherkin. And it has different implementations adopted to different programming language, the most famous:
– Cucumber for Ruby
– JBehave for Java
– NBehave and SpecFlow for C#
– Freshen for Python Behat for PHP
All of them have some common set of supported features but there’re some restrictions and abilities specific to the actual engine. So, we will collect useful features for each listed above engine and present it in some comparable form. Key features to be mentioned are:
- Documentation availability
- Flexibility in passing parameters
- Auto-complete
- Steps, scenario and feature scoping
- Complex steps
- Hooks and pre-conditions
- Binding to code
- Formatting flexibility
- Built-in reports
- Input data sources support
| Grade | Criteria |
|---|---|
| 0 | No support at all |
| 1 | Functionality exists but with serious restrictions |
| 2 | Major functionality exists |
| 3 | Full-featured support |
Documentation Availability
- Cucumber:
- Freshen – honestly speaking I didn’t find any specialized resourse dedicated to freshen only. Most likely it can be discussed in the more general forums dedicated to BDD in general.
- JBehave:
- JBehave – BDD Tutorial Site
- SpecFlow:
- SpecFlow: Pragmatic BDD for .NET
- SpecFlow Tutorial Site
- Behat:
- Behat/Mink Users LinkedIn group
With this sources to look the documentation of each engine, we can evaluate the criteria and the grades:
| GRADE | CRITERIA |
|---|---|
| 1 | Documentation is available in general (it makes grade 1 at once) |
| 2 | Every feature is described and has examples (if it fits it makes grade 2) |
| 3 | There’re additional well-grown resources (forums, blogs, user groups) where we can find additional information about the engine |
At the end we have:
| Engine | Documentation availability |
|---|---|
| Cucumber | 3 |
| Freshen | 2 |
| JBehave | 3 |
| NBehave | 1 |
| SpecFlow | 3 |
| Behat | 3 |
Other grades:
Flexibility in passing parameters:
| Engine | Regular expressions support | Tables support | Multi-line input support | Extra features |
|---|---|---|---|---|
| Cucumber | 3 | 3 | 3 | 0 |
| Freshen | 3 | 3 | 3 | 0 |
| JBehave | 2 | 3 | 0 | 3 |
| NBehave | 2 | 3 | 0 | 2 |
| SpecFlow | 2 | 3 | 3 | 2 |
| Behat | 3 | 3 | 3 | 0 |
Auto-Complete:
| Engine | Auto-complete support |
|---|---|
| Cucumber | 1 |
| Freshen | 1 |
| JBehave | 1 |
| NBehave | 1 |
| SpecFlow | 3 |
| Behat | 1 |
Step Scenario and feature scoping:
| Engine | Tagging support | Scoped steps support |
|---|---|---|
| Cucumber | 3 | 0 |
| Freshen | 3 | 0 |
| JBehave | 3 | 1 |
| NBehave | 0 | 0 |
| SpecFlow | 3 | 3 |
| Behat | 3 | 0 |
Complex Steps:
| Engine | Composite steps |
|---|---|
| Cucumber | 3 |
| Freshen | 3 |
| JBehave | 3 |
| NBehave | 2 |
| SpecFlow | 2 |
| Behat | 3 |
Hooks and pre-conditions:
| Engine | Backgrounds | Hooks |
|---|---|---|
| Cucumber | 3 | 3 |
| Freshen | 3 | 3 |
| JBehave | 1 | 1 |
| NBehave | 0 | 1 |
| SpecFlow | 3 | 3 |
| Behat | 3 | 3 |
Binding Code:
| Engine | Binding to code |
|---|---|
| Cucumber | 2 |
| Freshen | 3 |
| JBehave | 3 |
| NBehave | 3 |
| SpecFlow | 3 |
| Behat | 2 |
Formatting Flexibility:
| Engine | Formatting |
|---|---|
| Cucumber | 3 |
| Freshen | 3 |
| JBehave | 1 |
| NBehave | 3 |
| SpecFlow | 3 |
| Behat | 3 |
Reports:
| Engine | Built-in reports |
|---|---|
| Cucumber | 3 |
| Freshen | 2 |
| JBehave | 2 |
| NBehave | 2 |
| SpecFlow | 2 |
| Behat | 3 |
Input data sources support:
| Engine | External Data | Inclusions |
|---|---|---|
| Cucumber | 0 | 0 |
| Freshen | 2 | 3 |
| JBehave | 3 | 2 |
| NBehave | 0 | 0 |
| SpecFlow | 0 | 0 |
| Behat | 0 | 0 |
Conclusion:
Source: http://mkolisnyk.blogspot.co.uk/2012/06/gherkin-bdd-engines-comparison.html
Rafaela Azevedo 