We have started the SpecFlow project in 2009 to provide a tool for .NET developers to perform BDD-style functional testing. SpecFlow uses unit testing frameworks, like NUnit, MsTest or MbUnit to execute the Gherkin scenarios. While this was a pragmatic choice, it has turned out that for executing functional integration tests, these unit test frameworks are not optimal (quite understandable, as they are unit testing frameworks and not integration testing frameworks). While they can execute the tests very well, they lack features like
- (html) report as primary output
- fast integration test execution by parallelization
- advanced execution metrics: execution time, memory usage, memory leaking, benchmarking
- detection of “random failures”
- test variations (e.g. running the same suite for IE and Firefox)
- support for applying different test configurations
- provide fast feedback about failures (prioritize failing tests first, stop after a number of failures, etc.)
In addition to that, we were missing a test runner that treats Gherkin features, scenarios and scenario outlines as first-class citizens.
To address these problems, we have decided to launch a new product that fits into the SpecFlow-SpecLog context. It has been publicly announced today, so it’s here: SpecRun
We have been working for this tool for a while now and have tested it in our projects. It is really promising. It seems that with parallel execution, the local test execution time (that is more critical from the cost perspective, as developers can’t progress while the tests are running) can be dropped to ca. 60%, that is almost half time. This and the smarter feedback about the errors were received very positively by the developers on the selected projects. They were more willing to run the tests locally reducing the long round-trips and they were able to detect the root cause of the errors better.
SpecRun is also using SpecRun for testing itself obviously, and our own dog food tastes pretty good!
SpecRun will be a commercial product (with free versions for OSS projects of course). This was a hard decision. We have a wonderful community around SpecFlow and the support of TechTalk, but even this way we could not afford the costs of building up so many new things. I think this is a good combination. While we believe that SpecRun is a great help and provides quick ROI for the users, it is not the main concern of SpecFlow. With the separation of the two aspects, we can have still our SpecFlow efforts focused on the main concerns (editor, runtime API, documentation) and also provide an extension for the smarter execution experience.
SpecRun is currently in a closed beta stage, but we are planning to have the commercial release soon. Have a look at our website: http://www.specrun.com, stay tuned on our twitter account: @specrun and contact info@specrun.com if you are interested in participating in the beta program.