Gáspár Nagy on software

coach, trainer and bdd addict, creator of SpecFlow; owner of Spec Solutions

SpecRun – because integration tests are not unit tests

by Gáspár on September 2, 2011

We have started the SpecFlow project in 2009 to provide a tool for .NET developers to perform BDD-style functional testing. SpecFlow uses unit testing frameworks, like NUnit, MsTest or MbUnit to execute the Gherkin scenarios. While this was a pragmatic choice, it has turned out that for executing functional integration tests, these unit test frameworks are not optimal (quite understandable, as they are unit testing frameworks and not integration testing frameworks). While they can execute the tests very well, they lack features like

  • (html) report as primary output
  • fast integration test execution by parallelization
  • advanced execution metrics: execution time, memory usage, memory leaking, benchmarking
  • detection of “random failures”
  • test variations (e.g. running the same suite for IE and Firefox)
  • support for applying different test configurations
  • provide fast feedback about failures (prioritize failing tests first, stop after a number of failures, etc.)

In addition to that, we were missing a test runner that treats Gherkin features, scenarios and scenario outlines as first-class citizens.

To address these problems, we have decided to launch a new product that fits into the SpecFlow-SpecLog context. It has been publicly announced today, so it’s here: SpecRun

specrun_logo

We have been working for this tool for a while now and have tested it in our projects. It is really promising. It seems that with parallel execution, the local test execution time (that is more critical from the cost perspective, as developers can’t progress while the tests are running) can be dropped to ca. 60%, that is almost half time. This and the smarter feedback about the errors were received very positively by the developers on the selected projects. They were more willing to run the tests locally reducing the long round-trips and they were able to detect the root cause of the errors better.

SpecRun is also using SpecRun for testing itself obviously, and our own dog food tastes pretty good!

SpecRun will be a commercial product (with free versions for OSS projects of course). This was a hard decision. We have a wonderful community around SpecFlow and the support of TechTalk, but even this way we could not afford the costs of building up so many new things. I think this is a good combination. While we believe that SpecRun is a great help and provides quick ROI for the users, it is not the main concern of SpecFlow. With the separation of the two aspects, we can have still our SpecFlow efforts focused on the main concerns (editor, runtime API, documentation) and also provide an extension for the smarter execution experience.

SpecRun is currently in a closed beta stage, but we are planning to have the commercial release soon. Have a look at our website: http://www.specrun.com, stay tuned on our twitter account: @specrun and contact info@specrun.com if you are interested in participating in the beta program.

Enhanced upgrading options in SpecFlow v1.7

by Gáspár on August 1, 2011

We have just released SpecFlow v1.7 over the weekend. In this release there are extended integration options for the Silverlight async testing framework and also for SharpDevelop 4.

In addition to that, we have tried to make the upgrade process easier for the SpecFlow users, especially if they work in bigger teams or on different projects in parallel.

The decision to upgrade a tool for a project is usually made carefully. You have to consider your close deadlines, check the breaking changes, and discuss it with the team.

If you work on multiple projects in parallel (e.g. you have to support and older project too), it can be quite hard to synchronize that all projects upgrade at the same time.

As SpecFlow is (was) a single msi installer and the different versions cannot be easily installed side-by-side, the upgrade process was sometimes challenging.

From v1.7 we have managed to have better separation of the components that are directly used by the project (the generator and the runtime) and the IDE integration. With this, you can upgrade SpecFlow on your machine and defer the upgrade for the individual projects.

For that to work, please ensure that your project is using SpecFlow 1.6 or later and it is setup as one of the following (see further advanced options in the documentation):

  • SpecFlow is configured in the project as a NuGet package (this is the easiest and the recommended option).
  • SpecFlow assemblies (not only TechTalk.SpecFlow.dll, but (almost) all the assemblies from the SpecFlow installation folder) are copied into your project folder structure and the reference for the TechTalk.SpecFlow.dll is taken from this folder. See the detailed list of required assemblies in this documentation page (last section).

If you have this setup, you can safely install a newer SpecFlow version on your machine. The projects will still use the configured generator and runtime.

When you feel confortable to upgrade SpecFlow on your project as well, you can do that project-by-project (the easiest it to use NuGet upgrade).

Note 1: Please make sure that the SpecFlow version on the machine is the same or newer than the SpecFlow version used by the projects.

Note 2: This feature works currently with Visual Studio 2010 only.

 

SpecFlow is and open-source project sponsored by TechTalk. You can also support the project by contributing on github or by “adopting a feature” ;-). TechTalk also provides commercial support for SpecFlow.

Learn SpecFlow at the the Progressive .NET Tutorials: http://skillsmatter.com/event/open-source-dot-net/progressive-dot-net-tutorials-2011

Tracing Web Automation Test Errors with SpecFlow

by Gáspár on July 18, 2011

If you try to run your automated UI tests on the build server, you can quickly run into problems that are quite hard to analyze.

The most common scenario is that one of your assertion like “Then my name should be displayed” fails, because the web driver does not find your name in the resulting page. It works locally, but what’s the problem on the build server? Permission problems? Timing issues? Browser compatibility? Different locale? Web compilation or configuration errors? All of these are potential candidates, but it is hard to find out quickly which one.

We are using the following SpecFlow event binding to trace the content of the current page in case of an error. In many cases from the page content it’s easy to say what the problem was.

[AfterScenario]
public void AfterWebScenario()
{
if (ScenarioContext.Current.TestError != null)
{
Console.WriteLine("Page src for failing test: {0}",
BrowserContext.SeleniumWebDriver.PageSource);
}

//... (if you stop your browser session after
// each scenario, this comes here probably,
// to ensure that the tracing runs before)
}

Git & Dropbox for preparing live demos

by Gáspár on June 14, 2011

I have arrived from Oslo where we did two sessions with Jonas Bandi on NDC 2011 about SpecFlow.

As Jonas lives in Switzerland and I live in Hungary, we have used Dropbox to synchronize the content we are working on. This was fine for slides and other notes, but for the code demo we did not have the right toolset so far.

Actually I like to do demos where I modify and evolve the code during the session. But if you do so, you need to maintain different versions of the code base. You need a baseline to start the demo from and probably also a “done” version if something goes wrong. Additionally you need to be able to revert back to the baseline while practicing the demo. This begs for a SCC and Git is a perfect choice.

To have a little bit of “surprise” effect, I don’t like to publish my demos upfront so did not wanted to put it into a github repository. To be able to share the code with Jonas, this time we’ve tried to combine Git with Dropbox like this:

  • create a “bare” git repository into a dropbox folder
  • clone it to your working folder
  • work on the code & commit changes
  • push the changes to the dropbox folder (the origin)

Though I’m pretty sure that this is not a well scaling solution for real development, but for two persons it was just fine & simple.

Agile Developer Course in Vienna

by Gáspár on May 31, 2011

In summer everyone is on holidays, the projects are running on low frequency, so maybe it’s a good time to meet others and improve your agile development skills.

July 6-8, 2011 there will be a three-day course in Vienna instructed by me (in English) that focuses on agile engineering practices and team collaboration. The attendees can also practice behavior driven development (with SpecFlow) and TDD. This is all done through a lot of hands-on exercises and group discussions.

There are two days left for the discounted price!

Details & registration: http://www.techtalk.at/scrum-trainings/certified-scrum-developer.html

(Successful completion of the course can be credited towards the Certified Scrum Developer (CSD) certification of Scrum Alliance.)