Testing manually vs. SpecFlow

by Gáspár on May 6, 2011

No, I cannot answer this general question. Just had a nice story in our team and since I’m sure that Mátyás and Viktor will never blog this, I do it now on behalf of them.

We are working on a small project, where we have to implement a smaller module for an existing system. Interesting project, but not BDD at all. Especially there are no automated tests.

Since the project is really small and it has to fit into this bigger existing system, we decided not to introduce SpecFlow there (was a hard decision, you can imagine :).

We are near to finish the first sprint (well, we have introduced Scrum and user stories to the project still), so we started to think about the demo and that we should some manual testing. But how?

The majority of Sprint 1 was a background job that had to process some data depending on some complex rules. So we sat down after the daily standup yesterday and brainstormed the different cases we want to test. “Surprisingly” the cases were pretty much look like acceptance criteria illustrated with examples. As there were too many cases (11) for manual testing, we have selected the 5 most important to be tested.

We had two choices: either we try to find matching data in our test database and test the cases with them or we create test data for our own (with SQL scripts?).

Finally we have chosen the third option: decided to create our own test data, but through Gherkin & SpecFlow. It was a risky decision (budget, deadline, etc.), so we decided for a 1 dev, 1 day time-boxed spike to do it. This was at 9:30am.

To make the story short, at 17:11, the Gherkin files were created and bound for all 11 tests… and they were green (like Cucumber).

And the funniest thing is, that we’ve found a bug… a very tricky one that would not have shown up in the manual tests most probably!

This was very convincing! To be honest, I did not expect to be finished in one day. I was asking Mátyás today morning in the standup carefully, what happened. He said: “Yeah, I’m sitting next to Viktor and doing some pair programming”. I was asking back (trying to hold back my excitement): “And what happened with the SpecFlow tests?” He said very calmly (he is a cool guy): “Ah, yeah the SpecFlow tests are done already. They are all green”.

Not even mentioning the extra benefit, that we can show the automated tests to the customer so they can also create new test variants quickly.

So, thanks for the good job, guys!

P.S.: Just to improve your C# skills, I also share buggy code (without comment):

a ?? 0 + b ?? 0

Re: Rapid Feedback

by Gáspár on April 11, 2011

I’ve read yesterday evening the post of Attila Domokos about rapid feedback that made me thinking. Here is my reaction that didn’t fit to a regular blog post comment.

I think no company should afford wasted time and having a development environment that is determined to waste time is a usually forgotten, but important factor in this. The scenario that has been described by Attila is known to all of us. Well known. Though the story can end many different ways. For Attila, Rails was the “solution”. For me it is BDD and good practices. (I’m not against Rails at all. Rails is fine, but these problems have to be solved within .NET for most of us.) I’m absolutely agreeing with the idea however, to use the right tool (framework) for the right job, but it would be very sad if it would turn out that WPF development cannot be efficient. (I’m not sure though… I know another Attila who would immediately tell me that it cannot, but I’m still hoping.)

[BTW: I’m pretty sure that there are a lot of devs (not the ones working with me fortunately), who actually love this situation and use the compilation times to read the daily newspaper.]

As a developer and developer coach, I always try to ensure that the team is taking the time (early enough) to setup a development environment where you can get rapid feedback. For a WPF-frontend .NET app, here are my suggestions (I’m not saying that we can always do all unfortunately):

Do BDD/ATDD in a test-first, outside-in way, so that usually you don’t have to start up the application for testing the functionality. (We do this with SpecFlow of course. :-)
Consider testing the functionality at the controller level (as well?), in order to get feedback about the implemented functionality without the slow UI automation. (At least until we have a good test framework where the automated UI testing is really easy and fast.)
Do TDD for your inner components. Ensure that the unit testing environment and practices are confortable and helpful. If the tests are executed only after the run-and-test round has been done, you did not win too much.
For WPF, (try to) make sure that your controls can be seen in the VS designer or blend (aka “blendable”), so that you don’t have to start the application for pixel proofing. There are infrastructure for design-time data, but introducing it late is very hard.
Implement shortcuts to your application that the run-and-see test can be also done more efficiently. For example, you can make a stub authentication/authorization provider that bypasses the login screen.
Optimize the build process to be fast – review and remove projects, post-build events, slow signing process, etc. from the build configuration that you use for development. (Be very careful with post-IL tools, like Code Contracts or PostSharp.)
Measure the difference and argue for faster development machines, solid-state drives.

Still, it can happen that you have to wait. Personally, I’m using this waiting time for “development”: Even if I’m not doing TDD, I always follow the practice of “making a small piece work” and then “making it better” (refactoring). So while it is building, starting up, etc. I’m already thinking about the refactoring possibilities. Try it! It’s a little bit exhausting, but fun!

SpecFlow 1.6 and after…

by Gáspár on April 5, 2011

After a lot of work we have finally released SpecFlow 1.6. This was an important milestone of SpecFlow for several reasons:

Altogether 11 people have contributed (in compare to 1.5, where “only” 6)
Effective usage of the public build server, for the contribution and for preliminary results
NuGet package published together with the release

There are many smaller improvements and fixes, but the most important new features are these (see the announcement and the detailed change log for further details):

Visual Studio 2010 integration improvements (improved intellisense, go to binding, automatic table formatting)
Windows Phone 7 support
Assist improvements (dynamic getter for ScenarioContext,
Table.FillInstance())
Step argument transformation for tables and multi-line text
arguments
Integration with Gherkin parser v2.3.5

I have posted recently about the VS2010 improvements and automatic table formatting. If you experience any problems with this features, you can enable SpecFlow debug tracing or disable the problematic features in Tools / Options / SpecFlow. (The trace is emitted to the output window.)

And what will come after 1.6? We have of course several ideas how to improve SpecFlow. Here is a (probably incomplete) list of directions we want to go:

1. Using the generator from the project’s lib folder

The generator interface have been prepared now for running from a different AppDomain, so we want to change the VS2010 integration in a way, that it detects the generator in the project’s lib folder (that was placed there through NuGet for example) and perform the test generation with that. The benefit: the VS integration would be more independent of the runtime and the tools. You can then decide project-by-project when to upgrade to a newer SpecFlow release. Less problems with breaking changes – more frequent releases.

2. Share VS2010 improvements with MonoDevelop

The core infrastructure we use for intellisense and navigation is Visual Studio independent, so we want to reuse that also for the MonoDevelop integration. For that the implementation has to be restructured a bit and the VS-independent parts have to be extracted. We do this together with Dale Ragan, who is the master of the SpecFlow Mono/MonoDevelop integration.

On the CukeUp! I had a good discussion with Aslak Hellesøy about standardizing the data gathered for the autocomplete (e.g. in a json format). This data could be generated build-time and used for other tools in the Cucumber-family providing rich editing experiences. Interesting, isn’t it? Let’s see.

3. Remove duplicated binding-match logic

Currently the logic that decides whether a scenario step matches for a binding is duplicated in the runtime and in the VS integration. This should be cleaned up.

As a part of this, we will establish a dependency between the generator and the runtime (generator depends on runtime), so the file-based sharing can be eliminated.

4. Better dependency handling

I have started to setup a better DI solution for SpecFlow, for

better support for parallel scenarios (eliminate static stuff)
better extensibility – being able to set/override any dependencies from the config file
better support for unit testing

Currently I’m using a mini DI framework, that can be embedded as a single source file (called MiniDi). This is a huge refactoring, so I try to do it step-by-step.

5. Runtime platform improvements

We do want to further improve the support for alternative platform runtimes, like Silverlight or Windows Phone 7 with the help of the contributors. Currently Matt Ellis is working on the asynchronous Silverlight support.

6. Visual Studio 2010 integration improvements

There are a lot of ideas still to improve the editing and navigation experience, like showing the binding status of a step, commenting out scenario blocks or running the focused scenarios from context menu.

7. Documentation, documentation, documentation

Yes, I know. This is something to be improved. But I hope that by using the github wiki it will be easier to complete the docs and keep it up-to-date.

—

Like before, we are collecting feedback about SpecFlow also through a (new) survey – please fill it in (anonymously or with your name) to help our work.

Once again, thanks to the contributors for their great work in SpecFlow. Also thanks to TechTalk for sponsoring the development and the infrastructure of this open-source project.

—

At TechTalk we are developing another product, SpecLog, that supports managing your requirements in agile projects. Please have a look at our website: http://www.speclog.net/.

Automatic Table Formatting in SpecFlow

by Gáspár on April 5, 2011

In the new SpecFlow v1.6 release, besides other Visual Studio 2010 integration improvements there is one more that I did not mention in my previous post: the automatic table formatting.

I have spent a lot of time by aligning the table columns in the Gherkin files. This new feature tries to address this problem: whenever you type a pipe (|) character in a table that has at least two rows, it realigns the table columns. Here are the rules for the aligning:

the column widths are calculated based on the cell texts (header and body) without starting/ending whitespaces
the cell values are aligned to the left with a single padding space
there is at least one padding space on the right
the indentation whitespaces of the header row are repeated for the subsequent rows
comment lines between table rows are handled

The formatting is applied only to the table (parts) before the typed pipe character, simply because of technical reasons: it was easier to preserve the current cursor position this way. Though this sounds a bit weird, this should be still natural if you type in the table row-by-row. If you want to reformat an existing table, just simply re-type the last pipe character.

Gáspár Nagy on software

coach, trainer and bdd addict, creator of SpecFlow; owner of Spec Solutions