Detailed Report for PartCover in TeamCity 5.1

by Gáspár on September 20, 2010

I was configuring a build with TeamCity (for SpecFlow that is hosted on github). After the initial success, I thought I would also enable code coverage in the build. I don’t really believe in code coverage numbers, but if it is easy (and free) to configure, than why not?

TeamCity supports NCover and PartCover out of the box. As I wanted to go for a free solution (SpecFlow is a free, open-source product), I used PartCover (v4.0.10801). PartCover seems to be a great tool, but it is not so easy to set it up with TeamCity. (You can find a lot of questions and discussions about it.) As I can see, there are some improvements in the later versions of PartCover and TeamCity, but still there are some problems.

If you just start with the build server configuration, you should read first Jeremy Skinner’s post. (On TeamCity 5.1, you don’t have to worry with the assembly renaming.) The rest comes here:

Problem: Sometimes you see no results at all

Maybe there are many reasons that can cause this, but for me the problem was an x86/x64 issue. On an x64 build controller, you have to make sure that the test execution and the coverage are all running with the same processor architecture. I’ve achieved success with x86 compilation only, with x64 I always encountered a “…\tmpE1BE.tmp exited with code -1” error. But I did not play with this so much. These are my working settings:

Problem: The reports provided with PartCover don’t work with TeamCity

You can find fixes for the reports in stackoverflow and also on Jeremy’s post, but altogether these reports are not too detailed, e.g. they don’t go into method level. Without knowing which method causes the low class coverage, the numbers are not too helpful (except for your boss). So this leads me to the third problem…

Problem: There is no detailed HTML report available

As I have mentioned above, this was quite a pain for me. So I refreshed my old XSLT skills (again) and tried to put together a report that shows what I need:

global & assembly-level summary
top 10 uncovered method
detailed method-level report with source code line numbers (it is not possible to embed the source code file from XSLT unfortunately)
coverage values excluding the very simple members (simple property get/set, empty constructor, etc.)

The detailed report (for a small solution):

Class details with method-level numbers:

How to setup the detailed report

download the XSLT from here: PartCoverFullReport.xslt
copy the XSLT file in the “xslt” folder of PartCover on the build controller(s) (e.g.: C:\Program Files\PartCover\PartCover .NET 4.0\xslt)
in the build configuration, specify the “Report XSLT” field as:C:\Program Files\PartCover\PartCover .NET 4.0\xslt\PartCoverFullReport.xslt=>index.html

SpecFlow Workshop at Skills Matter / Progressive.NET Tutorials

by Gáspár on May 17, 2010

Luckily escaping the BA strike and the newest volcano ash cloud, on Friday I safely arrived home from the Skills Matter Progressive.NET Tutorials in London.

This was the first time I visited this event and it was a great honor that Jonas and me had been invited to do two workshops about behavior-driven development (BDD) and SpecFlow.

I enjoyed the event very much. I also visited a lot of interesting tracks, but maybe the best was to meet so many enthusiastic and “progressive” developers. It helped me a lot to charge my batteries – to keep on thinking about new ideas and improvements in the development process and technologies.

Chris has written a short summary on our sessions, I don’t want to repeat him. I was very pleased that we got so much interest and positive feedback after the sessions. It was a pleasure to work with Jonas and thanks also for TechTalk’s support.

Let me highlight only two things about the BDD session.

The best part in it (again) was the acceptance criteria writing task, where the attendees had to write 2-3 (the most important) acceptance criteria about the shopping cart feature of a book shop application. The goal of the exercise was to discuss good/bad practices and patterns in formulating scenarios. We thought that the best way to learn these was to learn them on the basis of personal experience. But as we can never guarantee that the attendees will really make those mistakes that we would like to highlight, this is always risky. But it worked again! We didn’t have to show any prepared solution (to be honest we didn’t prepare any), we could explain and discuss a lot of interesting things through the samples the attendees came up with. It was amazing!

In the beginning of the workshop where we wanted to collect possible hopes and fears about BDD to be addresses during the sessions, we had another task as well. Usually we do this with yellow stickers but this time we made a last minute adaption to our process – making the session “progressive”… Instead of stickers, now we used a twitter – everyone could post comments with the term #prognetbdd that we displayed on the projector and discussed. It was a fun!

SpecFlow: Pragmatic BDD for .NET

by Gáspár on October 19, 2009

SpecFlow icon Last year I worked a lot on finding better techniques for automated functional testing.

I was investigating topics like how to solve problems of automated web UI testing (especially ASP.NET), how to structure automated functional tests or how to ensure the preconditions of functional tests. We even created small tools to support this, but altogether it was not a big success.

Finally, we started to use cucumber and this was the first tool where I felt that it supported the concept we need. However, we had some issues with cucumber, too:

Using it with .NET projects (through IronRuby) was quite slow, and since the IronRuby documentation is not too detailed, we continually encountered ruby/.net conversion issues that were hard to debug and solve.
We had our continuous build infrastructure, but it was not prepared for running cucumber tests.
The development team did not have sufficient ruby knowledge.

Considering these problems, we finally decided to move forward: to implement a small .NET running environment for cucumber, where the step definitions are written in .NET. Since I’ve gathered quite a lot of experience with Visual Studio SDK, I came to the idea to implement this tool as a single-file generator that generates NUnit test fixture files from the cucumber file, which can be executed by NUnit runner, also on the build server, without extra configuration.

We resolved to implement the bindings as .NET methods, having [Given], [When] or [Then] attributes.

[Given(“I have entered (.*) into the calculator”)]       
public static void GivenIHaveEnteredANrIntoTheCalculator(int nr)       
{       
    //…       
}       

The tool contains a runtime part as well that finds the matching step definition method during the test executions and executes them. The first implementation used an Oslo parser, but finally we had to rewrite it to use ANTRL, because of performance issues.

The tool worked surprisingly well and we started to use it in a project where we had our cucumber tests already; we just had to rewrite the step definitions to .NET, since the cucumber files were compatible.

In the meanwhile, TechTalk decided to make an open-source project out of the in-house tool. But life is not easy, we have quickly realized that the names we have used so far (“NCucumber” or “Cucumber for .NET”) cannot really be used as a “trademark”. The cucumber team – somehow understandable – wanted to keep the name “cucumber” for products which are direct ports of cucumber. Although we are compatible in the cucumber file (gherkin) level, we don’t have an own execution environment (NUnit does it for us). So we have chosen the new name: SpecFlow.

The first version of SpecFlow is already available, so you can try it out (the source-code is also coming next week). To learn more, visit the website:

http://www.specflow.org

DevCamp09: Behavior Driven Development Applied in Unit Testing

by Gáspár on October 15, 2009

Session startup slide The presentation material (slides and demo source code) of my DevCamp09 presentation are on-line now and can be downloaded from the TechTalk website here.

The downloadable material also contains the framework that can be used to write BDD-style unit tests. This framework will most probably be joined to SpecFlow, the open-source BDD tool we have just launched (hence, the new namespace, TechTalk.SpecFlow.UnitTesting).

Session Abstract

Developer life was good in the golden age of the waterfall: big upfront designs left no surprises for later implementation, code documentation was a piece of cake as nobody else read it anyway, and testing was something developers just heard of remotely through a department called “QA”.

But things have changed for the worse with the rise of agile methodologies: designs are ruined by requirements coming in weekly and code that was already working perfectly suddenly breaks. Now, business wants to collaborate directly and is interested in details of the implementation, meaning that documentation can no longer be kept on the back burner. And QA is asking for a high coverage of unit tests out of the blue, causing big overheads to write and maintain them with all these ongoing changes.

However, a lot of concepts have been introduced that promise to ease the pain: IoC helps us reduce dependencies in the code, BDD helps us focus on the next functionality to deliver, TDD helps us structure code for testability and Fluent Interfaces are a way of building DSLs that help us communicate with business.

This talk will show how we put these concepts into practice in a project that followed an agile methodology. We’ll show how IoC helped us stay flexible, and how we structured our unit tests in a way that described the expected behavior and could also be understood by business. The source code of the supporting infrastructure we built will be made available to you after the talk as a download.

Screenshots from the demo

Execution result of a very simple BDD-style unit test

…and the unit test itself

Execution result of a more complex BDD-style unit test

…the unit test…

…and finally the implemented code

Doing it right

by Gáspár on April 20, 2009

I’m a software developer. From time to time, I come to the question why it is good to be a developer? Why is it good to write software?

I don’t have the ultimate answer yet, but I already have a few: making someone’s work easier; to able to get out more value from the things we know; because it is a joy for me; because I can teach other developers to find this joy; or simply because it provides a reasonable good salary. (I always pick the one that fits best to my mood, but honestly I can also argue with each of them.)

I’m not working on software that saves lives directly, and there was a time when I was very sad about that. But then I have realized that the world is full of people that also don’t do that, still you can be very happy that they do what they are doing. (Now I cut the long list of examples, because I tend to be too sentimental.) The point is that you are happy with them because they do their job right.

Today I have found the website of the TED community. TED stands for Technology, Entertainment, Design, and it is a community (and a yearly conference) where people try to bring these concepts together and make something remarkable. It is innovative, but not necessarily technology-wise. And very-very inspiring. The best talks from the conferences are also available on-line, so you should better look at them. They are all relatively short (the allowed time for a speech is 18 minutes).

Why does this come here? Because sometimes you have to step back and look at things from a broader perspective in order to step further or gather new ideas. Here are some:

Matthew Childs: Hang in there! 9 life lessons from rock climbing

You will start listening to this as it would be about rock climbing, but you’ll soon realize that this is also about you (and for example software development). (04:48)

Scott McCloud: Understanding comics

As you’ve guessed this is also about more than comics. It is about vision and about finding ways to understand things (take it apart and put it together). About how technology and art can cooperate with each-other. (In this sense, it can be also helpful for software usability.) And last but not least to show how to make a memorable presentation. (17:08)

In McCloud’s talk, there is an interesting list of rules which I have to think of:

Learn from everyone
Follow no one
Watch for patterns
Work like hell

I have learned a lot today.

Gáspár Nagy on software

coach, trainer and bdd addict, creator of SpecFlow; owner of Spec Solutions