Many teams who use SpecFlow setup their continuous integration environment with Microsoft Team Foundation Server (TFS) or its cloud based service, the Visual Studio Team Services (VSTS, fka VSO). TFS/VSTS has an improved build system, where it is much easier to setup and customize builds. Regardless of the fact, which unit test provider you use for SpecFlow, it is super easy to setup a build to run your BDD scenarios, because all the major unit test frameworks provide the necessary adapter for it (this is the same adapter that you need to run the tests from the Visual Studio Test Explorer Window).

Since the infrastructure is relatively new and very generic, it is sometimes hard to find information on particular questions. For SpecFlow users one of the most typical question is, how to run only specific scenarios (marked with a tag) in a TFS/VSTS build.

You might need such filtering for various reasons. I like capturing scenarios that can be tested only manually in the feature files (these you can run for example with Microsoft Test Manager if you synchronize the scenarios to test cases). I mark these scenarios with @manual (and/or mark the others with @automated).

SpecFlow converts the tags to test case categories (or the equivalent concept in the test framework you use), so basically the task is to filter the test execution in the build definition for particular categories.

The filter expression can be provided in the “Test Filter criteria” setting of the “Test Assemblies” build step, where you can use even complex filter expressions. To filter for scenario tags, you have to filter for the TestCategory property, like these:

• “TestCategory=automated” – executes the scenarios marked with @automated
• “TestCategory!=manual” – executes the scenarios that are not marked with @manual
• “TestCategory!=manual&(TestCategory=fast|TestCategory=important)” – executes the scenarios that are not marked with @manual but marked with @fast or @important

As you can see, you can formulate even complex expressions. Unfortunately the MSDN documentation is not very detailed with regard to the possibilities, but you can find a quite good description in an older MSDN blog post.

If you use this together with SpecFlow tags, the important rule is that you must not include the “@” tag prefix in the filter expression and you have to use the TestCategory filter property regardless of the fact what the concept is called in your test framework.