Sunday, September 5, 2010

How to integrate MSpec into Hudson CI server

If you’re find this post, you might already know what is MSpec and Hudson CI server. If not, I’ll tell you.

MSpec (short for Machine.Specifications) is an awesome .Net BDD style testing framework for, which allows you write tests without language noise.

Hudson CI is an open source continuous integration server, which is very easy to install and configure with a user friendly interface. It allows you continuously build you projects and verify the quality of these builds. And it has a lot of plugins, which make developing with Hudson more interesting and fun.

So what does it mean integrate and why should I do this?

MSpec has a command line runner, which can fail the build in case some tests are failing, and it can generate nice HTML report which can be placed in build artifacts. So what else do I need, and why i should bother with Hudson integration?

Because dealing with tests is one of the Hudson core features. It can show you information about tests for a particular build and tests history information, such as when they started breaking, how many new tests were added, how test duration changed from build to build, etc. Hudson can also show nice history trend charts.

TestResult

At the screenshot above you can see the test result for a single build.

 

TestResultTrend

At this screenshot (from another project) you can see tests trend.

And the most fun feature (at least to me) is the test tracking in a Hudson Continuous Integration game plugin, which gives points to user on improving the builds. This is my favorite plugin (ok, maybe after Chack Norris plugin).

SoreCard

Would not it be great if MSpec tests will participate in such activities?

How to achieve that?

So how does Hudson know about all available testing tools for various programming languages? Of cause it doesn’t know about all of them. It works internally with JUnit xml reports, but there are many special plugins for most popular testing tool, which in fact transform testing tool output to JUnit format, thus allowing them to be used by Hudson. Unfortunately, there is no such plugin for MSpec currently. And I’m not familiar with Java to write it :)

Good news is that MSpec can generate xml report (although it not as informative as i would like it to be). And we can write XSLT transformation to convert MSpec xml output to JUnit format and use it to provide test results to Hudson directly as JUnit tests.

So here is such XSLT:

So now you can transform MSpec results to JUnit output in you build script and than use it in Hudson. If you are using MSBuil you can use it XsltTransformation task to do the conversion:

And finally you configure Hudson to use test results by specifying converted xml output.

PublishTestResults 

Additionally it is a good idea to publish MSpec HTML report. You may do that with help of HTML Publisher Plugin:

PublishSpecs

That is all you need (except of course that you need to write build script which would run you specifications and will generate xml and html reports). At the end you may see you specifications included as usual tests in Hudson. And as a bonus you get nice HTML report:

HudsonView

The link at the top of the page points to the Specifications HTML report:

SpecificationsReport

Specifications

That’s not all I wanted to say. But I’m a bit tired already. So much words :) Hope it would be useful for someone. And hope that yours Mr. Hudson and Chack Noris will always be happy:).

5 comments:

Warren said...

Ivan thanks for posting this! I'm about to try setting it up now and will let you know if I have any issues.

I also saw in your hudson build you had a code coverage graph there. Can I ask what you are using to do this and integrate with hudson?

Thanks for the awesome blog and wish I could read Russian? so I could understand the rest of it!

Vansha said...

@Warren We've used NCover comunity edition earlier, but it does not work with .Net Framework 4. So I've tried PartCover with ReportGenerator. Thought, this solution does not work perfectly with totally uncovered classes.

Nestor Lobo said...

Hi, I really like the transformation and already tested, but I can see that your screenshot show the time of each test but the transformation doesn't transform them from mspec results.

I am missing something?

Thanks in advance.
Regards

Ivan Korneliuk said...

Hi! Actually at the time this post was written, MSpec xml report did not produce timings and failure details (time from screenshot are from NUnit tests :).

Good news is that MSpec xml report now contains both timing and failure details. I've just updated gist to transform them accordingly.

Nestor Lobo said...

Great, thank you for the update, I also already added timing and errors details.

Thanks for the prompt response...

Regards
Nestor