Testing with xunit and zephyr
Why?
Unit testing is great, but doesn’t give the full confidence of all components integrated and working together.
Automation testing does do this for a mature product, but does involve significantly more development effort.
It can’t be brute forced like manual testing.
It needs each step to be scripted together.
And to go beyond simple smoke testing, it needs scenarios.
From the automation tooling I’ve seen, they are mostly focused around login and data scraping.
Beyond that, they require development to insert data & configurations and then check the output.
Approach
Since we need to apply development effort anyways, why not put all the control into the developers hands? Go to our favorite unit testing tools, and turn them into a platform for unit testing. The proposal leans on these core tools:
Using these tools, the developer can further mix in ANY tools that could be used by xunit. These include:
- Selenium
- Blazemeter
- C#
- Javascript
- CSV/Excel
This doesn’t mean that dev shouldn’t enable a less development-inclined QA org from participating.
To complete this approach, a few pieces need to be constructed:
Your actual unit tests in xunit format.
Each testing area gets its own project. Use xUnit, declare Facts. Read params from the environment. Find connectivity and run your code. Ideally, you implement a ‘scenario’ based testing setup where you have lots of scenarios. Each scenario can be broken into multiple tests, or one single test
a way to build your test projects and store the artifact.
a program that can pickup the compiled test projects and execute them (injecting runtime params).
the same program then posting its output to Zephyr for tracking.
Translating the output from xUnit to the custom Zephyr format
What’s great about Zephyr
Zephyr is a really simple product. It is not trying to do everything for you. Instead, think of it like your favorite IDE’s unit testing console. It lists the tests with pass and fail plus notes. Where it gets more powerful is that you can track these tests over time and find out when it was last run, last failed, etc. This gives you the ability to report on progress and regression by non-developers running the unit tests. Additionally, it has some good support for linking into JIRA/Confluence so that testing status can be reported out there. So as long as you don’t think that Zephyr is going to be your entire solution, it does a good job.
What’s great about xUnit
It is .NET! Any power of scripting etc can be executed in xUnit.
If you need to start your scenario with an API call and then poll for an output file… you can do it!
If you need to start your scenario with a button click and then wait for an API callback… you can do it!
If you want your scenario to be dynamic/random… you can do it!
Using the tools listed above, you can use Selenium for scraping, BlazeMeter for API collecting, javascript for extensibility… all is possible.