In Depth Test Reporting with Allure

Always wanted to do test reporting properly? Allure, an excellent framework to properly report your test runs does just what you’d expect from it.

Test run overview in Allure
Test run overview in Allure

The main purpose of Allure should be clear. It provides in reports, graphs and different types of overviews (including optional attachments and links) to present the status of your System Under Test (SUT).

Allure in a nutshell according to it’s developers:

A flexible, lightweight multi-language test report tool, with the possibility of adding to the report of additional information such as screenshots, logs and so on.

Lightweight and lightning fast, it is, the framework can generate a report site in less then 2 seconds (mvn site). The list of programming languages and test frameworks you can use with the framework is impressive as well. Don’t worry, all the big ones are definitely covered!

Core Functionality

In fact, the features Allure offer are limited, but that’s also it’s great power. Allure tends to make use of Annotations. An example of one of those is the @Step annotation. When a method is annotated with it and the method gets called in one of your tests, it will be included as a reproduction step in an issue overview. You will be able to see when the step started and how long the execution of it took.

Similarly, you could attach screenshots, los, JSon, XML, … To your failed tests for later reference.

@Attachment(value = "Page screenshot", type = "image/png")
public byte[] saveScreenshot(byte[] screenShot) {
    return screenShot;
}

Furthermore you could arrange different test cases under features and stories. This is purely to add some structure to your reports. Make use of @Features and @Stories annotations on your test case classes and/or methods.

Side Info

You could always dig into an example report to see what the end product looks like.

Another interesting thing to note, is that Allure is an open source project initially developed for Yandex, a Russian Google Chrome variant for internal testing. Find it on GitHub. They now claim thousands of software testers are giving the Allure experience a high note and that it’s being used all around the globe.

You would be silly not to take a look at it, as it is such an incredible easy setup, but a tool that can help you, your co-developers and even your product owner(s) (as they might want to get some insights at some point during development). Presenting a snapshot of the SUT status is done in no-time. A good approach would be to generate a new report after every CI build and iteration.

Basically all you need to do:

  1. mvn clean;
  2. mvn site;
  3. mvn jetty:run (as the front-end can be displayed locally using Jetty).

Those are all Maven commands. Make sure your dependencies are all in place in your pom.xml and you are good to go.