Generating JUnit Reports for Flutter Integration Tests with Gitlab Integration

Flutter is an open source framework by Google for building beautiful, natively compiled, multi-platform applications from a single codebase. It provides smooth integration for integration testing also. To do a setup for integration testing, You can refer Flutter Integration Testing Doc.

Prerequisite :

1. A repository that has some integration tests for the flutter app.

If You need a reference, you can make a clone of my repository. Flutter Integration Test Setup Project Reference.

2. You should be able to run the test on Your local.

JUnit Report Integration On Your Local Machine:

1. Dart provides one package for JUnit Report, JUnit Package
2. We will create a shell file to run our Test script. Let's Say the File name of the shell script is — testRunner.sh
3. We will install the Junit package using flutter and will activate it for reporting.
4. We will write our command to run the flutter integration test and store the results in JUnit XML report format.

Install Junit Package and Activate Junit Report

Update your Runner shell file with the above content and run this Shell file. Once your execution will be completed, the test result will be stored in the ./junit-report.xml file.

Running Test on GitLab Runner and Generating JUnit Report:

we will create a simple gitlab.yml file that will run the above shell script. You can enhance the same based on your requirement and configurations.

gitlab yml file

Yeah, In the above steps we created a simple gitlab.yml file that will execute our shell script. As per our above shell script, it will store all execution data in Junit Report File i.e './junit-report.xml'.

we will store the XML file as an artifact for each pipeline execution, that would be helpful to get more logs and failure results. We can generate the trend also with the same.

Till now we are good with integrating the test execution with GitLab and generating readable reports and storing previous reports as artifacts.

One thing you will notice with your execution is, In all cases whether tests are passed or failed, Your pipeline's status will come as PASSED. This is due to the case that, on flutter test execution, if some test cases failed, it gives only errors instead of showing as failure in the JUnit report. Due to this reason, pipeline status always comes as PASSED.

The above drawback could be misleading in some situations. For example, You have integrated slack/email notification on pipeline failure. But due to the above issue, your notification will always indicate that pipeline is passed.

To achieve the correct status of the pipeline, we will update our above shell script file.

Now we are all set . If Any test will fail, You will get FAILED status and if all Test passed, You will get PASSED status.

References -

Unit test reports | GitLab

junitreport | Dart Package

Happy Learning..