Note: If you haven't checked out how to setup living documentation article, you can find it here.
Sometimes you need to run your tests against different resources or combinations of resources, or indifferent environments (Testing, Staging, Production). To achieve this, multiple jobs (often called Matrix Jobs or Builds) are created in your CI pipeline to trigger Cucumber with these different combinations.
Behave Pro can display the pass and fail results for each of the different Job runs of Cucumber but the results need to be published to Behave Pro for each job.
Publishing test results from each job
To tell Behave Pro which combination of properties a test report belongs to, you need to include one or more parameters using the
X-PARAM header in the HTTP request as shown below in the example.
curl -L -X PUT https://test-reports.behave.pro/REST/1.0/bdd/report \
-H "X-API-KEY: your_api_key” \
-H "X-COMMIT-ID: your_commit_sha" \
-H "X-PARAM: Os=Linux" \
-H "X-PARAM: JVM=Java 1.8" \
In the screenshot below, you can see that the test results are separated by the Java version and grouped together by the OS type. This is purely for example purposes, you don't have to use OS and Java parameters, you can specify any parameter you want to.
Note: Please note that, the order of the parameters matters how are they going to be displayed. In the example above the parameter that specifies the OS type has to be the first one.
These parameters are provided via the
X-PARAM header and the header value must be a key-value pair, for example
OS=Linux. Please note that the order of these parameters matters. In the curl example above, the
Os=Linux parameter is treated with higher priority than the
JVM=Java 1.8 parameter. If your tool doesn't support multiple header values within the same header name, you can use comma to delimiter the parameters within the same header.
X-PARAM: Os=Linux, JVM=Java 1.8.
You can also create a deeper structure with the matrix build test results by providing additional parameter with the
X-PARAM header. The following example was created with the following headers via a curl command:
-H "X-PARAM: Os=Mac OS" -H "X-PARAM: JVM=Java 1.8"\
-H "X-PARAM: Test=World"