Integrating Tracetest w/GitHub Actions in CI pipeline

At Tracetest we care a lot about software quality, and that applies to Tracetest itself, obviously.

We were working with a basic CI pipeline, running some unit tests before merging PRs. But that wasn’t enough. We want to be able to deliver the best possible quality software, and that requires more.

We need a way to deploy and run integration tests over new code, and we have the perfect tool to run those tests: Tracetest. Our goal is to test Tracetest using Tracetest!

Designing the Pipeline

Probably the hardest part of any design process is defining the goals. Setting a CI pipeline is not an exception.

We wanted to set up a GitHub Actions pipeline so that we can run Tracetest tests in that CI environment. So, we created a tentative step-by-step recipe for that:

1. When creating a Pull Request, build the code and deploy it using our Helm chart.
2. Run Tracetest tests, and other end-to-end tests.
3. Clean up created resources.

While it sounds simple to do that, in practice it has its nuances.

1. Which Tracetest instance will run the tests ("main instance")?
2. How are we going to communicate from the GitHub Action to the "main instance"?
3. How will the "main instance" communicate with the newly deployed version (called "target instance")? Same question goes for running Cypress and other end-to-end tests.
4. Deploying and running integration or end-to-end tests is time-consuming. How can we minimize that?

We already have developed most of the tools needed to solve the problems listed above:

1. We deploy a real, working Tracetest instance to our demo environment using our Helm chart, so it would be easy to build Docker images and deploy them for each Pull Request on demand. We can also keep a separate "main instance" running the latest stable version.
2. We have a CLI that we can use to talk with the "main instance". We just need to use kubectl and port-forward to the "main instance", and point the CLI to the correct address.
3. Making two services that are not exposed to the internet talk to each other can be tricky. Luckily, we use Kubernetes to run the "main instance" as well as the "target instance", so we can make this work with little effort by deploying both instances to the same cluster, and relying on the Kubernetes internal DNS. If needed, we can pass environment variables to the Tracetest CLI so we can configure tests as needed.
4. We can organize the pipeline so the fastest processes are run at the beginning and the slowest at the end, so we can fail fast instead of waiting several minutes just to see a unit test failing.

The Implementation

This is what our final pipeline looks like:

You can see it in action here.

It is divided into 3 main stages:

1. Run the fastest failing test in parallel: unit tests for CLI, server, and web. This also ensures that we are ready to build the Docker image and CLI tool used later.
2. Build and deploy a docker image, tagged specifically for this Pull Request, and deploy it to its own Kubernetes namespace. We use our Helm chart for this since it is easy to configure the instance. Also, we can wait until the instance is ready to run.
3. Run long tests:

• Cypress end-to-end tests: For this, we set a port-forward using kubectl, so Cypress can work as if it was a locally started instance listening at <http://localhost:8080/>.

• Tracetest tests. We use Tracetest’s CLI, so we build it at this stage. We set a port-forward to the "main instance" so the CLI can talk to it. Since the "target instance" runs in a dynamically created namespace, we need to use environment variables to make the CLI inject the correct URL when running test definitions. The good thing about this is that we are implicitly testing the CLI too!

After those steps are completed, the output of the tests is persisted with the GitHub Action, and we can safely remove the created Helm release and namespace.

We are currently working on actually testing Tracetest with trace test. The pipeline right now is only "ready" to support this, but we will move our tests in the following days.

You can see the actual pipeline definition here:

tracetest/pull-request.yaml at main · kubeshop/tracetest · GitHub

Final Thoughts

While Tracetest is still in its early stage, it has all the tools needed to easily integrate into a CI/CD pipeline. Creating end-to-end and integration tests is usually not an easy task, but we found that doing it with Tracetest and Kubernetes was straightforward.

We are eager to hear your feedback and to talk to you. Please share your thoughts on how trace-based testing can help you and what we should do next to improve Tracetest. Have an idea to improve it? Please create an issue, give us a star on GitHub, and join our community on Slack.

Tracetest is supported by Kubeshop (Open-Source Accelerator).