Sigma Software built load testing for microservices with k6 + Tracetest

## The Problem

Our team of 15 people at [Sigma Software](https://sigma.software/) are working on a B2C application. The system is microservice-based and divided into 5 primary services. It has to handle high load volumes. The front end is based on a single page app (SPA), so the constraint on handling the load will be the back-end architecture.

## The Challenge

We needed to load test the application to ensure it would handle the expected order volume and uncover any hidden performance issues. We went in search of tools to use.

## The Requirements

I had past experience with load testing with JMeter. While it worked, it did not feel modern. In addition, investigating the root cause of any detected slow tests run in JMeter was cumbersome. You would have to “self instrument”, using tools such as SQL Profiler to capture SQL logs as load testing was running. You could get to the answer, but it was not easy. We went in search of more modern and developer friendly tools.

## The Solution

After some investigation of the performance testing space, we discovered k6. We liked how easy the load tests were to create and run, and the tool was lightweight and easy to use. As a part of this investigation, we were introduced to a second tool, Tracetest, as we watched an episode of the k6 office hours. The episode was titled "[Integrating k6 and Tracetest](https://www.youtube.com/watch?v=Dog70D7aVyg)", and covered how to use k6 and Tracetest together in load testing. Tracetest utilizes OpenTelemetry-based distributed tracing to enable tests to “see inside the black box”. The visibility exposed by enabling distributed tracing and viewing tests in Tracetest enables you to see the cause of the problem, not just that it occurred.

Convinced of the power that having visibility into what was occurring during failed test runs would make a real difference, we discussed it with our CTO and decided to add OpenTelemetry to our microservice-based app. The application’s are written in C#, so we used its auto instrumentation capability to enable distributed tracing. The developers were surprised how easy tracing was to add, asking “is that all?” a couple of times. We were initially going to send the trace data to Datadog, but the load tests hit API request limits. We ended up setting up Jaeger to store trace data in our testing environment. Tracetest natively integrates with Jaeger and Datadog, so connecting it was easy.

We had an initial issue when we ramped up the number of requests to 200/users per second. We communicated with the Tracetest dev team via Discord, and they were able to reproduce the two issues we saw and put out fixes within a few days. We then began some test runs at volume to test the order management flow.

We ran the load test multiple times to measure the system’s performance. As a result, we identified the performance bottleneck when computing the ETA for order processing. Our analysis showed excessive database queries as the root cause of the performance bottleneck. To address this, we implemented a solution using Redis caching, thereby minimizing direct database hits for ETA calculations. Given the findings, we are now working to address the issues and will retest once we finish.

## The Future

We have future opportunities to use Tracetest in our CI/CD pipeline. We want to explore using Tracetest to help developers write custom instrumentation and more easily monitor their traces during the development process. We may also look at building trace-based tests to perform integration tests which can be run in the CI/CD process.

If you are looking for a comprehensive load testing tool, you should give k6 + Tracetest a try. Tracetest gave our team a comprehensive view of the system's performance and identified potential bottlenecks before they became critical issues. For any team serious about ensuring the robustness and efficiency of their systems, k6 + Tracetest is an invaluable asset.