OTel uses Tracetest for trace-based integration testing

OpenTelemetry has taken the observability world by storm. It’s no longer the tool of tomorrow since it officially applied for [graduation in the CNCF](https://github.com/cncf/toc/pull/1271) just before [KubeCon + CloudNativeCon Europe 2024](https://tracetest.io/blog/observability-at-kubecon-cloudnativecon-europe-2024-in-paris). It’s the tool of today.

This case study covers the importance of OpenTelemetry in the observability community and how [Tracetest is part of the official OpenTelemetry Demo repository](https://opentelemetry.io/blog/2023/testing-otel-demo/).

OpenTelemetry is an [Observability](https://opentelemetry.io/docs/concepts/observability-primer/#what-is-observability) framework and toolkit designed to create and manage telemetry data such as [traces](https://opentelemetry.io/docs/concepts/signals/traces/), [metrics](https://opentelemetry.io/docs/concepts/signals/metrics/), and [logs](https://opentelemetry.io/docs/concepts/signals/logs/). Crucially, OpenTelemetry is vendor- and tool-agnostic, meaning that it can be used with a broad variety of Observability backends, including open source tools like [Jaeger](https://www.jaegertracing.io/) and [Prometheus](https://prometheus.io/), as well as commercial offerings.

This is helpful in big companies where different teams might be using different vendors. OpenTelemetry makes sure that if someone moves teams, they can keep using the same libraries. This is changing the industry by reducing vendor lock-in.

At KubeCon + CloudNativeCon Europe 2024, we had the opportunity to speak with [Juliano Costa](https://www.linkedin.com/in/julianocosta89/), an [OpenTelemetry Demo maintainer](https://github.com/open-telemetry/opentelemetry-demo), about the critical importance of OpenTelemetry in the space of observability.

> *View the video on YouTube, [here](https://www.youtube.com/watch?v=vgBKuSMa5J0).*

We delve into Juliano’s responses to key questions, shedding light on **the significance of OpenTelemetry** and its impact on decision-making in the realm of observability.

These are the questions we asked Juliano in our conversation. Below you will find a small glimpse of what we talked about, but you can watch the full video for more information:

- Can you share some details about your career journey up to today and your role in the OpenTelemetry Demo?
- What is OpenTelemetry, why is it important, and can you explain what the OpenTelemetry Demo is?
- What do you hope to achieve with the OpenTelemetry Demo and what specific issue were you trying to solve?
- Can you describe the biggest challenge when testing the OpenTelemetry Demo?
- What are your future plans for the OpenTelemetry Demo?

## The Background — How was the OpenTelemetry Demo born?

Juliano initially created an internal demo to showcase OpenTelemetry. This pushed him to study and learn more about OpenTelemetry to enhance his knowledge. He shared this story with us:

> *Every night when my wife went to bed I started coding. I took the project from Google, the online boutique, which is a demo, and cleaned up everything that is Google-related. I think it had OpenCensus, and I added OpenTelemetry, the Collector, Jaeger, and instrumented all the services. At that time I think it was 5 different languages. I was super happy with it. A couple of months later the OpenTelemetry community got together and said, what about creating an official OpenTelemetry demo to showcase all the languages? Then we started evaluating all the forks from the Google microservices demo, and mine was cleaner than the rest with no vendor-related things. They decided to use my fork as the base for the OpenTelemetry demo. That’s how I got involved. Since day -1.*

Juliano’s involvement with the demo, which started from its inception, continues today. He is now a Developer Advocate at Datadog, allowing him to remain active in the community, contribute to the demo, and educate the community about about OpenTelemetry.

## The Benefit — What is OpenTelemetry and why is it important?

OpenTelemetry, a standard in the observability space, has revolutionized the approach to observability. Juliano shared this with us:

> *OpenTelemetry is a standard that changes the approach observability has been done for the last couple of years. We always had vendors that had their own solution. If you want to switch from vendor A to vendor B, you would have to reinstrument your application, or redeploy the vendor’s approach to observability.*

This is particularly beneficial in large companies, where various teams might use different vendors. OpenTelemetry ensures that anyone transferring from one team to another can continue using the same system.

## The Goal — What issue is the OpenTelemetry Demo solving?

The objectives of the OpenTelemetry Demo are multifaceted, aiming not only to showcase vendor support for OpenTelemetry but also to facilitate user onboarding and support maintainers with new releases.

Juliano shared an extensive explanation:

> *There are three goals to the OpenTelemetry demo. One is for vendors to showcase their support for OpenTelemetry. The second is to help people be onboarded to OpenTelemetry so they can see in a real application how OpenTelemetry works and the benefits of it. The third is to help maintainers within the release. When there is a new release of a specific instrumentation or language, they can come to the demo, update it and see if everything is working as expected. I’m really happy to see these three goals being achieved.*

Additionally, Juliano also mentioned that the demo can be used to compare different vendors' handling of OpenTelemetry data, providing actual data to base decisions on, which simplifies the process of selecting the most suitable vendor.

## The Challenge — Why is testing the OpenTelemetry Demo difficult?

Juliano mentioned one of the biggest challenges in running tests against the OpenTelemetry demo was during PR merges when validating the demo's functionality. The process involved pulling the PR, building the image with the changes, and then running it to see if everything was working. Maintainers or approvers did not always do this. There were instances when changes seemed fine at the code level, were approved, merged, and only then, when running the demo, it was found that the trace was broken.

## The Solution — What issue is Tracetest solving for the OpenTelemetry demo?

To address this, the suggestion was made to use Tracetest in PRs. This approach would involve running the trace-based test whenever a PR was received to see if it broke anything. After a few months of advocating for this change, it was finally implemented.

Juliano shared his experience:

> *Tracetest joined the OpenTelemetry demo pretty soon. The front end was redesigned by [Oscar Reyes](https://www.linkedin.com/in/%F0%9F%87%B2%F0%9F%87%BD-oscar-rafael-reyes-gaucin-8aa843a8/). It was a great contribution. They were involved in the project since day one. So I said… why aren’t we using Tracetest to test changes? […] After a couple of months pushing the maintainers we finally got it. Now, whenever you send a PR to the demo, as soon as there is one approval, we build all the images, we run the demo, and we run Tracetest. That ensures the traces are not broken, everything is connected as expected. If there are breaking changes now, we fix as we go and not whenever a user opens a bug.*

## The Future — What are future plans for the OpenTelemetry Demo?

In a recent SIG meeting, the team, including Juliano, Mikko, and Austin, discussed plans for the OpenTelemetry demo. Juliano was eager to share:

> *We got some credits from Oracle Cloud. We want to have the demo up-and-running all the time so people can navigate the demo and see the value. They will be able to access the front end, Jaeger, Grafana, and see the metrics, logs, and traces. You have somewhere to check the data the demo is producing before even bothering deploying it yourself.*

Although users will still have to deploy the demo themselves to try out vendors and see how they behave, this gives them a place to check the demo's data before deploying it themselves. The goal for now is to host the demo in the cloud, allowing people to see the end product before investing the time to build the images. The team is also considering obtaining build credits to secure a build machine, from which they can pull the built images.

If you are looking for a comprehensive trace-based testing tool, you should give Tracetest a try. Tracetest gave our team a view of the system's traces and key flows. For any team serious about ensuring the robustness and efficiency of their systems, Tracetest is an invaluable asset.