Sumo Logic and Tracetest: AI-Driven Observability Meets Testing

Today, I’m thrilled to announce that Tracetest now integrates with Sumo Logic, the AI-driven Observability platform. With OpenTelemetry traces, you can make data-driven decisions and ensure reliable releases through trace-based testing.

*Check out this [hands-on code example](https://github.com/kubeshop/tracetest/tree/main/examples/quick-start-sumologic-nodejs) of how Tracetest works with Sumo Logic! You can also read the [quick start recipe in the docs, here](https://docs.tracetest.io/examples-tutorials/recipes/running-tracetest-with-sumologic).*

## What is Sumo Logic?

[Sumo Logic](https://www.sumologic.com/) is a cloud-based platform that provides log management and analytics services. It helps organizations monitor, troubleshoot, and secure their applications by offering features like AI/ML-powered log analytics, cloud infrastructure security, compliance and audit readiness, and Cloud SIEM for enhanced incident visibility. The platform integrates various tools and technologies for improved operational efficiency and security posture in cloud and on-premise environments.

## What is Tracetest?

[Tracetest](https://tracetest.io/) is a tool for trace-based testing. It’s [open source](https://github.com/kubeshop/tracetest) and part of the CNCF landscape. Now, with a [managed platform](https://app.tracetest.io/) offering as well!

Tracetest uses your existing [OpenTelemetry](https://opentelemetry.io/) traces to power trace-based testing with assertions against your trace data at every point of the request transaction. You only need to point Tracetest to your existing trace data source, or send traces to Tracetest directly!

Tracetest makes it possible to:

- Define tests and assertions against every single microservice that a trace goes through.
- Build tests based on your already instrumented system.
- Define multiple transaction triggers, such as a GET against an API endpoint, a GRPC request, a Kafka message queue, etc.
- Define assertions against both the response and trace data, ensuring both your response and the underlying processes worked as intended.
- Save and run the tests manually or via CI build jobs with the Tracetest CLI.

## Tracetest Now Works with Sumo Logic!

[Tracetest now works with Sumo Logic](https://docs.tracetest.io/configuration/connecting-to-data-stores/sumologic), allowing you to combine the power of Tracetest and Sumo Logic in your developer workflows. With Tracetest, you can write trace-based tests and enhance Sumo Logic's always-on regression detection capabilities.

As an existing Sumo Logic user, you have OpenTelemetry instrumentation configured in your code and are using the Sumo Logic OpenTelemetry Collector to send traces to Sumo Logic. Tracetest integrates seamlessly with Sumo Logic by retrieving traces from Sumo Logic's API endpoint to execute detailed trace-based tests.


*Image 1: Architecture overview of how traces are ingested into Sumo Logic and fetched by Tracetest for testing.*

When running integration tests, it can be challenging to identify the exact location of an HTTP transaction failure within a network of microservices. Tracetest addresses this issue by allowing you to run tests with assertions using trace data from all services. These tests can be easily integrated into your CI/CD process to ensure the proper functioning of your system and detect any potential regressions.


*Image 2: In this example, within the Tracetest UI you can see that a test assertion for a HTTP trace span succeeded because it took less than 500ms to complete.*

Improve your testing approach by utilizing Tracetest for test creation and Sumo Logic for comprehensive test result analysis. With Sumo Logic, you can monitor test executions, establish connections between relevant services across different time frames, and gain valuable perspectives on system performance. This combination allows you to understand system behavior, gain insights into system performance, and highlight the impact of changes on performance.


*Image 3: Traces triggered by Tracetest are surfaced in Sumo Logic’s Traces page.*

When using Tracetest, a helpful method for identifying issues is by reviewing trace data in Sumo Logic. Any problems identified can then be transformed into new tests or areas to investigate in Tracetest. This iterative process ensures continuous improvement.


*Image 4: Here you see a trace drill down of a test in the Sumo Logic UI.*

## Try Tracetest with Sumo Logic

Start using Sumo Logic by signing up for a [free trial](https://www.sumologic.com/brief/free-trial/) then configure the [Sumo Logic OpenTelemetry Collector](https://help.sumologic.com/docs/send-data/opentelemetry-collector/) to send traces to Sumo Logic.

You’ll need an [installation token](https://help.sumologic.com/docs/manage/security/installation-tokens/) for this. Make sure to replace the configuration below with your installation token.

```yaml
receivers:
 otlp:
   protocols:
     grpc:
     http:

exporters:
 sumologic:

extensions:
 sumologic:
   # Create an Installation Token in your account at
   # Administration > Security > Installation Tokens:
   # https://help.sumologic.com/docs/manage/security/installation-tokens/
   installation_token: <installation token>

service:
 extensions: [sumologic]
 pipelines:
   traces:
     receivers: [otlp]
     exporters: [sumologic]
```

[Tracetest is a managed platform](https://app.tracetest.io) while [Tracetest Core is open-source](https://github.com/kubeshop/tracetest). Both work with Sumo Logic! However since it’s a bit easier to use the managed platform, I’ll showcase that.

Start by installing the Tracetest CLI. Below is a sample for MacOS, and [here’s the guide for other operating systems](https://docs.tracetest.io/getting-started/installation#install-the-tracetest-cli).

```bash
brew install kubeshop/tracetest/tracetest
```

From here, follow the [official documentation](https://docs.tracetest.io/getting-started/installation#install-the-tracetest-agent) to install the Tracetest Agent and sign up to `app.tracetest.io`. Get the [Tracetest Agent API key](https://docs.tracetest.io/configuration/agent) from the `Settings > Agent` page.

Run the start command under `Settings > Agent`.

```bash
tracetest start --api-key ttagent_1234567890 # replace with your API key
```

A green dot will show up and show the Agent is connected!


*Image 5: Running Tracetest Agent.*

Next, configure Sumo Logic as the tracing backend. Create an [access key in your Sumo Logic account](https://help.sumologic.com/docs/manage/security/access-keys/#create-your-access-key). You’ll use it to connect Tracetest to Sumo Logic.


Alternatively, you can also use the Tracetest CLI. [Configure the Tracetest CLI by creating an environment token](https://docs.tracetest.io/concepts/environment-tokens) under `Settings > Tokens` .

```bash
tracetest configure -t tttoken_1234567890 # replace with your environment token
```

Add the Sumo Logic AccessID and AccessKey to a file called `tracetest.datastore.yaml` .

```yaml
---
type: DataStore
spec:
 id: current
 name: Sumo Logic
 type: sumologic
 sumologic:
   # The URL will differ based on your location. View this
   # docs page to figure out which URL you need:
   # https://help.sumologic.com/docs/api/getting-started/#which-endpoint-should-i-should-use
   url: "https://api.sumologic.com/api/"
   # Create your ID and Key under Administration > Security > Access Keys
   # in your Sumo Logic account:
   # https://help.sumologic.com/docs/manage/security/access-keys/#create-your-access-key
   accessID: "accessid"
   accessKey: "accesskey"
```

Apply the tracing backend configuration.

```bash
tracetest apply datastore -f ./tracetest.datastore.yaml
```

You’re ready to create trace-based tests!

## Create a Trace-based Test in Tracetest

For this example, I’ll showcase this [simple example app for Tracetest and Sumo Logic](https://github.com/kubeshop/tracetest/tree/main/examples/quick-start-sumologic-nodejs). To quickly access the example, you can run the following:

```bash
git clone https://github.com/kubeshop/tracetest.git
cd tracetest/examples/quick-start-sumologic-nodejs

tracetest start --api-key ttagent_1234567890 # replace with your API key
tracetest configure -t tttoken_1234567890 # replace with your environment token
tracetest apply datastore -f ./tracetest.datastore.yaml # replace with your Sumo Logic keys

docker compose up -d --build
tracetest run test -f ./test-api.yaml
```

*Note: To try Tracetest Core, follow the [guide in the example readme](https://github.com/kubeshop/tracetest/tree/main/examples/quick-start-sumologic-nodejs#steps-to-run-tracetest-core).*

Let me explain what’s going on above! 😁

After cloning the sample, make sure to add your Sumo Logic API keys and installation token to the `collector.config.yaml` and `tracetest.datastore.yaml`. Also, get the [Tracetest Agent API key](https://docs.tracetest.io/configuration/agent) from the `Settings > Agent` page and the [Environment Token](https://docs.tracetest.io/concepts/environment-tokens) from `Settings > Tokens`.

To create a test in Tracetest, start by clicking Create > Create New Test > HTTP Request > Next > Add a name for your test > Next > The URL field should be `http://localhost:8080` > Create and Run.

This will trigger the test and display a distributed trace in the Trace tab. You’ll also see the results of the Trace Analyzer. These results show rules and conventions to adhere to while writing code instrumentation.


*Image 6: Trace Analyzer in the Tracetest Web UI. Validate the quality of the code instrumentation.*

Proceed to add a test spec to assert that all HTTP requests return within 500 ms. Click the Test tab and proceed to click the Add Test Spec button.

In the span selector, make sure to add this selector:

```css
span[tracetest.span.type="http"]
```

It will select the HTTP spans.

In the assertion field, add:

```css
attr:tracetest.span.duration < 500ms
```

Save the test spec and publish the test.


*Image 7: Adding assertions to a test in Tracetest.*

If an HTTP span is returning in more than 500ms it will be labeled in red. This is an example of a trace-based test that can assert against every single part of an HTTP transaction, including external API calls. (See image 2)

However, Tracetest cannot give you a historical overview of all test runs nor can you get real-time insight. Let's show how Sumo Logic makes it possible.

## Monitor Trace-based Tests Over Time with Sumo Logic

Because you are sending all traces to Sumo Logic, Tracetest integrates natively to fetch traces from Sumo Logic when running trace-based tests.

Filter for all the traces from the sample Node.js application.


*Image 8: Filter sample application traces.*

You can also view a chart by clicking “Show chart” and view the trace duration and performance over time.


*Image 9: View the chart to see performance and select a distinct trace to drill down.*

From here, you can drill down into the specific trace to troubleshoot. For example, choose a trace that is slow. Once open, the trace waterfall within Sumo Logic can help you pinpoint exactly which span is causing an issue. (Shown in Image 4, above)

## What's next?

Would you like to learn more about Tracetest and what it brings to the table? Check the [docs](https://docs.tracetest.io/examples-tutorials/recipes/running-tracetest-with-sumologic) and try it out by [signing up today](https://app.tracetest.io/)!

Want to learn more about Sumo Logic? [Read more here](https://help.sumologic.com/).

Also, please feel free to join our [Slack Community](https://dub.sh/tracetest-community), give [Tracetest a star on GitHub](https://github.com/kubeshop/tracetest), or schedule a [time to chat 1:1](https://calendar.app.google/8qHAomjQZDKKDARd8).

‍