Announcing the Tracetest integration with Keptn

Today, we're happily announcing that Tracetest now works with Keptn, a control plane and micro platform for cloud-native application delivery and operations.

> *Check out this [hands-on Demo example](https://github.com/kubeshop/tracetest/tree/main/examples/keptn-integration) of how Tracetest works with Keptn!*

## What is Keptn?

[Keptn](https://keptn.sh/) is an [open-source CNCF project](https://github.com/keptn/keptn). It’s a control plane and micro platform for cloud-native application delivery and operations following best practices of putting observability data into the driving seat of making automated decisions.

Keptn augments any standard Kubernetes cluster to support delivery based on automated quality gates and self-healing operations workflows. Enabled by a fully event-driven architecture, Keptn can be easily extended to integrate with new toolchains. The declarative approach versus traditional scripts enables enterprise-wide deployments and easy maintenance.

If you are interested in Keptn, also take a look at the latest evolution called [KLT (Keptn Lifecycle Toolkit)](https://lifecycle.keptn.sh/). KLT uses a K8s Operator approach which brings automated deployment observability and orchestration to your GitOps deployments.

## What is Tracetest?

[Tracetest](https://tracetest.io/) is an [open-source CNCF project](https://github.com/kubeshop/tracetest). It allows you to quickly build integration and end-to-end tests, powered by your OpenTelemetry traces.

Tracetest leverages your OpenTelemetry traces to enable trace-based testing with assertions against your trace data at every point of the request transaction. To use Tracetest, simply point it to your existing trace data source or send traces directly to Tracetest.

We make it possible to

- Define tests and assertions for every microservice that a request passes through.

- This solution works with your existing distributed tracing, allowing you to build tests based on your already-instrumented system.

- Define multiple transaction triggers, such as a GET against an API endpoint, a GRPC request, etc.

- Return both the response data and a distributed trace.

- Define assertions against both the response and trace data. This will ensure that both your response and underlying processes work correctly, quickly, and without errors.

- Save tests.

- Run the tests manually or via CI build jobs with the Tracetest CLI.

## Tracetest Now Works with Keptn!

[Tracetest now works with Keptn](https://artifacthub.io/packages/keptn/keptn-integrations/tracetest), by using the [Keptn Job Executor Service](https://github.com/keptn-contrib/job-executor-service) plugin. You can now upload a Tracetest test definition and a CLI configuration to a service and run a test with this simple job definition below.

```yaml

apiVersion: v2
actions:
- name: "Run tracetest on your service"
 events:
   - name: "sh.keptn.event.test.triggered"
 tasks:
   - name: "Run tracetest"
     files:
       - data/test-definition.yaml
       - data/tracetest-cli-config.yaml
     image: "kubeshop/tracetest:latest"
     cmd:
       - tracetest
     args:
       - --config
       - /keptn/data/tracetest-cli-config.yaml
       - test
       - run
       - --definition
       - /keptn/data/test-definition.yaml
       - --wait-for-result

```

**Why is it important?**

When running integration tests, it can be difficult to pinpoint where, within a network of microservices, an HTTP transaction fails. With tracing enabled, Tracetest can run tests with assertions against existing data throughout every service involved in the transaction. These tests can be used as part of your CI/CD process to ensure system functionality and catch regressions.


By enabling trace-based testing from within your Keptn application life-cycle orchestration, you can ensure every part of your multi-stage delivery is covered to match SLOs.


‍

Read more about the Keptn integration with Tracetest, [here](https://tracetest.io/integrations/keptn).

## Try Tracetest with Keptn

Install Keptn in your Kubernetes cluster by following the official docs, [here](https://keptn.sh/docs/1.0.x/install/helm-install/). This shows both a simple Helm install and a full install of Keptn. Next, you need to [install](https://github.com/keptn-contrib/job-executor-service/blob/main/docs/INSTALL.md) the `Job Executor Service 0.3.x`.

Finally, on your local machine, you need:

1. `Keptn CLI` [installed](https://keptn.sh/docs/1.0.x/install/cli-install/)

2. and already [authenticated](https://keptn.sh/docs/1.0.x/install/authenticate-cli-bridge/) with Keptn API.

With Keptn set up, you can configure Keptn and Tracetest.

Tracetest is open-source and easy to install. Start by installing the Tracetest CLI:

```yaml

brew install kubeshop/tracetest/tracetest

```

> **Note**: Check out the [download page](https://tracetest.io/download) for more info.

From here, follow the [official documentation](https://docs.tracetest.io/getting-started/installation) to install the Tracetest server in your Kubernetes cluster. Once the server is installed, open up Tracetest Web UI in the browser and follow the instructions for connecting your trace data store with Tracetest.

## Configure Tracetest with Keptn

Keptn works with two [concepts](https://keptn.sh/docs/concepts/glossary/). A **Project** and a **Service**. A **Project** is an element that holds multiple services forming an application in stages. And, a **Service** is the smallest deployable unit and is deployed in all project stages according to the defined order.

Usually, these resources are managed by Keptn during **Sequences** — a set of tasks for realizing a delivery or operations process.

Start by creating a **Shipyard** to contain the **Project**. Then, create a skeletal [`shipyard.yaml`](https://github.com/kubeshop/tracetest/tree/main/examples/keptn-integration/shipyard.yaml) file with the following content:

```yaml

apiVersion: "spec.keptn.sh/0.2.2"
kind: "Shipyard"
metadata:
 name: "<shipyard_name>"
spec:
 stages:
   - name: "production"
     sequences:
       - name: "validate-app"
         tasks:
           - name: "test"

```

Create a new project using the `shipyard.yaml` file:

```bash

keptn create project <project_name> -y -s shipyard.yaml

```

> **Note:** Keptn may ask you to have a Git repository for this project to enable GitOps. If so, you need to create an empty Git repository and a Git token and pass it through the flags `--git-remote-url`, `--git-user`, and `--git-token`. More details about this setup can be seen on [Keptn docs/Git-based upstream](https://keptn.sh/docs/1.0.x/manage/git_upstream).


Now you can create a service:

```bash

keptn create service <service_name> --project <project_name> -y

```

Now, we will set up a job associated with the service, listening to a task event.

Create the [`tracetest-cli-config.yaml`](https://github.com/kubeshop/tracetest/tree/main/examples/keptn-integration/tracetest-cli-config.yaml) configuration file for the Tracetest CLI in your current directory, identifying the Tracetest instance that should run the tests:

```yaml

scheme: http
endpoint: tracetest.tracetest.svc.cluster.local:11633

```

Create the [`test-definition.yaml`](https://github.com/kubeshop/tracetest/tree/main/examples/keptn-integration/test-definition.yaml) Tracetest test definition in your current directory:

```yaml

type: Test
spec:
 id: apdCx-h4g
 name: Pokeshop - List Pokemon
 description: Get a Pokemon
 trigger:
   type: http
   httpRequest:
     url: http://demo-pokemon-api.demo/pokemon?take=20&skip=0
     method: GET
     headers:
     - key: Content-Type
       value: application/json
 specs:
 - selector: span[name = “Tracetest trigger”]
   assertions:
   - attr:tracetest.span.duration < 500ms

```

> **Note:** The Tracetest test definition will vary based on your own test definitions.

Add these files as resources for your service in the `production` stage:

```bash

keptn add-resource --project <project_name> --service <service_name> --stage production --resource test-definition.yaml --resourceUri data/test-definition.yaml

keptn add-resource --project <project_name> --service <service_name> --stage production --resource tracetest-cli-config.yaml --resourceUri data/tracetest-cli-config.yaml

```

These files will be located in the folder `data` and will be injected into our Keptn job that we will set up in the next step.

Create a [`job-config.yaml`](https://github.com/kubeshop/tracetest/tree/main/examples/keptn-integration/job-config.yaml):

```yaml

apiVersion: v2
actions:
- name: "Run tracetest on your service"
 events:
   - name: "sh.keptn.event.test.triggered"
 tasks:
   - name: "Run tracetest"
     files:
       - data/test-definition.yaml
       - data/tracetest-cli-config.yaml
     image: "kubeshop/tracetest:latest"
     cmd:
       - tracetest
     args:
       - --config
       - /keptn/data/tracetest-cli-config.yaml
       - test
       - run
       - --definition
       - /keptn/data/test-definition.yaml
       - --wait-for-result

```

This job will run Tracetest every time a `test` event happens, listening to the event `sh.keptn.event.test.triggered`. This event is emitted by the `test` task on the `validate-app` sequence.

Add this job as a resource in Keptn:

```bash

keptn add-resource --project <project_name> --service <service_name> --stage production --resource job-config.yaml --resourceUri job/config.yaml

```


To guarantee that the job will be called by Keptn when you execute the `production` sequence, you need to configure the `Job Executor Service` integration in your project to listen to `sh.keptn.event.test.triggered` events.

You do that through the Keptn Bridge, the Web UI, by going to the project, choosing the `Settings` option, and `Integrations`.

Choose the `job-executor-service` integration, and add a subscription to the event `sh.keptn.event.test.triggered` and your project.

Finally, to see the integration running, we only need to execute the following command:

```bash

keptn trigger sequence validate-app --project <project_name> --service <service_name> --stage production

```

Now you should be able to see the sequence running in Keptn Bridge.


## What's next?

Would you like to learn more about Tracetest and what it brings to the table? Check the [docs](https://docs.tracetest.io/tools-and-integrations/keptn) and try it out today by [downloading](https://tracetest.io/download) it today! Want to learn more about Keptn? [Read more here](https://keptn.sh/), or check out [more integrations here](https://keptn.sh/docs/integrations/).

Also, please feel free to join our [Slack community](https://dub.sh/tracetest-community), give [Tracetest a star on GitHub](https://github.com/kubeshop/tracetest), or schedule a [time to chat 1:1](http://calendly.com/ken-kubeshop/otel-user-interview-w-tracetest).