Event-driven Kubernetes testing with Testkube and Tracetest

We are pleased to announce that Tracetest now works with [Testkube](https://testkube.io/), the Kubernetes-native testing framework. By using a [Testkube executor to build an integration with Tracetest](https://github.com/kubeshop/testkube-executor-tracetest), you can now run event-driven trace-based tests in your Kubernetes cluster!

> ***Note**: Check out this [hands-on Demo example](https://docs.tracetest.io/examples-tutorials/recipes/running-tracetest-with-testkube) of how Tracetest works with Testkube!*

Configuring CI pipelines for running trace-based tests in Kubernetes is tedious work. Especially if you need to trigger tests based on Kubernetes events. Look no further!

Once you’re done reading, you’ll learn how to set up event-driven, trace-based testing in Kubernetes!


## What is Testkube?

[Testkube](https://testkube.io/) is an [open-source project, part of the CNCF landscape,](https://github.com/kubeshop/testkube) and testing framework designed for testers and developers who use Kubernetes. It integrates test orchestration and execution into Kubernetes and your CI/CD/GitOps pipeline. You can automate the execution of your tests, regardless of the testing framework, by using Testkube's executors or creating your own.

By adopting Kubernetes constructs and GitOps, you can perform K8s-native testing. You can use Kubernetes CRDs to manage and store tests, allowing you to validate your applications by executing tests from inside your cluster.

Use any CI/CD framework for any testing scenario. By decoupling them from your CI/CD, you will spend less time integrating different testing tools.

Analyze all your test results in a centralized place. After running your tests, you can view the results in an intuitive UI, regardless of which testing framework you used.

Debug test failures with ease. With Testkube, it's easy to see all the results, logs, and artifacts of your tests in one place.

In addition, Testkube allows you to easily store and download files generated by your tests from your Kubernetes cluster. All files generated from your tests are saved.

## What is Tracetest?

[Tracetest](https://tracetest.io/) is an [open-source project, part of the CNCF landscape](https://github.com/kubeshop/tracetest). It allows you to quickly build integration and end-to-end tests, powered by your distributed traces.

Tracetest uses your existing distributed traces to power trace-based testing with assertions against your trace data at every point of the request transaction. You only need to point Tracetest to your existing trace data source, or send traces to Tracetest directly!

Tracetest makes it possible to:

- Define tests and assertions against every single microservice that a trace goes through.

- Work with your existing distributed tracing solution, allowing you to build tests based on your already instrumented system.

- Define multiple transaction triggers, such as a GET against an API endpoint, a GRPC request, etc.

- Define assertions against both the response and trace data, ensuring both your response and the underlying processes worked correctly, quickly, and without errors.

- Save and run the tests manually or via CI build jobs with the Tracetest CLI.

## Tracetest Now Works with Testkube!

Tracetest now works with Testkube, allowing you to unlock Testkube's capacity with Tracetest, and leverage OpenTelemetry instrumentation in your services to run end-to-end and integration testing. It works thanks to the [Testkube Tracetest Executor](https://github.com/kubeshop/testkube-executor-tracetest) — a test executor to run Tracetest tests with Testkube.

### Why is the Tracetest integration with Testkube important?

By integrating with Testkube you can now add Tracetest to the native CI/CD/GitOps pipeline in your Kubernetes cluster. It allows you to run scheduled test runs on set intervals, as well as asynchronous tests triggered by Kubernetes events. All while following the trace-based testing principle and enabling full in-depth assertions against trace data, not just the response.

Combining the ability to create tests with Tracetest with a Kubernetes-native test runner like Testkube enables you to use the native events from the environment of your Kubernetes cluster as test triggers in your CI/CD/GitOps pipelines!

### Why run trace-based tests?

When running integration tests, you have no way of knowing precisely at which point an HTTP transaction goes wrong in a network of microservices! With tracing enabled, Tracetest can run tests with assertions against existing trace data throughout every service in the entire transaction. You can utilize these tests as part of your CI/CD process to ensure system functionality, and to catch regressions.


## Try Tracetest with Testkube

To run trace-based tests with Tracetest and Testkube, make sure you have these three things installed before starting.

1. A running Kubernetes cluster, either locally or in the cloud

2. [Kubectl](https://kubernetes.io/docs/tasks/tools/)

3. [Helm](https://helm.sh/docs/intro/install/)

### Install Testkube

Testkube is open-source and easy to install. Start by installing the Testkube CLI by following [these instructions](https://docs.testkube.io/getting-started/step1-installing-cli) for your operating system.

```bash

# MacOS example

brew install testkube

```

Install Testkube in your Kubernetes cluster via the [CLI](https://docs.tracetest.io/examples-tutorials/recipes/running-tracetest-with-testkube/#2-install-testkube-in-your-kubernetes-cluster).

From here, follow the [official documentation](https://docs.testkube.io/getting-started/step2-installing-cluster-components) to install the Testkube cluster.

```bash

testkube init

```

Confirm that Testkube is running:

```bash

kubectl get all -n testkube

```

By default, Testkube is installed in the `testkube` namespace. To explore the Testkube dashboard, run the command:

```bash

testkube dashboard

```


### Install Tracetest

Tracetest is open-source and easy to install. Start by installing the Tracetest CLI by following [these instructions](https://docs.tracetest.io/getting-started/installation) for your operating system.

```bash

# MacOS example
‍
brew install kubeshop/tracetest/tracetest

```

> *Note: Check out the [download page](https://tracetest.io/download) for more info.*

From here, follow the [official documentation](https://docs.tracetest.io/getting-started/installation) to install the Tracetest server.

```bash

tracetest server install

[Output]
How do you want to run TraceTest? [type to search]:
 Using Docker Compose
> Using Kubernetes

```

Select `Using Kubernetes`.

```bash

[Output]
Do you have OpenTelemetry based tracing already set up, or would you like us to install a demo tracing environment and app? [type to search]:
 I have a tracing environment already. Just install Tracetest
> Just learning tracing! Install Tracetest, OpenTelemetry Collector and the sample app.

```

Select `Just learning tracing! Install Tracetest, OpenTelemetry Collector and the sample app.`.

Confirm that Tracetest is running:

```bash

kubectl get all -n tracetest

```

By default, Tracetest is installed in the `tracetest` namespace.

To explore the Tracetest Web UI, run the command:

```bash

kubectl --kubeconfig ${HOME}/.kube/config --context kind-kind --namespace tracetest port-forward svc/tracetest 11633

```


Once the server is installed, open the Tracetest Web UI in the browser and follow the instructions for connecting the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) with Tracetest, if it has not been connected already. If you followed the steps above, the Tracetest server will have been automatically provisioned to connect to the OpenTelemetry Collector instance running in the `tracetest` namespace.


If you look closely, you’ll see the OpenTelemetry Sample Configuration from the settings page above matches the `collector.config.yaml` that was generated by the Tracetest CLI when provisioning the Tracetest server.

```yaml

# collector.config.yaml

receivers:
 otlp:
   protocols:
     grpc:
     http:

processors:
 batch:
   timeout: 100ms

exporters:
 otlp/1:
   endpoint: tracetest:21321
   tls:
     insecure: true

service:
 pipelines:
   traces/1:
     receivers: [otlp]
     processors: [batch]
     exporters: [otlp/1]

```

### Create a Test in Tracetest

Start by clicking `Create` > `Create New Test` > `HTTP Request` > `Next` > `Choose Example` (dropdown) > `Pokeshop - List` (generates a sample test from the Tracetest demo) > `Next` > `URL` is prefilled with `http://demo-pokemon-api.demo/pokemon?take=20&skip=0` > `Create and Run`.

This will trigger the test and display a distributed trace in the `Trace` tab to run assertions against.


Proceed to add a test spec to assert all database queries return within `500 ms`. Click the `Test` tab and proceed to click the `Add Test Spec` button.

In the span selector make sure to add this selector:

```css

span[tracetest.span.type="database"]

```

In the assertion field add:

```css

attr:tracetest.span.duration < 500ms

```

Save the test spec and publish the test.


The database spans that are returning in more than `500ms` are labeled in red.


This is an example of a trace-based test that asserts against every single part of an HTTP transaction, including all interactions with the database.

However, Tracetest cannot run this test as part of your CI/CD without integrating it with another tool.

Let's introduce how Testkube makes it possible.

### Deploy the Tracetest Testkube Executor

> ***Note**: As of the latest Teskube release, the Tracetest Testkube executor has been added to the Testkube’s available executors out-of-the-box. If you have an older version of Testkube running, proceed with deploying the Tracetest Testkube executor manually.*

Testkube works with the concept of Executors. An Executor is a wrapper around a testing framework, Tracetest in this case, in the form of a Docker container and runs as a Kubernetes job.

To start you need to register and deploy the Tracetest executor in your cluster using the Testkube CLI. Run the command below in your terminal.

```bash

kubectl testkube create executor --image kubeshop/testkube-executor-tracetest:latest --types "tracetest/test" --name tracetest-executor --icon-uri icon --content-type string --content-type file-uri

[Output]
Executor created tracetest-executor 🥇

```

### Trigger a Trace-based Test in Tracetest with Testkube

In the Tracetest Web UI, click the ⚙️ button in the top right. Then click `Test Definition`.


This will open a YAML definition for the test run.


Save this into a file called `test.yaml`.

```yaml

# test.yaml

type: Test
spec:
 id: RUkKQ_aVR
 name: Pokeshop - List
 description: Get a Pokemon
 trigger:
   type: http
   httpRequest:
     url: http://demo-pokemon-api.demo/pokemon?take=20&skip=0
     method: GET
     headers:
     - key: Content-Type
       value: application/json
 specs:
 - name: Database queries less than 500 ms
   selector: span[tracetest.span.type="database"]
   assertions:
   - attr:tracetest.span.duration  <  500ms

```

Execute the following command to create the test object in Testkube. Do not forget to provide the path to your Tracetest definition file using the `--file` argument, and also the Tracetest server endpoint using the `TRACETEST_ENDPOINT` `--variable`.

Remember that your `TRACETEST_ENDPOINT` should be reachable from Testkube in your cluster. Use your Tracetest service's `CLUSTER-IP:PORT`. E.g: `10.96.93.106:11633`.

```bash

kubectl testkube create test --file ./test.yaml --type "tracetest/test" --name pokeshop-tracetest-test --variable TRACETEST_ENDPOINT=http://CLUSTER-IP:PORT

[Output]
Test created testkube / pokeshop-tracetest-test 🥇

```

Opening the Testkube Dashboard will show the test is created successfully.


Finally, to run the test, execute the following command, or run the test from the Testkube Dashboard.

```bash

kubectl testkube run test --watch pokeshop-tracetest-test

```

Here's what the Testkube CLI will look like if the test fails.

```bash

[Output]

Type:              tracetest/test
Name:              pokeshop-tracetest-test
Execution ID:      641885f39922b3e1003dd5b6
Execution name:    pokeshop-tracetest-test-3
Execution number:  3
Status:            running
Start time:        2023-03-20 16:12:35.268197087 +0000 UTC
End time:          0001-01-01 00:00:00 +0000 UTC
Duration:

 Variables:    1
 - TRACETEST_ENDPOINT = http://10.96.93.106:11633

Getting logs from test job 641885f39922b3e1003dd5b6
Execution completed
🔬 Executing in directory :
$ tracetest test run --server-url http://10.96.93.106:11633 --definition /tmp/test-content737616681 --wait-for-result --output pretty
✘ Pokeshop - List (http://10.96.93.106:11633/test/RUkKQ_aVR/run/2/test)
   ✘ Database queries less than 500 ms
       ✘ #2b213392d0e3ff21
           ✘ attr:tracetest.span.duration  <  500ms (502ms) (http://10.96.93.106:11633/test/RUkKQ_aVR/run/2/test?selectedAssertion=0&selectedSpan=2b213392d0e3ff21)
       ✔ #7e6657f6a43fceeb
           ✔ attr:tracetest.span.duration  <  500ms (72ms)
       ✔ #6ee2fb69690eed47
           ✔ attr:tracetest.span.duration  <  500ms (13ms)
       ✘ #a82c304a3558763b
           ✘ attr:tracetest.span.duration  <  500ms (679ms) (http://10.96.93.106:11633/test/RUkKQ_aVR/run/2/test?selectedAssertion=0&selectedSpan=a82c304a3558763b)
       ✔ #6ae21f2251101fd6
           ✔ attr:tracetest.span.duration  <  500ms (393ms)
       ✔ #2a9b9422af8ba1a8
           ✔ attr:tracetest.span.duration  <  500ms (61ms)
       ✔ #010a8a0d53687276
           ✔ attr:tracetest.span.duration  <  500ms (36ms)
       ✘ #895d66286b6325ae
           ✘ attr:tracetest.span.duration  <  500ms (686ms) (http://10.96.93.106:11633/test/RUkKQ_aVR/run/2/test?selectedAssertion=0&selectedSpan=895d66286b6325ae)

```

And, here's the Testkube Dashboard.


If the test passes, it'll look like this in the terminal.

```bash

[Output]

Type:              tracetest/test
Name:              pokeshop-tracetest-test
Execution ID:      6418873d9922b3e1003dd5b8
Execution name:    pokeshop-tracetest-test-4
Execution number:  4
Status:            running
Start time:        2023-03-20 16:18:05.60245717 +0000 UTC
End time:          0001-01-01 00:00:00 +0000 UTC
Duration:

 Variables:    1
 - TRACETEST_ENDPOINT = http://10.96.93.106:11633

Getting logs from test job 6418873d9922b3e1003dd5b8
Execution completed
🔬 Executing in directory :
$ tracetest test run --server-url http://10.96.93.106:11633 --definition /tmp/test-content1901459587 --wait-for-result --output pretty
✔ Pokeshop - List (http://10.96.93.106:11633/test/RUkKQ_aVR/run/3/test)
   ✔ Database queries less than 500 ms

✅ Execution succeeded
Execution completed ✔ Pokeshop - List (http://10.96.93.106:11633/test/RUkKQ_aVR/run/3/test)
   ✔ Database queries less than 500 ms

```

And, like this in Testkube Dashboard.


## Running Scheduled Trace-based Tests

Integrating with Testkube enables you to add Tracetest to the native CI/CD/GitOps pipeline in your Kubernetes cluster. This allows for scheduled test runs on set intervals, also called [synthetic tests](https://en.wikipedia.org/wiki/Synthetic_monitoring). Now with trace-based testing available, full, in-depth assertions against trace data is available, not just a response.

By using Testkube's [scheduling](https://docs.testkube.io/concepts/scheduling), you can trigger the same test you defined above every minute. It works by providing a CRON schedule. You’ll add an additional `--schedule="*/1 * * * *"` flag.

```bash

kubectl testkube create test --file ./test.yaml --type "tracetest/test" --name pokeshop-tracetest-scheduled-test --schedule="*/1 * * * *" --variable TRACETEST_ENDPOINT=http://CLUSTER-IP:PORT

[Output]
Test created testkube / pokeshop-tracetest-scheduled-test 🥇

```

In your Testkube Dashboard, you'll see this test run continuously and get triggered every minute.


## Running Event-driven Trace-based Tests

Event-based testing in Kubernetes is a critical aspect of ensuring the reliability and performance of microservices in Kubernetes. This testing approach involves observing events that are emitted by various components and services in the system to trigger tests against the system’s components under various conditions.

The main benefit of event-based testing is that it provides a more comprehensive testing approach than traditional unit, integration, and functional testing. With event-based testing, testers can simulate real-world scenarios and test the system's response to different types of input, load, and failure, while also verifying the system's ability to recover from such events.

To effectively perform event-based testing in Kubernetes, you’ll use Testkube as an event monitoring and management system that can capture and analyze the events generated by the system.

This sample will trigger a test when a `deployment` is scaled.

You’ve configured the Tracetest assertions to make sure all database queries finish within `500ms`. Now, define a trigger that will run the trace-based test every time the deployment scales to ensure each replica satisfies the defined assertions.

Define a Test Trigger for the Deployment resource to run the trace-based test when a `deployment-scale-update` event occurs:

```yaml

# testkube-trigger.yaml

apiVersion: tests.testkube.io/v1
kind: TestTrigger
metadata:
 name: deployment-scale-update-trigger
 namespace: testkube
spec:
 resource: deployment
 resourceSelector:
   labelSelector:
     matchLabels:
       app.kubernetes.io/instance: demo
 event: deployment-scale-update
 action: run
 execution: test
 testSelector:
   name: pokeshop-tracetest-test
   namespace: testkube

```

Save the file, name it `testkube-trigger.yaml` and apply it.

```bash

kubectl apply -f ./testkube-trigger.yaml

```


This will configure a trigger to run the test every time the demo app deployment is scaled. Try it yourself by running:

```bash

kubectl scale deployment demo-pokemon-api --replicas=4 -n demo

```

Moving back to the `testkube dashboard` you’ll see the test was triggered by the event.


## Running Tests in a Test Suite or Transaction

Running singular, isolated, event-driven tests has its own important use cases and values. But, in the wild, you’ll more often rely on chaining multiple tests together into a transaction or test suite.

Both Testkube and Tracetest support such logical constructs.

In Tracetest they’re called [transactions](https://docs.tracetest.io/concepts/transactions/). In Testkube they’re called [test suites](https://docs.testkube.io/concepts/test-suites/testsuites-creating).

### Tracetest Transactions

Running end-to-end tests is not simple. It requires configuration before the actual test can be run, such as creating a new user or removing all items from a cart. Therefore, it's important to be able to execute multiple steps as part of your transaction. Tracetest introduces the concept of **Transactions** to achieve this goal.

A transaction is a group of steps executed in a defined order, where each step is a test that can access information exported by previous tests.

The main benefit of using transactions is the ability to chain tests together and use values obtained in one test as input for a subsequent test.

When a test is executed within a transaction, if it generates any outputs, the test outputs will be injected into the transaction context environment. After the outputs are injected, all subsequent tests to be run within the transaction will be able to reference those values with `env:VARIABLE_NAME`.

> ***Note**: Outputs generated by steps don't modify the selected environment. It only modifies the transaction run context object.*

Tracetest allows tests to declare `outputs`. An output is a value that is extracted from a trace by providing a [selector](https://docs.tracetest.io/concepts/transactions/selectors) to choose which spans to use to get the information from, and an [expression](https://docs.tracetest.io/concepts/transactions/expressions) to get the value from the selected spans.

### Run a Tracetest Transaction

Start by adding a new test by clicking `Create` > `Create New Test` > `HTTP Request` > `Next` > `Choose Example` (dropdown) > `Pokeshop - Add` (generates a sample test from the Tracetest demo) > `Next` > `URL` is prefilled with `http://demo-pokemon-api.demo/pokemon` > `Create and Run`.

The request body will be populated with this JSON.

```bash

{"name":"meowth","type":"normal","imageUrl":"https://assets.pokemon.com/assets/cms2/img/pokedex/full/052.png","isFeatured":true}

```

Navigate to `Test` > `Outputs` and click `Add Test Output`. Select the `create pokeshop.pokemon` database span.

```bash

span[tracetest.span.type="database" name="create pokeshop.pokemon" db.system="postgres" db.name="pokeshop" db.user="ashketchum" db.operation="create" db.sql.table="pokemon"]

```

The attribute to export the Pokemon’s `id` as a value is:

```bash

attr:db.result | json_path 'id'

```

Finally give it a name:

```bash

add_pokemon_db_result_id

```


Save the test output and publish the variables. Here’s what the YAML definition of this test looks like:

```yaml

type: Test
spec:
 id: RAt3JIfVg
 name: Pokeshop - Add - In Transaction
 description: Add a Pokemon
 trigger:
   type: http
   httpRequest:
     url: http://demo-pokemon-api.demo/pokemon
     method: POST
     headers:
     - key: Content-Type
       value: application/json
     body: '{"name":"meowth","type":"normal","imageUrl":"https://assets.pokemon.com/assets/cms2/img/pokedex/full/052.png","isFeatured":true}'
 outputs:
 - name: add_pokemon_db_result_id
   selector: span[tracetest.span.type="database" name="create pokeshop.pokemon" db.system="postgres" db.name="pokeshop" db.user="ashketchum" db.operation="create" db.sql.table="pokemon"]
   value: attr:db.result | json_path 'id'

```

Now you can edit the `List Pokemon` test to use this variable in an assertion.

Select the HTTP span.

```bash

span[tracetest.span.type="http" name="GET /pokemon?take=20&skip=0" http.method="GET"]

```

Set this assertion.

```bash

attr:http.response.body contains '${env:add_pokemon_db_result_id}'

```

Finally, give the assertion a name.


Save the test spec and click publish.

Now you can add a transaction and see how it works together. Click `Create` > `Create New Transaction` > Give it a name > `Create`.

Once created, add the `Add` and `List` tests to the transaction list. You’ll see the defined variables on the right, below the execution steps. The `List` test is passing as it is correctly asserting that the result of the `List` request contains the exported variable from the `Add` test.


Copy the transaction definition file and save it as a file named `transaction.yaml`.

```yaml

# transaction.yaml

type: Transaction
spec:
 id: MnUSxIf4g
 name: Add+List Pokemon
 steps:
 - RAt3JIfVg
 - UQe_xIBVg

```

Now you can create a test in Testkube to trigger the transaction.

```bash

kubectl testkube create test --file ./transaction.yaml --type "tracetest/test" --name pokeshop-tracetest-transaction --variable TRACETEST_ENDPOINT=http://CLUSTER-IP:PORT

[Output]
Test created testkube / pokeshop-tracetest-transaction 🥇

```

Trigger the transaction in the same way as you did the test.

```bash

kubectl testkube run test --watch pokeshop-tracetest-transaction

[Output]
Type:              tracetest/test
Name:              pokeshop-tracetest-transaction
Execution ID:      641b19a9183f824d354b51d1
Execution name:    pokeshop-tracetest-transaction-2
Execution number:  2
Status:            running
Start time:        2023-03-22 15:07:21.796730144 +0000 UTC
End time:          0001-01-01 00:00:00 +0000 UTC
Duration:

 Variables:    1
 - TRACETEST_ENDPOINT = http://10.96.4.232:11633

Getting logs from test job 641b19a9183f824d354b51d1
Execution completed
🔬 Executing in directory :
$ tracetest test run --server-url http://10.96.4.232:11633 --definition /tmp/test-content1800237311 --wait-for-result --output pretty
✔ Add+List Pokemon (http://10.96.4.232:11633/transaction/MnUSxIf4g/run/11)
✔ Pokeshop - Add - In Transaction (http://10.96.4.232:11633/test/RAt3JIfVg/run/19/test)
✔ Pokeshop - List - In Transaction (http://10.96.4.232:11633/test/UQe_xIBVg/run/10/test)
✔ Make sure pokemon id from the ADD is contained in the LIST

✅ Execution succeeded
Execution completed ✔ Add+List Pokemon (http://10.96.4.232:11633/transaction/MnUSxIf4g/run/11)
✔ Pokeshop - Add - In Transaction (http://10.96.4.232:11633/test/RAt3JIfVg/run/19/test)
✔ Pokeshop - List - In Transaction (http://10.96.4.232:11633/test/UQe_xIBVg/run/10/test)
✔ Make sure pokemon id from the ADD is contained in the LIST

```

### Run a Testkube Test Suite

[Test suites](https://docs.testkube.io/concepts/test-suites/testsuites-creating) in Testkube are a way to orchestrate different test steps and entirely different testing frameworks to run in a suite. Your front-end team uses Cypress for browser tests, while the back-end team uses Tracetest. You may also have Postman collections testing various parts of your apps.

With test suites, you can orchestrate different test steps and combine them to run in a sequence. Even if each team runs its tests on its own, they can ultimately be combined and triggered from one location by a test suite.

## Learn More About Kubernetes Testing

Combined, Testkube and Tracetest provide a comprehensive testing solution for Kubernetes applications. By utilizing Testkube triggers, you can automatically initiate trace-based tests with Tracetest, ensuring that your services are adhering to defined SLAs.

Tracetest provides detailed distributed trace data, allowing you to gain insight into the behavior of your application at a granular level, and ultimately create assertions against this data to write bullet-proof tests.

Would you like to learn more about Tracetest and what it brings to the table? Check the [docs](https://docs.tracetest.io/examples-tutorials/recipes/running-tracetest-with-lightstep/) and try it out today by [downloading](https://tracetest.io/download) it today!

To explore more options Testkube gives you, check out the documentation on [test triggers](https://docs.testkube.io/concepts/triggers). They enable you to trigger tests based on Kubernetes events. Want to learn more about Testkube? [Read more here](https://testkube.io/get-started).

Also, please feel free to join our [Slack community](https://dub.sh/tracetest-community), give [Tracetest a star on GitHub](https://github.com/kubeshop/tracetest), or schedule a [time to chat 1:1](http://calendly.com/ken-kubeshop/otel-user-interview-w-tracetest).