Responsive

Observability-driven development with Go and Tracetest

Observability-driven development with Go and Tracetest
Jan 19, 2023
22 min
read
Adnan Rahic
Staff Developer Advocate
Tracetest

Hands-on tutorial covering observability-driven development, how to develop microservices with Go & how to run trace-based tests with Tracetest.

Share on X
Share on LinkedIn
Share on Reddit
Share on HackerNews
Copy URL

Table of Contents

Get Started with Observability-Driven Development

Try Managed Tracetest

Subscribe to our monthly newsletter to stay up to date with all things Tracetest.


We’re entering a new era of [observability-driven development](https://stackoverflow.blog/2022/10/12/how-observability-driven-development-creates-elite-performers/). ODD uses OpenTelemetry instrumentation as assertions in tests!


> [Here's](https://twitter.com/mipsytipsy/status/1203578004397383681) an awesome explanation on Twitter!


This is driving a new culture of [trace-based testing](https://tracetest.io/blog/introducing-tracetest-trace-based-testing-with-opentelemetry). With trace-based testing, you can generate integration tests from OpenTelemetry-based traces, enforce quality, encourage velocity, and increase test coverage in microservices and distributed apps.


Today, you’ll learn to build a distributed system with Go and Docker. You’ll instrument it with OpenTelemetry traces and use Tracetest to run trace-based tests on top of OpenTelemetry infrastructure.


![https://res.cloudinary.com/djwdcmwdz/image/upload/v1671802096/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1671802081166_b2euer.png](https://res.cloudinary.com/djwdcmwdz/image/upload/v1671802096/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1671802081166_b2euer.png)


We’ll follow observability-driven development principles and showcase why it’s powerful in today’s world of developing distributed systems in the cloud.


By the end of this tutorial you’ll learn observability-driven development, how to develop microservices with Go, and how to run trace-based tests with Tracetest.


To check out the [entire code, jump over to GitHub](https://github.com/kubeshop/tracetest/tree/main/examples/observability-driven-development-go-tracetest/bookstore).


## What are we building?


The distributed app we’re building will have two microservices and a dedicated module for OpenTelemetry trace instrumentation.


We’ll use Go to write the code and Docker Compose to deploy the microservices. The app itself is a bookstore that shows books with their price and availability.


We will follow observability-driven development best practices of first writing a test, then writing the code and adding OpenTelemetry instrumentation to validate the test spec, and finally run trace-based tests with Tracetest and make sure they pass.


The tutorial consists of 3 parts.


The first part will be dedicated to configuring the basic bookstore infrastructure and books service, setting up OpenTelemetry instrumentation, and installing Tracetest.


In the second part we will focus on running ODD tests hands-on and creating trace-based tests for the books service.


The third, and final, part will focus on creating the availability service and covering it with tests.


Before we begin, let’s quickly explain what OpenTelemetry and Tracetest are.


## What is OpenTelemetry?


[OpenTelemetry](https://opentelemetry.io/) is an observability framework that assist in generating and capturing telemetry data from cloud-native software.


OpenTelemetry gathers observability data including traces, metrics, and logs.


OpenTelemetry is a community-driven open-source project and as of August 2021 is a CNCF incubating project. OpenTelemetry is the [second most active CNCF project](https://all.devstats.cncf.io/d/1/activity-repository-groups?orgId=1) behind Kubernetes.


These are the three components we’ll use in this guide:


- [OpenTelemetry Go APIs and SDKs](https://opentelemetry.io/docs/instrumentation/go/manual/) for generating and emitting telemetry

- [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) to receive, process and export telemetry data

- [OTLP protocol](https://opentelemetry.io/docs/reference/specification/protocol/) for transmitting telemetry data


Because OpenTelemetry is a framework, you need a data store to persist traces. We’ll demo how to use Jaeger as the trace data store and the OpenTelemetry Collector as the gateway to funnel traces to Jaeger for storage.


## What is Tracetest?


[Tracetest](https://tracetest.io/) uses your existing [OpenTelemetry](https://opentelemetry.io/docs/getting-started/) traces to power trace-based testing with assertions against your trace data at every point of the request transaction.


You’ll point Tracetest to the existing Jaeger trace data source. Tracetest will then pull traces from Jaeger while running integration tests to run assertions against the trace data itself.


There is an option to bypass using a trace data store entirely, and send traces to Tracetest right away by configuring the OpenTelemetry collector as [explained in our docs](https://docs.tracetest.io/configuration/connecting-to-data-stores/opentelemetry-collector).


With that, we’re ready to start coding!


## Setting up the infrastructure for observability-driven development


This section will explain the initial configuration of both the bookstore app and Tracetest. We’ll use Docker Compose for the infrastructure. Once done, you’ll have a running app with Tracetest configured for running tests.


### Installing Tracetest for local development


You can follow the [sample code we’ve prepared for part 1 of this tutorial](https://github.com/kubeshop/tracetest/tree/main/examples/observability-driven-development-go-tracetest/bookstore/part1). Follow the instructions below:


```bash

git clone git@github.com:kubeshop/tracetest.git

cd examples/observability-driven-development-go-tracetest/bookstore/part1

docker compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up

```



Let’s walk through the installation step-by-step.


Start by installing the Tracetest CLI. It’s the easiest way of getting started with Tracetest.


```bash

brew install kubeshop/tracetest/tracetest

```

Note: Follow [this guide](https://docs.tracetest.io/getting-started/installation) to install for your specific operating system.


Once the CLI is installed, create a directory called `bookstore`, and install a Tracetest server.


```bash

tracetest server install

```


Follow the prompts and install a bare-bones setup with just Tracetest. This will generate an empty `docker-compose.yaml` file and a `./tracetest/` directory that contains another `docker-compose.yaml`.


### Configuring OpenTelemetry Collector and Jaeger with Tracetest


Let’s edit the `docker-compose.yaml` to add OpenTelemetry Collector and Jaeger.


```yaml

# ./tracetest/docker-compose.yaml


services:

   jaeger:

       healthcheck:

           test:

               - CMD

               - wget

               - --spider

               - localhost:16686

           timeout: 3s

           interval: 1s

           retries: 60

       image: jaegertracing/all-in-one:latest

       networks:

           default: null

       restart: unless-stopped

   otel-collector:

       command:

           - --config

           - /otel-local-config.yaml

       depends_on:

           jaeger:

               condition: service_started

       image: otel/opentelemetry-collector:0.54.0

       networks:

           default: null

       volumes:

           - type: bind

             source: tracetest/otel-collector.yaml

             target: /otel-local-config.yaml

             bind:

               create_host_path: true

   postgres:

       environment:

           POSTGRES_PASSWORD: postgres

           POSTGRES_USER: postgres

       healthcheck:

           test:

               - CMD-SHELL

               - pg_isready -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"

           timeout: 5s

           interval: 1s

           retries: 60

       image: postgres:14

       networks:

           default: null

   tracetest:

       depends_on:

           otel-collector:

               condition: service_started

           postgres:

               condition: service_healthy

       extra_hosts:

           host.docker.internal: host-gateway

       healthcheck:

           test:

               - CMD

               - wget

               - --spider

               - localhost:11633

           timeout: 3s

           interval: 1s

           retries: 60

       image: kubeshop/tracetest:v0.9.3

       networks:

           default: null

       ports:

           - mode: ingress

             target: 11633

             published: 11633

             protocol: tcp

       volumes:

           - type: bind

             source: tracetest/tracetest.yaml

             target: /app/config.yaml

networks:

   default:

       name: _default

```


Let me explain what’s going on in the `docker-compose.yaml` file:


- We hook up the OpenTelemetry Collector to act as a gateway for all the traces our app will generate and Jaeger as a trace data store.

- The OpenTelemetry Collector will receive all traces from our Go microservices and send them to Jaeger.

- We will then configure Tracetest to fetch trace data from Jaeger when running trace-based tests.


Make sure your config files for Tracetest and the OpenTelemetry Collector match the sample code. First copy-paste this into your `otel-collector.yaml`.


```yaml

# ./tracetest/otel-collector.yaml


exporters:

   jaeger:

       endpoint: jaeger:14250

       tls:

           insecure: true

processors:

   batch:

       timeout: 100ms

receivers:

   otlp:

       protocols:

           grpc: null

           http: null

service:

   pipelines:

       traces:

           exporters:

               - jaeger

           processors:

               - batch

           receivers:

               - otlp

```


Now, from the `bookstore` directory, start Docker Compose to test the Tracetest installation.


```bash

docker compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up

```


This command will spin up the infrastructure, and expose Tracetest on port `11633`. Open up `http://localhost:11633/` in your browser.


Configure your trace data store to point to Jaeger in the Web UI.


![](https://res.cloudinary.com/djwdcmwdz/image/upload/v1674152344/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1674152336009_jcwb5v.png)


You can also [configure Jaeger via the CLI](https://docs.tracetest.io/cli/creating-data-stores#jaeger).


### Adding the `books` microservice


In the `bookstore` directory, create a `books` directory, and initialize a Go module.


```bash

cd ./books

go mod init github.com/your-username/bookstore/books

```


In the `books` directory create a `main.go` file. Paste this code into the `main.go`:


```go

// ./books/main.go


package main


import (

 "context"

 "fmt"

 "io"

 "log"

 "net/http"

 "time"


 "github.com/gorilla/mux"

 "go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux"

 "go.opentelemetry.io/otel"

 "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"

 "go.opentelemetry.io/otel/propagation"

 "go.opentelemetry.io/otel/sdk/resource"

 sdktrace "go.opentelemetry.io/otel/sdk/trace"

 semconv "go.opentelemetry.io/otel/semconv/v1.12.0"

 "go.opentelemetry.io/otel/trace"

 "google.golang.org/grpc"

 "google.golang.org/grpc/credentials/insecure"

)


const svcName = "books"


var tracer trace.Tracer


func newExporter(ctx context.Context) (sdktrace.SpanExporter, error) {

 ctx, cancel := context.WithTimeout(ctx, time.Second)

 defer cancel()

 conn, err := grpc.DialContext(ctx, "otel-collector:4317", grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithBlock())

 if err != nil {

   return nil, fmt.Errorf("failed to create gRPC connection to collector: %w", err)

 }


 traceExporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithGRPCConn(conn))

 if err != nil {

   return nil, fmt.Errorf("failed to create trace exporter: %w", err)

 }


 return traceExporter, nil

}


func newTraceProvider(exp sdktrace.SpanExporter) *sdktrace.TracerProvider {

 // Ensure default SDK resources and the required service name are set.

 r, err := resource.Merge(

   resource.Default(),

   resource.NewWithAttributes(

     semconv.SchemaURL,

     semconv.ServiceNameKey.String(svcName),

   ),

 )


 if err != nil {

   panic(err)

 }


 tp := sdktrace.NewTracerProvider(

   sdktrace.WithBatcher(exp),

   sdktrace.WithResource(r),

 )


 otel.SetTextMapPropagator(

   propagation.NewCompositeTextMapPropagator(

     propagation.TraceContext{},

     propagation.Baggage{},

   ),

 )


 return tp

}


func main() {

 ctx := context.Background()


 exp, err := newExporter(ctx)

 if err != nil {

   log.Fatalf("failed to initialize exporter: %v", err)

 }


 // Create a new tracer provider with a batch span processor and the given exporter.

 tp := newTraceProvider(exp)


 // Handle shutdown properly so nothing leaks.

 defer func() { _ = tp.Shutdown(ctx) }()


 otel.SetTracerProvider(tp)


 // Finally, set the tracer that can be used for this package.

 tracer = tp.Tracer(svcName)


 r := mux.NewRouter()

 r.Use(otelmux.Middleware(svcName))


 r.HandleFunc("/books", booksListHandler)


 http.Handle("/", r)


 log.Fatal(http.ListenAndServe(":8001", nil))

}


func booksListHandler(w http.ResponseWriter, r *http.Request) {

 _, span := tracer.Start(r.Context(), "Books List")

 defer span.End()


 io.WriteString(w, "Hello!\n")

}

```


Let’s walk through what’s happening in the `./books/main.go` file:


- The `newExporter` function is defining how to export trace data and to forward it to the OpenTelemetry Collector we have running on `otel-collector:4317`.

- The `newTraceProvider` function is initializing the tracer that we use to instrument the code.

- The main function is initializing everything and defines an HTTP route called `/books` to trigger a `booksListHandler`.

- The `booksListHandler` function will return a simple `"Hello!"` string. It also starts the OpenTelemetry tracer and defines a span called `"Books List"` .


With all this added, fetch Go dependencies by running this command in the terminal from the `books` directory:


```bash

go mod tidy

```


This will generate a `go.sum` file.


Lastly, add a `books` service to the `docker-compose.yaml` file in the `bookstore` directory. This is the root `docker-compose.yaml` file, not the one inside the `./tracetest/` directory.


```yaml

# ./docker-compose.yaml


services:

 books:

   image: your_username/books

   build:

     args:

       SERVICE: books

   ports:

     - 8001:8001

   depends_on:

     otel-collector:

       condition: service_started

```


Next, create a `Dockerfile` and paste this code into it:


```docker

# ./Dockerfile


FROM golang:1.19


ARG SERVICE


WORKDIR /app/${SERVICE}


COPY ./${SERVICE}/go.* /app/${SERVICE}

RUN go mod download


COPY ./${SERVICE}/* /app/${SERVICE}

RUN go build -o /app/server .


ENTRYPOINT [ "/app/server" ]

```


Finally, restart Docker Compose to try out the `books` service.


```bash

docker compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up

```


### Running a trace-based test in the Tracetest web UI


With the Tracetest service running on port `11633`, open it up on `http://localhost:11633/` in your browser.


![https://res.cloudinary.com/djwdcmwdz/image/upload/v1672045450/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1672045442359_i6bx8r.png](https://res.cloudinary.com/djwdcmwdz/image/upload/v1672045450/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1672045442359_i6bx8r.png)


Create a new HTTP test. Give it a name and make sure to set the URL to `http://books:8001/books`.


![https://res.cloudinary.com/djwdcmwdz/image/upload/v1672045846/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1672045826154_sgl1re.png](https://res.cloudinary.com/djwdcmwdz/image/upload/v1672045846/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1672045826154_sgl1re.png)


Click create. This will trigger the test to run right away.


![https://res.cloudinary.com/djwdcmwdz/image/upload/v1672045925/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1672045918599_aano1x.png](https://res.cloudinary.com/djwdcmwdz/image/upload/v1672045925/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1672045918599_aano1x.png)


The test will return a `200` status code. Next, we need to add assertions against the trace data to make sure our OpenTelemetry tracing instrumentation works in our Go code.


Open the `Trace` tab and let’s start by adding a status code assertion. Click on the `Tracetest trigger` span. In the left navigation, select `tracetest.response.status` and click `Create test spec`. If you’re writing the assertion by hand, make sure to preface the attribute with a `attr:` to enable autocompletion when selecting what attribute to assert on.


![https://res.cloudinary.com/djwdcmwdz/image/upload/v1672046199/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1672046190788_jesaei.png](https://res.cloudinary.com/djwdcmwdz/image/upload/v1672046199/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1672046190788_jesaei.png)


Save the test spec and add another assertion to the `Books list` span. This time add the attribute called `attr:tracetest.selected_spans.count = 1`.


![https://res.cloudinary.com/djwdcmwdz/image/upload/v1672046555/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1672046532214_knnl08.png](https://res.cloudinary.com/djwdcmwdz/image/upload/v1672046555/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1672046532214_knnl08.png)


Save and publish the test specs. Re-run the test.


![https://res.cloudinary.com/djwdcmwdz/image/upload/v1672046663/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1672046656385_wjjtws.png](https://res.cloudinary.com/djwdcmwdz/image/upload/v1672046663/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1672046656385_wjjtws.png)


You now have passing tests that ensure the service responds with a `200` status code and verifies the OpenTelemetry manual code instrumentation works!


### Running a trace-based test with the Tracetest CLI


Let’s re-trace, no pun intended, our steps with the Tracetest CLI.


Create an `e2e` directory in the `bookstore` directory. Create a file called `books-list.yaml`. This will contain a test definition we will trigger with the CLI.


Paste this code into the `books-list.yaml`:


```yaml

# ./e2e/books-list.yaml


type: Test

spec:

 id: k6hEWU54R

 name: Books Listing

 description: Try books service

 trigger:

   type: http

   httpRequest:

     url: http://books:8001/books

     method: GET

     headers:

     - key: Content-Type

       value: application/json

 specs:

   - selector: span[name="Tracetest trigger"]

     assertions:

       - attr:tracetest.response.status = 200

   - selector: span[name="Books List"]

     assertions:

       - attr:tracetest.selected_spans.count = 1

```


Take a moment to read the code. You see the assertions match what we just added in the Tracetest Web UI.


To trigger the test from the command line, first configure the Tracetest CLI. Make sure to point the CLI to the URL where the Tracetest service is running. In this sample it’s `http://localhost:11633/`.


```bash

tracetest configure


[Output]

Enter your Tracetest server URL [http://localhost:11633]: http://localhost:11633


[Output]

Enable analytics? [Y/n]: Yes

```


Now, we can run the test. From the `bookstore` dir, run:


```bash

tracetest test run -d ./e2e/books-list.yaml -w


[Output]

✔ Books Listing (http://localhost:11633/test/k6hEWU54R/run/2/test)

```


Clicking the link will open the test run in the Web UI.


With the initial setup done, we’re ready to move on and tackle getting hands-on with observability-driven development!


## Hands-on observability-driven development


To follow along, you can [check out the sample code we’ve prepared for part 2.1](https://github.com/kubeshop/tracetest/tree/main/examples/observability-driven-development-go-tracetest/bookstore/part2.1). Follow the instructions below:


```bash

git clone git@github.com:kubeshop/tracetest.git

cd examples/observability-driven-development-go-tracetest/bookstore/part2.1

docker compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up

###

tracetest test run -d ./e2e/books-list.yaml -w

```



To get started let’s first add more detailed assertions to the `books-list.yaml`, and make our test fail.


Open the `books-list.yaml` file and add a custom attribute called `attr:books.list.count`. This means we are expecting the `Books List API` test to return 3 books.


```yaml

# ./e2e/books-list.yaml


# ...


 specs:

   - selector: span[name="Tracetest trigger"]

     assertions:

       - attr:tracetest.response.status = 200

   - selector: span[name="Books List"]

     assertions:

       - attr:tracetest.selected_spans.count = 1

       - attr:books.list.count = 3

```


Jump back into the terminal, and run the test again.


```bash

tracetest test run -d ./e2e/books-list.yaml -w


[Output]

✘ Books Listing (http://localhost:11633/test/k6hEWU54R/run/1/test)

 ✔ span[name="Tracetest trigger"]

   ✔ #ebae1f382ecb81f6

     ✔ attr:tracetest.response.status = 200 (200)

 ✘ span[name="Books List"]

   ✘ #f6c5fa3aa5527a7a

     ✔ attr:tracetest.selected_spans.count = 1 (1)

     ✘ attr:books.list.count = 3 (http://localhost:11633/test/k6hEWU54R/run/1/test?selectedAssertion=1&selectedSpan=f6c5fa3aa5527a7a)

```


The `"Books List"` span now fails the test.


![](https://res.cloudinary.com/djwdcmwdz/image/upload/v1671107357/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1671107290230_mjegwh.png)


In true ODD fashion, let’s add the code to satisfy the test spec.


We need to add a `getBooks` function to retrieve the books and make sure to add OpenTelemetry instrumentation to validate that it is indeed returning the books in the form of an array.


Open up the `./books/main.go` file. We will edit the `booksListHandler` function, and add a `getBooks` function that simulates getting books from a database.


```go

// ./books/main.go


// ...


func booksListHandler(w http.ResponseWriter, r *http.Request) {

 ctx, span := tracer.Start(r.Context(), "Books List")

 defer span.End()


 books, err := getBooks(ctx)

 if err != nil {

   w.WriteHeader(http.StatusInternalServerError)

   io.WriteString(w, "cannot read books DB")

   return

 }


 // This is how we instrument the code with OpenTelemetry

 // This is the attribute we run the assertion against

 span.SetAttributes(

   attribute.Int("books.list.count", len(books)),

 )


 jsonBooks, err := json.Marshal(books)

 if err != nil {

   w.WriteHeader(http.StatusInternalServerError)

   io.WriteString(w, "cannot json encode books DB")

   return

 }


 w.Write(jsonBooks)

}


type book struct {

 ID    string `json:"id"`

 Name  string `json:"name"`

 Price int    `json:"price"`

}


// Mocking a database request

func getBooks(ctx context.Context) ([]book, error) {

 return []book{

   {"1", "Harry Potter", 0},

   {"2", "Foundation", 0},

   {"3", "Moby Dick", 0},

 }, nil

}

```


Save the changes, and restart Docker Compose. Now, run the same test.


```bash

tracetest test run -d ./e2e/books-list.yaml -w


[Output]

✔ Books Listing (http://localhost:11633/test/k6hEWU54R/run/1/test)

```


The test passes. Clicking the link in the test will open up the Tracetest Web UI.


![https://res.cloudinary.com/djwdcmwdz/image/upload/v1672054765/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1672054760024_mkzyc0.png](https://res.cloudinary.com/djwdcmwdz/image/upload/v1672054765/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1672054760024_mkzyc0.png)


We’re starting to look like ODD pros right now! But, we’re not done yet. We want to add an availability check to our bookstore. What if a book is not in stock? We need to be able to check that.


## Setting up observability-driven tests for multiple microservices


To follow along, you can [check out the sample code we’ve prepared for part 2.2](https://github.com/kubeshop/tracetest/tree/main/examples/observability-driven-development-go-tracetest/bookstore/part2.2). Follow the instructions below:


```bash

git clone git@github.com:kubeshop/tracetest.git

cd examples/observability-driven-development-go-tracetest/bookstore/part2.2

docker compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up

###

tracetest test run -d ./e2e/books-list.yaml -w

```


First add another book entry to the `getBooks` function. We’ll add an availability check that will confirm it is out of stock.


```go

// ./books/main.go


// ...


func getBooks(ctx context.Context) ([]book, error) {

 return []book{

   {"1", "Harry Potter", 0},

   {"2", "Foundation", 0},

   {"3", "Moby Dick", 0},

   {"4", "The art of war", 0}, // Add this book

 }, nil

}

```


Once again, open up the `books-list.yaml`. Let’s add assertions for the availability.


```yaml

# books-list.yaml


# ...

 

 specs:

   - selector: span[name="Tracetest trigger"]

     assertions:

       - attr:tracetest.response.status = 200

   - selector: span[name="Books List"]

     assertions:

       - attr:tracetest.selected_spans.count = 1

       - attr:books.list.count = 3

   

   # This selector will look for a descendant of the

   # "Books List" span called "Availability Check"

   - selector: span[name = "Books List"] span[name = "Availability Check"]

     assertions:

       - attr:tracetest.selected_spans.count = 4

```


We want to make sure that an availability check is performed for every single book from the `getBooks` function.


Re-running the test will cause it to fail because of the availability check, as expected.


```bash

tracetest test run -d ./e2e/books-list.yaml -w


[Output]

✘ Books Listing (http://localhost:11633/test/k6hEWU54R/run/2/test)

 ✔ span[name="Tracetest trigger"]

   ✔ #b81c6b68711908e1

     ✔ attr:tracetest.response.status = 200 (200)

 ✔ span[name="Books List"]

   ✔ #392fcfe7690310d8

     ✔ attr:tracetest.selected_spans.count = 1 (1)

     ✔ attr:books.list.count = 3 (3)

 ✘ span[name = "Books List"] span[name = "Availability Check"]

   ✘ #meta

     ✘ attr:tracetest.selected_spans.count = 4 (0) (http://localhost:11633/test/k6hEWU54R/run/2/test?selectedAssertion=2)

```


Next, let’s write the code to send an HTTP request to an `availability` microservice.


```go

// ./books/main.go


// ...


func httpError(span trace.Span, w http.ResponseWriter, msg string, err error) {

 w.WriteHeader(http.StatusInternalServerError)

 io.WriteString(w, msg)

 span.RecordError(err)

 span.SetStatus(codes.Error, msg)

}


func booksListHandler(w http.ResponseWriter, r *http.Request) {

 ctx, span := tracer.Start(r.Context(), "Books List")

 defer span.End()


 books, err := getAvailableBooks(ctx)

 if err != nil {

   httpError(span, w, "cannot read books DB", err)

   return

 }


 span.SetAttributes(

   attribute.Int("books.list.count", len(books)),

 )


 jsonBooks, err := json.Marshal(books)

 if err != nil {

   httpError(span, w, "cannot json encode books", err)

   return

 }


 w.Write(jsonBooks)

}


func getAvailableBooks(ctx context.Context) ([]book, error) {

 books, err := getBooks(ctx)

 if err != nil {

   return nil, err

 }


 availableBook := make([]book, 0, len(books))

 for _, book := range books {

   available, err := isBookAvailable(ctx, book.ID)

   if err != nil {

     return nil, err

   }


   if !available {

     continue

   }

   availableBook = append(availableBook, book)

 }


 return availableBook, nil

}


var httpClient = &http.Client{

 Transport: otelhttp.NewTransport(http.DefaultTransport),

}


func isBookAvailable(ctx context.Context, bookID string) (bool, error) {

 ctx, span := tracer.Start(ctx, "Availability Request", trace.WithAttributes(

   attribute.String("bookID", bookID),

 ))

 defer span.End()


 url := "http://availability:8000/" + bookID

 req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)

 resp, err := httpClient.Do(req)

 if err != nil {

   span.RecordError(err)

   span.SetStatus(codes.Error, "cannot do request")

   return false, err

 }


 if resp.StatusCode == http.StatusNotFound {

   span.SetStatus(codes.Error, "not found")

   return false, nil

 }


 stockBytes, err := io.ReadAll(resp.Body)

 if err != nil {

   span.RecordError(err)

   span.SetStatus(codes.Error, "cannot read response body")

   return false, err

 }


 stock, err := strconv.Atoi(string(stockBytes))

 if err != nil {

   span.RecordError(err)

   span.SetStatus(codes.Error, "cannot parse stock value")

   return false, err

 }


 return stock > 0, nil

}

```


Let me explain the code in detail.


- We’re adding a `isBookAvailable` function that checks if a book is available based on a provided `bookID`. It calls the `"http://availability:8000/"` endpoint and appends a `bookID` value.

- The `isBookAvailable` function is then used in the `getAvailableBooks` function that iterates through the books from the `getBooks` function.

- The `booksListHandler` function now calls the `getAvailableBooks` function instead of calling `getBooks`.

- The `httpError` is just a helper function.


*Note: Do not forget to re run `go mod tidy` if you change code that requires modules to be downloaded. Make sure to also restart Docker Compose after editing code!*


Let’s re-run the test.


```bash

tracetest test run -d ./e2e/books-list.yaml -w


[Output]

✘ Books Listing (http://localhost:11633/test/qasYcU54R/run/1/test)

 ✘ span[name="Tracetest trigger"]

   ✘ #2f9bc366597fb472

     ✘ attr:tracetest.response.status = 200 (500) (http://localhost:11633/test/qasYcU54R/run/1/test?selectedAssertion=0&selectedSpan=2f9bc366597fb472)

 ✘ span[name="Books List"]

   ✘ #1f0e9347869fd8c2

     ✔ attr:tracetest.selected_spans.count = 1 (1)

     ✘ attr:books.list.count = 3 (http://localhost:11633/test/qasYcU54R/run/1/test?selectedAssertion=1&selectedSpan=1f0e9347869fd8c2)

 ✘ span[name = "Books List"] span[name = "Availability Check"]

   ✘ #meta

     ✘ attr:tracetest.selected_spans.count = 4 (0) (http://localhost:11633/test/qasYcU54R/run/1/test?selectedAssertion=2)

```


We’re getting a different error now! The response status of the `"Tracetest trigger"` span equals `500`. Hmm, not great, right?


Wrong! We’re on the correct path! The test is failing because we added code that sends an HTTP request to an availability service that does not exist. Let’s fix that.


Next up, creating an availability service.


## Trace-based testing across multiple services


To follow along, you can [check out the sample code we’ve prepared for part 3.1](https://github.com/kubeshop/tracetest/tree/main/examples/observability-driven-development-go-tracetest/bookstore/part3.1). Follow the instructions below:


```bash

git clone git@github.com:kubeshop/tracetest.git

cd examples/observability-driven-development-go-tracetest/bookstore/part3.1

docker compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up

###

tracetest test run -d ./e2e/books-list.yaml -w

```


When developing distributed apps and microservices it’s best practice to extract the OpenTelemetry instrumentation to a dedicated module.


This will let you import the OpenTelemetry configuration into all of your microservices without duplicating code.


Let’s start with pulling out the OpenTelemetry SDKs from the `./books/main.go` and putting them in a dedicated file called `instrumentation.go`.


Create a `lib` directory in the root of the `bookstore` directory.


Initialize a module with:


```bash

cd ./lib

go mod init github.com/your-username/bookstore/lib

```


*Note: File paths can be tricky in Go. Makes sure the name of the file path matches the location on GitHub when you import the module in your microservices.*


Once you create the Go module, create another directory called `instrumentation`. Add a single file called `instrumentation.go`.


Remove the OpenTelemetry instrumentation code from `./books/main.go`, and add i to the `./lib/instrumentation/instrumentation.go`.


```go

// ./lib/instrumentation/instrumentation.go


package instrumentation


import (

 "context"

 "fmt"

 "time"


 "go.opentelemetry.io/otel"

 "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"

 "go.opentelemetry.io/otel/propagation"

 "go.opentelemetry.io/otel/sdk/resource"

 sdktrace "go.opentelemetry.io/otel/sdk/trace"

 semconv "go.opentelemetry.io/otel/semconv/v1.12.0"

 "google.golang.org/grpc"

 "google.golang.org/grpc/credentials/insecure"

)


func NewExporter(ctx context.Context) (sdktrace.SpanExporter, error) {

 ctx, cancel := context.WithTimeout(ctx, time.Second)

 defer cancel()

 conn, err := grpc.DialContext(ctx, "otel-collector:4317", grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithBlock())

 if err != nil {

   return nil, fmt.Errorf("failed to create gRPC connection to collector: %w", err)

 }


 traceExporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithGRPCConn(conn))

 if err != nil {

   return nil, fmt.Errorf("failed to create trace exporter: %w", err)

 }


 return traceExporter, nil

}


func NewTraceProvider(svcName string, exp sdktrace.SpanExporter) *sdktrace.TracerProvider {

 // Ensure default SDK resources and the required service name are set.

 r, err := resource.Merge(

   resource.Default(),

   resource.NewWithAttributes(

     semconv.SchemaURL,

     semconv.ServiceNameKey.String(svcName),

   ),

 )


 if err != nil {

   panic(err)

 }


 tp := sdktrace.NewTracerProvider(

   sdktrace.WithBatcher(exp),

   sdktrace.WithResource(r),

 )


 otel.SetTextMapPropagator(

   propagation.NewCompositeTextMapPropagator(

     propagation.TraceContext{},

     propagation.Baggage{},

   ),

 )


 return tp

}

```


Don’t forget to run `go mod tidy` in the terminal from the `./lib` folder to make sure the dependencies are downloaded and saved. You can now safely commit and push this code to GitHub. This will let you download it and use it in both the `books` and `availability` microservices. Lets move on to updating the `books` service first.


```go

// ./books/main.go


package main


import (

 "context"

 "encoding/json"

 "io"

 "log"

 "net/http"

 "strconv"


 "github.com/gorilla/mux"


 // Add the instrumentation module from lib

 // Make sure to first push the module to GitHub

 // Watch out to get the directory tree and name to match

 "github.com/your-username/bookstore/lib/instrumentation"


 "go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux"

 "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

 "go.opentelemetry.io/otel"

 "go.opentelemetry.io/otel/attribute"

 "go.opentelemetry.io/otel/codes"

 "go.opentelemetry.io/otel/trace"

)


const svcName = "books"


var tracer trace.Tracer


func main() {

 ctx := context.Background()


 // Calling the "instrumentation" module

 exp, err := instrumentation.NewExporter(ctx)

 if err != nil {

   log.Fatalf("failed to initialize exporter: %v", err)

 }


 // Calling the "instrumentation" module

 // Create a new tracer provider with a batch span processor and the given exporter.

 tp := instrumentation.NewTraceProvider(svcName, exp)


 // Handle shutdown properly so nothing leaks.

 defer func() { _ = tp.Shutdown(ctx) }()


 otel.SetTracerProvider(tp)


 // Finally, set the tracer that can be used for this package.

 tracer = tp.Tracer(svcName)


 r := mux.NewRouter()

 r.Use(otelmux.Middleware(svcName))


 r.HandleFunc("/books", booksListHandler)


 http.Handle("/", r)


 log.Fatal(http.ListenAndServe(":8001", nil))

}


// ...

```


The `./books/main.go` looks exactly the same except for removing the OpenTelemetry instrumentation code and importing the `./lib/instrumentation` module instead. Make sure to edit the import to use the instrumentation module you pushed to GitHub.


Then, we use the `instrumentation` object to call the `NewExporter` and `NewTraceProvider` methods.


To make sure the behavior is the same after this change let’s restart Docker Compose and re-run the same test as above.


```bash

tracetest test run -d ./e2e/books-list.yaml -w


[Output]

✘ Books Listing (http://localhost:11633/test/qasYcU54R/run/1/test)

 ✘ span[name="Tracetest trigger"]

   ✘ #831e781a89050f81

     ✘ attr:tracetest.response.status = 200 (500) (http://localhost:11633/test/qasYcU54R/run/1/test?selectedAssertion=0&selectedSpan=831e781a89050f81)

 ✘ span[name="Books List"]

   ✘ #9f05d0fe6d4966e6

     ✔ attr:tracetest.selected_spans.count = 1 (1)

     ✘ attr:books.list.count = 3 (http://localhost:11633/test/qasYcU54R/run/1/test?selectedAssertion=1&selectedSpan=9f05d0fe6d4966e6)

 ✘ span[name = "Books List"] span[name = "Availability Check"]

   ✘ #meta

     ✘ attr:tracetest.selected_spans.count = 4 (0) (http://localhost:11633/test/qasYcU54R/run/1/test?selectedAssertion=2)

```


Awesome! We’re getting the same issue as before. Wild, isn’t it? I’m cheering because we’re getting the same issue as before!


With that out of the way, time to build our `availability` service.


## Hands-on observability-driven tests across multiple microservices


Check out the [sample code we’ve prepared for part 3.2](https://github.com/kubeshop/tracetest/tree/main/examples/observability-driven-development-go-tracetest/bookstore/part3.2) to follow along. Follow the instructions below:


```bash

git clone git@github.com:kubeshop/tracetest.git

cd examples/observability-driven-development-go-tracetest/bookstore/part3.2

docker compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up

###

tracetest test run -d ./e2e/books-list.yaml -w

```


To begin, we need a new directory for our additional microservice. Create an `availability` directory in the `bookstore` directory. Initialize a Go module.


```bash


cd ./availability

go mod init github.com/your-username/bookstore/availability

```


Create a file called `./availability/main.go`. Paste this code into it.


```go

package main


import (

 "context"

 "io"

 "log"

 "net/http"


 // Make sure this module matches the lib/instrumentation

 // module from the previous section!

 "github.com/your-username/bookstore/lib/instrumentation"

 "github.com/gorilla/mux"

 "go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux"

 "go.opentelemetry.io/otel"

 "go.opentelemetry.io/otel/attribute"

 "go.opentelemetry.io/otel/codes"

 "go.opentelemetry.io/otel/trace"

)


const svcName = "availability"


var tracer trace.Tracer


func main() {

 ctx := context.Background()


 exp, err := instrumentation.NewExporter(ctx)

 if err != nil {

   log.Fatalf("failed to initialize exporter: %v", err)

 }


 // Create a new tracer provider with a batch span processor and the given exporter.

 tp := instrumentation.NewTraceProvider(svcName, exp)


 // Handle shutdown properly so nothing leaks.

 defer func() { _ = tp.Shutdown(ctx) }()


 otel.SetTracerProvider(tp)


 // Finally, set the tracer that can be used for this package.

 tracer = tp.Tracer(svcName)


 r := mux.NewRouter()

 r.Use(otelmux.Middleware(svcName))


 r.HandleFunc("/{bookID}", stockHandler)


 http.Handle("/", r)


 log.Fatal(http.ListenAndServe(":8000", nil))

}


var books = map[string]string{

 "1": "10",

 "2": "1",

 "3": "5",

 "4": "0",

}


func stockHandler(w http.ResponseWriter, r *http.Request) {

 _, span := tracer.Start(r.Context(), "Availability Check")

 defer span.End()


 vars := mux.Vars(r)

 bookID, ok := vars["bookID"]

 if !ok {

   span.SetStatus(codes.Error, "no bookID in URL")

   w.WriteHeader(http.StatusBadRequest)

   io.WriteString(w, "missing bookID in URL")

   return

 }


 // The span we will run an assertion against

 span.SetAttributes(

   attribute.String("bookID", bookID),

 )


 stock, ok := books[bookID]

 if !ok {

   span.SetStatus(codes.Error, "book not found")

   w.WriteHeader(http.StatusNotFound)

   io.WriteString(w, "book not found")

   return

 }


 w.WriteHeader(http.StatusOK)

 io.WriteString(w, stock)

}

```


As always, run `go mod tidy` to generate a `go.sum` file and download modules.


Let me explain the code:


- We are using the `NewExporter` and `NewTraceProvider` in the `main` function just as we did in the `./books/main.go`.

- We are running an HTTP server on port `8000` that expects a `bookID` as a parameter.

- The HTTP route `/{bookID}` will trigger a `stockHandler` function. This function checks if the book is in stock or not.


Wonderful! With the availability service added, we need to add it to the `docker-compose.yaml` as well.


```yaml

services:

 books:

   image: your_username/books

   build:

     args:

       SERVICE: books

   ports:

     - 8001:8001

   depends_on:

     - otel-collector


 availability:

   image: your_username/availability

   build:

     args:

       SERVICE: availability

   depends_on:

     - otel-collector

```


The `availability` service will use the same `Dockerfile` as the `books` service.


That’s all! We’re done! Let’s restart Docker Compose and see if the `"Books Listing"` test passes.


For reference, here’s the full `./e2e/books-list.yaml` test file we’re running:


```yaml

# ./e2e/books-list.yaml


type: Test

spec:

 id: qasYcU54R

 name: Books Listing

 description: Try books service

 trigger:

   type: http

   httpRequest:

     url: http://books:8001/books

     method: GET

     headers:

     - key: Content-Type

       value: application/json

 specs:

   - selector: span[name="Tracetest trigger"]

     assertions:

       - attr:tracetest.response.status = 200

   - selector: span[name="Books List"]

     assertions:

       - attr:tracetest.selected_spans.count = 1

       - attr:books.list.count = 3

   - selector: span[name = "Books List"] span[name = "Availability Check"]

     assertions:

       - attr:tracetest.selected_spans.count = 4

```


In your terminal, run:


```bash

tracetest test run -d ./e2e/books-list.yaml -w


[Output]

✔ Books Listing (http://localhost:11633/test/qasYcU54R/run/1/test)

```


Clicking the link will open the Tracetest Web UI and show the assertions in detail.


![https://res.cloudinary.com/djwdcmwdz/image/upload/v1671802096/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1671802081166_b2euer.png](https://res.cloudinary.com/djwdcmwdz/image/upload/v1671802096/Blogposts/observability-driven-development-with-go-and-tracetest/screely-1671802081166_b2euer.png)


We can clearly see how an availability check was triggered four times. One time for every book in the list.


## Conclusion


You’ve learned how to practice observability-driven development with Go and Tracetest across multiple microservices by using a dedicated OpenTelemetry instrumentation module. Well done!


To continue learning about observability-driven development, check out our [3-part video tutorial](https://www.youtube.com/watch?v=LXVBnPqxspY&t=0s) about observability-driven development with Go and Tracetest!


Give Tracetest a try in your own applications and tracing infrastructure by following either our [quick start guide](https://docs.tracetest.io/getting-started/installation), which sets you up with the CLI tooling and the Tracetest server in a few steps, or our [detailed guide](https://docs.tracetest.io/getting-started/detailed-installation), for more details. Feel free to [give us a star on GitHub](https://github.com/kubeshop/tracetest) as well!


By practicing observability-driven development and trace-based testing best practices we want you to have a more developer friendly experience by increasing your test coverage, freeing yourself from manual testing procedures, and identifying bottlenecks you didn’t even know existed.


We’d love to hear about your ODD success stories in [Slack](https://dub.sh/tracetest-community)! We truly value your feedback, so don't be shy!