OpenTelemetry Collector Basics and Tracing

Buenas buenas 👋🏽 Welcome to a new entry in the Tracetest Tip series, where we provide valuable tricks and shortcuts to make your OpenTelemetry journey successful by saving you precious time and avoiding you banging your head into the keyboard, you can leave that to us 😉!!

So, let’s get into the nitty and gritty. Today we are talking about one of the core participants in the OpenTelemetry ecosystem, the **OpenTelemetry Collector**.

The OpenTelemetry Collector is an open-source tool built specifically as a vendor-agnostic system to **`receive, process and export`** your telemetry data in a performant and extendible way.

Many amazing people from different backgrounds have come together to make it as simple and as powerful as possible so you can focus on what matters, which is instrumenting your Systems!

## The three amigos: receivers, processors and exporters 🤠

As mentioned in the previous section, the Collector’s purpose is composed by three things, which are:

**Receivers:** Allow you to configure the protocols in which the telemetry data is going to be ingested into the pipelines, if you are using OTLP SDKs, Datadog, OpenSearch or AWS X-Ray, you can configure it so the collector understands how to get the data.

**Processors:** These are completely optional, but powerful configuration methods to filter, format and manipulate the telemetry data after it is received. It goes from tail-sampling, inserting or deleting attributes to batch processors.

**Exporters:** This is where you can specify where to send your telemetry data. In today’s tech world, there are many Tracing Backend Vendors and each company decides where to store data.  Here is the spot where you would specify endpoints, output protocols, auth, etc.

```yaml
# protocols, ports and rules on how to receive data from the exterior
receivers:
 # receives telemetry DATA from with the OpenTelemetry format
 otlp:
   protocols:
   # from both GRPC (4317) and HTTP (4318, CORS can also be configured!) ports
     grpc:
     HTTP:
# data manipulation and filtering are part of processors
processors:
 # waits a predefined set amount of time to send spans in bursts
 batch:
   timeout: 100ms
# sending the telemetry data to be stored, processed or to another instance of the collector
exporters:
 # logging the tracing data to the terminal - good for debugging 🐛
 logging:
   loglevel: debug
 # sending the data to the tracetest agent GRPC OTLP port
 otlp/1:
   endpoint: tracetest-agent:4317
   tls:
     insecure: true
service:
 pipelines:
   # combining the three prev configurations into a single pipeline
   traces/1:
     receivers: [otlp]
     processors: [batch]
     exporters: [otlp/1]
```

In other words, how the data comes in, if it needs to do some fancy work on it, and where it should be sent afterward!

> A combination of any number of the three amigos is called a `pipeline`

Let’s start with some scenarios, shall we?!

## Scenario 1: I’m new to OpenTelemetry, what should I do?

For those who got sent on a quest by someone at their company or want to try something simple just to see how it looks I’d recommend:

1. Write a simple Node.js/Golang/NetCore HTTP Service, a simple “hello world” is enough.

```jsx
const express = require("express")
const app = express()
app.get("/", (req, res) => {
 setTimeout(() => {
   res.send("Hello World")
 }, 1000);
})
app.listen(8080, () => {
 console.log(`Listening for requests on http://localhost:8080`)
})
```

2. Add auto-instrumentation. This is a very good start to having your app generating telemetry data.

```jsx
const opentelemetry = require('@opentelemetry/sdk-node')
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node')
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const dotenv = require("dotenv")
dotenv.config()
const sdk = new opentelemetry.NodeSDK({
 traceExporter: new OTLPTraceExporter(),
 instrumentations: [getNodeAutoInstrumentations()],
 serviceName: 'quick-start-nodejs',
})
sdk.start()
```

> Start the app by requiring the instrumentation code `node -r ./tracing.otel.grpc.js app.js`

3. With docker and docker-compose, create the services for your app.

```yaml
version: '3'
services:
 app:
   image: quick-start-nodejs
   build: .
   ports:
     - "8080:8080"
   environment:
     - OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=${OTEL_EXPORTER_OTLP_TRACES_ENDPOINT}
   depends_on:
     otel-collector:
       condition: service_started
 otel-collector:
   image: otel/opentelemetry-collector-contrib:0.100.0
   command:
     - "--config"
     - "/otel-local-config.yaml"
   volumes:
     - ./collector.config.yaml:/otel-local-config.yaml
```

2. Include the tracing backend of your choice (Jeager and/or Tempo are really good to start).

```yaml
jaeger:
   image: jaegertracing/all-in-one:latest
   ports:
     - 14250:14250
     - 16685:16685
     - 16686:16686
   healthcheck:
     test: ["CMD", "wget", "--spider", "localhost:16686"]
     interval: 1s
     timeout: 3s
     retries: 60
```

3. Include the OpenTelemetry Collector Image and config.

```yaml
# protocols, ports and rules on how to receive data from the exterior
receivers:
 # receives telemetry DATA from with the OpenTelemetry format
 otlp:
   protocols:
   # from both GRPC (4317) and HTTP (4318, CORS can also be configured!) ports
     grpc:
     http:
# data manipulation and filtering are part of processors
processors:
 # waits a predefined set amount of time to send spans in bursts
 batch:
   timeout: 100ms
# sending the telemetry data to be stored, processed or to another instance of the collector
exporters:
 # logging the tracing data to the terminal - good for debugging 🐛
 logging:
   loglevel: debug
 # sending the data to the jaeger instance GRPC OTLP port
 otlp:
   endpoint: jaeger:4317
   tls:
     insecure: true
service:
 pipelines:
 # combining the three prev configurations into a single pipeline
   traces/1:
     receivers: [otlp]
     processors: [batch]
     exporters: [logging, otlp]
```

> If you want to find a more robust example, visit [the OpenTelemetry Demo](https://opentelemetry.io/docs/demo/).

> If you want to have everything described here done for you plus some trace-based testing, take a [look at one of our examples!](https://docs.tracetest.io/examples-tutorials/recipes/running-tracetest-with-jaeger)

There you go, a simple scenario showcasing the three basic Collector concepts and a way to familiarize yourself.

## Scenario 2: The company I work for uses X vendor as tracing backend but it’s not using the OpenTelemetry Protocols

Migrating from one tool to another comes with its set of challenges. Happily, the collector offers different options to achieve this in the smoothest way possible.

But first, let's talk about something significant. When starting the OpenTelemetry journey, you’ll find two different versions of the Collector.

1. The [official Collector](https://github.com/open-telemetry/opentelemetry-collector) contains the core functionality alongside basic integrations which are primarily vendor-agnostic and focused solely on OpenTelemetry tools.

2. The [Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) version includes a set of extended integrations for different vendors, helping teams smoothly transition from their current setup to OpenTelemetry.

With this in mind, you can decide which distribution of the Collector fits your use case better and pick the appropriate integrations for it.

Here’s a quick Collector config that receives OpenTelemetry and exports telemetry data to some of the most common vendors:

```yaml
exporters:
 # AWS X-RAY - https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/awsxrayexporter
 awsxray:
   region: ${AWS_REGION}
 # AURE MONITOR - https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/azuremonitorexporter
 azuremonitor:
   connection_string: ${env:INSTRUMENTATION_KEY}
 # DATA DOG - https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/datadogexporter/examples/otlp.yaml
 datadog:
   api:
     site: ${env:DD_SITE}
     key: ${env:DD_API_KEY}
 # Splunk - https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/splunkhecexporter
 splunk_hec:
   retry_on_failure:
     max_elapsed_time: 60
```

## Scenario 3:  The System instrumentation is not built on using OTEL SDKs, but we want to migrate.

If your team wants to gradually move to OpenTelemetry but the system is instrumented using other vendors’ SDKs and tools, multiple supported `receivers` by the Collector’s Contrib distribution come in handy for this use case.

Normally, each vendor has its own proprietary protocols, ports, and methods to receive your telemetry data. In this case, by using the OpenTelemetry Collector, you can configure it in a way that it can hook into the same protocols, ports and methods the vendor would using custom `receivers`.

Here’s an example of a Collectors configuration to support incoming vendor data from the most common use cases:

```yaml
receivers:
 # AWS X-RAY - https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awsxrayreceiver
 awsxray:
   endpoint: 0.0.0.0:2000
   transport: udp
   proxy_server:
     endpoint: 0.0.0.0:2000
     proxy_address: ""
     tls:
       insecure: false
       server_name_override: ""
     region: ""
     role_arn: ""
     aws_endpoint: ""
     local_mode: false
 # AZURE MONITOR - https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/azuremonitorreceiver
 azuremonitor:
   subscription_id: "${subscription_id}"
   auth: "managed_identity"
   client_id: "${env:AZURE_CLIENT_ID}"
 # DATA DOG - https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver
 datadog:
   endpoint: localhost:8126
   read_timeout: 60s
 # Splunk - https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver
 splunk_hec:
   access_token_passthrough: true
   tls:
     cert_file: /test.crt
     key_file: /test.key
   raw_path: "/raw"
   hec_metadata_to_otel_attrs:
     source: "mysource"
     sourcetype: "mysourcetype"
     index: "myindex"
     host: "myhost"
   ack:
     extension: ack/in_memory
```

## Final Thoughts

Take it slow! Starting with a basic setup that works is better than getting overwhelmed by the number of possibilities and permutations that exist. Just remember the three basic things `receivers`, `processors` and `exporters`.

Trust the magic phrase 🪄! As developers when looking for quick wins and simple ways of doing things we hate hearing “it depends”, but in this case understanding the current state of your system, the architecture, which third-party services are in use, etc, can be helpful to configure the OpenTelemetry Collector in a way that you can get the most out of it!

Have a question? Feel free to join our [Slack community](https://dub.sh/tracetest-community) or [schedule a time to chat 1:1](https://calendly.com/ken-kubeshop/tracetest-walkthrough).