Testing AWS Lambda & Serverless with OpenTelemetry

A year ago, I published a blog post about how to use [Tracetest with Lambda and AWS](https://tracetest.io/blog/trace-based-testing-aws-lambda-with-tracetest). That post took me on an adventure as I tried to figure out the best way to create a simple, repeatable, easy-to-understand approach to setting up a complete FaaS (Function as a Service) distributed system. I thought I’d figured it all out. However, reading it again, I believe that wasn’t the case 😅.

Since then, the ecosystem has changed. Using the [Serverless Framework](https://www.serverless.com/) makes deployment simpler. We released the managed [Tracetest App](https://app.tracetest.io/) making any serverless-based systems simpler to instrument and test. You can now [test public-facing apps](https://docs.tracetest.io/concepts/cloud-agent) with no infra overhead!

Buckle up and get ready for the second round; this time improved, faster, bigger and more efficient! With explosions… Ok, just kidding! 💥💣

## Why should I care about this?

You know the drill. Cloud Native systems can become a pain to debug. Information moves around to different places, pipelines, services, workers, message brokers, you name it.

The story often repeats itself. Our small team must provision infrastructure, write code, and figure out bugs often working late into the hours of a Friday night to solve production issues with only logs to accompany us in those dark moments.

_**After that, we promise ourselves that we’ll come back and fix all of it… this time for real.**_

Well, that day has come, because today I’m going to show you how you and your team can easily instrument a Node.js Serverless App using the revamped [Pokeshop Demo Serverless](https://github.com/kubeshop/pokeshop/tree/master/serverless) implementation.

And all this while I teach you how to take it to the next level by using trace-based testing with Tracetest tools and libraries! 🕺🏽

## I have never heard about Tracetest and Trace-based testing. What is that?

Excellent question my friend! 🤝🏽

Tracetest is an observability-enabled testing tool for Cloud Native architectures, leverages these distributed traces as part of testing, providing you with better visibility and testability enabling you to run trace-based tests.

Trace-based testing is the technique of running validations against the telemetry data generated by the distributed system’s instrumented services.


## What are we building today?

Today, we are going to be provisioning the Serverless version of the [Pokeshop Demo API](https://docs.tracetest.io/live-examples/pokeshop/overview) which is a fully distributed and instrumented Node.js application running on AWS Lambda. Here’s the list of resources that will be used outside of the regular Serverless setup.

- AWS RDS (Postgres).
- AWS SQS.
- AWS ElastiCache.


The networking will be handled by the `serverless-vpc` plugin, which is a simple way to spin off the required resources to manage ingress and egress rules, as well as protecting our precious services behind a private network!

## Requirements

### Tracetest Account:

- Sign up to [`app.tracetest.io`](https://app.tracetest.io/) or follow the [get started](https://docs.tracetest.io/getting-started/installation) docs.
- Create an [environment](https://docs.tracetest.io/concepts/environments).
- Select `Application is publicly accessible` to get access to the environment's [Tracetest Cloud Agent endpoint](https://docs.tracetest.io/concepts/cloud-agent).
- Select OpenTelemetry as the tracing backend.
- Create an [environment token](https://docs.tracetest.io/concepts/environment-tokens).

### AWS:

- Have access to an [AWS Account](https://aws.amazon.com/).
- Install and configure the [AWS CLI](https://aws.amazon.com/cli/).
- Use a role that is allowed to provision the required resources.

## What are the steps to run it myself?

For the self-made developers out there, here’s what you need to run to do it yourself 🦾.

First, clone the Pokeshop repo.

```bash
git clone https://github.com/kubeshop/pokeshop.git
cd pokeshop/serverless
```

Then, follow the instructions to run the deployment and the trace-based tests:

1. Copy the `.env.template` file to `.env`.
2. Fill the `TRACETEST_AGENT_ENDPOINT` value from your environment’s tracing backend information. It should be formatted like this `https://agent-<redacted>-<redacted>.tracetest.io:443`.
3. Fill the `TRACETEST_API_TOKEN` value with the one generated for your Tracetest environment. It’ll look like this `tttoken_***************`.
4. Run `npm i`.
5. Run the Serverless Framework deployment with `npm run deploy`. Use the API Gateway endpoint from the output in your test below.
6. Run the trace-based tests with `npm test https://<api-gateway-id>.execute-api.us-east-1.amazonaws.com`.

Now, let’s dive-in into the nitty-gritty details. 🤓

## Instrumenting the AWS Lambda Functions

First, each Lambda function is preloading the OpenTelemetry configuration by executing the setup file before the actual handler execution.

```yaml
environment:
   NODE_OPTIONS: --require ./src/setup
```

This is going to execute the `createTracer` function from the `src/telemetry/tracing.ts` file that configures the trace provider with the exporter options.

```jsx
let globalTracer: opentelemetry.Tracer | null = null;

async function createTracer(): Promise<opentelemetry.Tracer> {
 const provider = new NodeTracerProvider();

 const spanProcessor = new BatchSpanProcessor(
   new OTLPTraceExporter({
     url: COLLECTOR_ENDPOINT,
   })
 );

 provider.addSpanProcessor(spanProcessor);
 provider.register();

 registerInstrumentations({
   instrumentations: [
     new AwsLambdaInstrumentation({
       disableAwsContextPropagation: true,
     }),
   ],
 });

 const tracer = provider.getTracer(SERVICE_NAME);

 globalTracer = tracer;

 return globalTracer;
}

async function getTracer(): Promise<opentelemetry.Tracer> {
 if (globalTracer) {
   return globalTracer;
 }

 return createTracer();
}
```

The telemetry data generated by the AWS Lambda function is going to be [sent to the `COLLECTOR_ENDPOINT`](https://github.com/kubeshop/pokeshop/blob/cc0286044db3cf5319cb74ff9e23c5f6da157b93/serverless/serverless.yml#L35), which, in this case, is set to the [Tracetest Cloud Agent](https://docs.tracetest.io/concepts/cloud-agent), with extra no setup, no collectors, no side carts. The Tracetest platform is ready to ingest your traces.


That’s it, that’s all you need to instrument your AWS Lambda functions. You don’t believe me?! Take a look at the official [OpenTelemetry Serverless docs](https://opentelemetry.io/docs/languages/js/serverless/).

## Test Case: Importing a Pokemon

This is what we are going to be using as test case:

- Execute an HTTP request against the import Pokemon service.
- This is a two-step process that includes an initial handler that puts a message into SQS.
- Then, a worker picks up the message to trigger an external service (PokeAPI) request to grab the raw Pokemon data.
- Finally the worker executes the required database operations to store the Pokemon data to both RDS Postgres and ElastiCache.

**What are the key parts we want to validate?**

1. Validate that the external service from the worker is called with the proper `POKEMON_ID` and returns `200`.
2. Validate that the duration of the DB operations is less than `100ms`.
3. Validate that the response from the initial API Gateway request is `200`.

### Running the Trace-Based Tests

To run the tests, we are using the `@tracetest/client` [NPM package](https://www.npmjs.com/package/@tracetest/client). It allows teams to enhance existing validation pipelines written in JavaScript or TypeScript by including trace-based tests in their toolset.

Because, who doesn’t like JavaScript, right? …Right? 👀

The code can be found in [the `tracetest.ts` file](https://github.com/kubeshop/pokeshop/blob/master/serverless/tracetest.ts).

```jsx
import Tracetest from '@tracetest/client';
import { TestResource } from '@tracetest/client/dist/modules/openapi-client';
import { config } from 'dotenv';

config();

const { TRACETEST_API_TOKEN = '' } = process.env;
const [url = ''] = process.argv.slice(2);

// The Tracetest test JSON definition
const definition: TestResource = {
 type: 'Test',
 spec: {
   id: 'ZV1G3v2IR',
   name: 'Serverless: Import Pokemon',
   trigger: {
     type: 'http',
     httpRequest: {
       method: 'POST',
       url: '${var:ENDPOINT}/import',
       body: '{"id": ${var:POKEMON_ID}}\n',
       headers: [
         {
           key: 'Content-Type',
           value: 'application/json',
         },
       ],
     },
   },
   specs: [
     // Validate the external service from the worker is called with the proper POKEMON_ID and returns 200
     {
       selector: 'span[tracetest.span.type="http" name="GET" http.method="GET"]',
       name: 'External API service should return 200',
       assertions: ['attr:http.status_code   =   200', 'attr:http.route  =  "/api/v2/pokemon/${var:POKEMON_ID}"'],
     },
     // Validate the duration of the DB operations is less than 100ms.
     {
       selector: 'span[tracetest.span.type="database"]',
       name: 'All Database Spans: Processing time is less than 100ms',
       assertions: ['attr:tracetest.span.duration < 100ms'],
     },
     // Validate the response from the initial API Gateway request is 200
     {
       selector: 'span[tracetest.span.type="general" name="Tracetest trigger"]',
       name: 'Initial request should return 200',
       assertions: ['attr:tracetest.response.status = 200'],
     },
   ],
 },
};

const main = async () => {
 if (!url)
   throw new Error(
     'The API Gateway URL is required as an argument. i.e: `npm test https://75yj353nn7.execute-api.us-east-1.amazonaws.com`'
   );

 // configure
 const tracetest = await Tracetest(TRACETEST_API_TOKEN);

 // create
 const test = await tracetest.newTest(definition);

 // run!
 await tracetest.runTest(test, {
   variables: [
     {
       key: 'ENDPOINT',
       value: `${url.trim()}/pokemon`,
     },
     {
       key: 'POKEMON_ID',
       value: `${Math.floor(Math.random() * 100) + 1}`,
     },
   ],
 });

 // and wait and log results (optional)
 console.log(await tracetest.getSummary());
};

main();
```

### Visualizing the Results

With everything set up and the trace-based tests executed against the Pokeshop demo, we can now view the complete results. Follow the links provided in the `npm test` command output to find the full results, which include the generated trace and the test specs validation results.

```bash
npm test https://<api-gateway-id>.execute-api.us-east-1.amazonaws.com

[Output]
> api@1.0.0 test
> ts-node tracetest.ts https://<api-gateway-id>.execute-api.us-east-1.amazonaws.com

Successful: 1
Failed: 0

[✔️ Serverless: Import Pokemon] #5 - https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_a7c6870903f808ce/test/ZV1G3v2IR/run/5
```

👉 [Join the demo organization where you can start playing around with the Serverless example with no setup!!](https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/invites/invite_f9f784f30c85dc97/accept) 👈

From the Tracetest test run view, we can view the list of spans generated by the Lambda function, their attributes, and the test spec results, which validate the key points.


## Takeaways

You have seen how simple it can be to instrument an AWS Lambda Function but, not only that, you now know how to run trace-based tests against an asynchronous Serverless process.

You are now ready to face the world and give it a try by yourself. Remember that testing and observability is a process, but as everything it can always be improved, so don’t be afraid to start with something small!

Have questions? you can find me lurking around the [Tracetest Slack channel](https://dub.sh/tracetest-community) - join, ask, and we will answer.