placeholder

OpenTelemetry basics: Trace ingest to Dynatrace

author

Adam Gardner

November 10, 2022

Get started with OpenTelemetry and Dynatrace with our tutorial.

OpenTelemetry is powerful but can be challenging to get started with. You've probably already read the documentation and spun up a demo system.

But a lot of what you've seen is "magic." A lot happens behind the scenes. Sometimes, it is helpful to strip it back and start from the basics to understand what is truly happening.

TL;DR Experiment with the In-Browser Live Demo Here

A basic OpenTelemetry setup

The most basic OpenTelemetry setup has three components:

  1. An OpenTelemetry trace generator. Usually, an application that emits OTEL data (metrics, logs, traces, etc.).
  2. An OpenTelemetry collector. The aggregation point for all that data.
  3. A storage back end. That's where you want to send the data. In this demo, the back end will be Dynatrace.

Of course, you can have multiple components, but the above is the most basic setup.

A basic OpenTelemetry Architecture

What will be built

In this demo, tracegen generates trace data that will be sent to an OpenTelemetry collector.

The collector will be configured to pass the trace data to Dynatrace.

In real life, tracegen would be replaced by your application(s). Of course, if you choose to install the OneAgent on your hosts, the OpenTelemetry collector becomes an optional component.

OpenTelemetry + Dynatrace architecture

Create a new Docker network

By default, Docker starts containers on the bridge network. In this mode, containers can only address each other via their IPs.

That's a bit clunky so let's make our lives easier and create a new network called tracegen-demo-net just for this demo. That way, containers can address each other via their name instead — which is easier for us humans and easier to automate.

docker network create tracegen-demo-net

You can see all Docker networks with the following command.

docker network ls

As a result, you should see four networks, one of which is tracegen-demo-net.

Run the collector

Run the collector on the tracegen-demo-net network.

Give the container a name: otelcol, which means other containers can find it via the name instead of a unique ID or IP (which will change at some point).

docker run \
--net tracegen-demo-net \
--name otelcol \
-v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml \
ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.63.0

Generate trace data

Open a second terminal window. Run the trace generator utility.

Again, give the container a name (just for consistency). Also, set the network, so the container runs in the same network. The following command does all that for you.

Notice we're passing the endpoint as otelcol:4317 which means "send trace data to the otelcol container on port 4317" (the default OpenTelemetry gRPC port).

The -otlp-insecure flag is needed because we don't have a proper TLS cert configured on the demo system.

docker run \
--net tracegen-demo-net \
--name tracegenerator \
ghcr.io/open-telemetry/opentelemetry-collector-contrib/tracegen:v0.63.0 -traces 1 -otlp-insecure -otlp-endpoint otelcol:4317

If it works, you should see lots of output that ends with: stopping the exporter.

View traces in Dynatrace

Open Dynatrace, go to Distributed Traces and then toggle over to Ingested traces.

Distributed Traces in Dynatrace

In a few seconds, you should see a trace called lets-go (that's what tracegen names the traces it sends).

Generate more trace data by re-running:

docker run \
--net tracegen-demo-net \
--name tracegenerator \
ghcr.io/open-telemetry/opentelemetry-collector-contrib/tracegen:v0.63.0 -traces 1 -otlp-insecure -otlp-endpoint otelcol:4317

Of course, suppose you have the OneAgent deployed on the host. In that case, a lot of this complexity vanishes: You don't necessarily need to instrument your applications (although if you do, OneAgent will respect those traces and ingest them), and you probably don't need an OpenTelemetry collector either.


OpenTelemetry basics: Trace ingest to Dynatrace was originally published in Dynatrace Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Written by

author

Adam Gardner