OpenTelemetry

01The three signals and how they connect

Telemetry comes in three main signals. A trace is the story of one request: it is a tree of spans, where each span is one unit of work (an HTTP handler, a SQL query, a call to a payment API) with a start time, a duration, a status and attributes such as http.route or db.system. Metrics are numeric measurements aggregated over time: request rate, error rate, latency histograms, queue depth, memory. They are cheap to store and great for dashboards and alerts. Logs are timestamped records of events. The power of OpenTelemetry is correlation: logs written inside a span carry its trace ID and span ID, and metrics can carry exemplars that link to example traces. So an alert on p95 latency leads you to a slow trace, and the trace leads you to the exact log lines. Traces, metrics and logs are stable in the specification; profiling is being added as a fourth signal.

• Trace = tree of spans for one request.
• Metrics = cheap aggregates for alerts and dashboards.
• Logs = events, correlated by trace ID and span ID.

02Architecture: API, SDK, exporters and the Collector

OTel splits responsibilities cleanly. The API is what libraries and your code call to create spans and record metrics; on its own it does nothing, so libraries can depend on it safely. The SDK is the implementation you configure in your application: it decides sampling, batches data and hands it to exporters. Exporters send data out, usually over OTLP (the OpenTelemetry Protocol), on port 4317 for gRPC or 4318 for HTTP. Instrumentation libraries patch popular frameworks and clients (Express, Fastify, Next.js, Laravel, pg, mysql, Redis, HTTP clients) to create spans automatically. The Collector is a separate service that receives telemetry, processes it (batching, filtering, removing sensitive attributes, sampling) and exports it to one or more backends. You can run it as an agent next to each app, as a central gateway, or both. Resources describe who sent the data: service.name, service.version, deployment.environment.name.

03Instrumenting a Node.js service

The quickest start in Node.js is the SDK with the auto-instrumentations bundle. Install @opentelemetry/sdk-node, @opentelemetry/auto-instrumentations-node and an OTLP exporter, then load a small setup file before your application code, using node --import for ES modules. Configure it with standard environment variables rather than hard-coded values, so the same image works in every environment: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_RESOURCE_ATTRIBUTES and OTEL_TRACES_SAMPLER. Next.js has a built-in instrumentation.ts hook and the @vercel/otel helper. Laravel and PHP use the OpenTelemetry PHP extension plus auto-instrumentation packages. Start with auto-instrumentation, look at real traces, then add custom spans only around important business steps.

1// instrumentation.mjs: run with  node --import ./instrumentation.mjs server.mjs
2import { NodeSDK } from "@opentelemetry/sdk-node";
3import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
4import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
5import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
6import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
7 
8// Endpoint and service name come from OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_SERVICE_NAME
9const sdk = new NodeSDK({
10  traceExporter: new OTLPTraceExporter(),
11  metricReader: new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter() }),
12  instrumentations: [getNodeAutoInstrumentations({ "@opentelemetry/instrumentation-fs": { enabled: false } })],
13});
14 
15sdk.start();
16process.on("SIGTERM", () => sdk.shutdown().finally(() => process.exit(0)));

04Custom spans and attributes

Auto-instrumentation shows HTTP and database calls, but not your business logic. Add a span around meaningful steps such as "calculate price", "reserve stock" or "generate invoice PDF", and attach attributes that help you search later: order.items_count, tenant.tier, cache.hit. Follow the semantic conventions for common names (http.request.method, db.system.name, messaging.system) so backends understand them. Keep attribute cardinality in mind: an order ID is fine on a span, which is a single event, but never put user IDs or URLs with IDs into metric attributes, because each unique value creates a new time series. Record exceptions on the span and set an error status so failed requests are easy to find. Never put passwords, tokens or full card numbers in attributes.

1import { trace, SpanStatusCode } from "@opentelemetry/api";
2 
3const tracer = trace.getTracer("checkout");
4 
5export async function placeOrder(cart: { items: unknown[] }) {
6  return tracer.startActiveSpan("checkout.place_order", async (span) => {
7    span.setAttribute("order.items_count", cart.items.length);
8    try {
9      const order = await saveOrder(cart); // pg / HTTP calls become child spans
10      span.setAttribute("order.id", order.id);
11      return order;
12    } catch (err) {
13      span.recordException(err as Error);
14      span.setStatus({ code: SpanStatusCode.ERROR });
15      throw err;
16    } finally {
17      span.end();
18    }
19  });
20}

05Running the Collector

Sending telemetry from apps straight to a vendor works, but a Collector in between gives you control. Apps send OTLP to a local Collector quickly and return to work; the Collector batches, retries and buffers if the backend is slow, so your app is not affected. It is the right place to add environment attributes, drop noisy spans like health checks, remove sensitive data and apply tail sampling. It can also scrape Prometheus metrics and read log files, so it becomes your single telemetry pipeline. Use the contrib distribution (otelcol-contrib) for the full set of processors and exporters, or build a minimal one with the OpenTelemetry Collector Builder. Always include the memory_limiter processor first and batch last in each pipeline, and monitor the Collector's own metrics for dropped data.

1# otel-collector.yaml
2receivers:
3  otlp:
4    protocols:
5      grpc: { endpoint: 0.0.0.0:4317 }
6      http: { endpoint: 0.0.0.0:4318 }
7processors:
8  memory_limiter: { check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 20 }
9  attributes/scrub:
10    actions: [{ key: http.request.header.authorization, action: delete }]
11  batch: {}
12exporters:
13  otlphttp: { endpoint: https://otlp.example.com }
14service:
15  pipelines:
16    traces: { receivers: [otlp], processors: [memory_limiter, attributes/scrub, batch], exporters: [otlphttp] }
17    metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp] }

06Tail sampling at scale

Tail sampling keeps whole traces based on what happened in them, but a trace's spans arrive from many services at different times. The Collector holds spans in memory for a decision wait period (often 10–30 seconds) and then evaluates policies: status_code ERROR, latency above a threshold, specific attributes, or a probabilistic rate for the rest. With several Collector replicas, all spans of one trace must reach the same instance, so you run two tiers: agents or a first gateway layer using the load-balancing exporter keyed by trace ID, then a sampling tier. Size memory for traffic × wait time, and watch the processor's dropped-traces metrics. Remember that sampled-away traces also affect metrics derived from spans; generate RED metrics (rate, errors, duration) with the spanmetrics connector before sampling, so dashboards stay accurate.

1processors:
2  tail_sampling:
3    decision_wait: 15s
4    num_traces: 100000
5    policies:
6      - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
7      - { name: slow, type: latency, latency: { threshold_ms: 1000 } }
8      - { name: baseline, type: probabilistic, probabilistic: { sampling_percentage: 5 } }

07Propagation through queues and async work

Automatic propagation covers synchronous HTTP and gRPC. Asynchronous work needs care. When you publish a job to Redis, RabbitMQ or Kafka, inject the active context into the message headers; when a worker picks it up, extract it and start a consumer span. Semantic conventions model this with producer and consumer span kinds. A long-delayed job raises a design question: should the job's spans be children of the request that created it? For short pipelines yes, but a job that runs an hour later makes a strange, very long trace. The alternative is a new trace with a span link to the original context, which preserves the relationship without stretching the trace. Batch consumers should use links to many producer contexts. Baggage can carry values such as tenant ID across services, but it is sent on every request, so keep it small and never put secrets in it.

1import { context, propagation, trace } from "@opentelemetry/api";
2 
3// Producer: carry the trace context inside the job payload
4export function withTraceHeaders<T extends object>(payload: T) {
5  const headers: Record<string, string> = {};
6  propagation.inject(context.active(), headers);
7  return { ...payload, otel: headers };
8}
9 
10// Consumer: continue the trace when the job runs
11export function runJob(job: { otel?: Record<string, string> }, work: () => Promise<void>) {
12  const parent = propagation.extract(context.active(), job.otel ?? {});
13  return context.with(parent, () =>
14    trace.getTracer("worker").startActiveSpan("job.process", (span) => work().finally(() => span.end())),
15  );
16}

08Metrics design: instruments, temporality and cardinality

OTel metrics use instruments: Counter for things that only go up (requests, bytes sent), UpDownCounter for values that rise and fall (queue size, active connections), Histogram for distributions (latency, payload size), and Gauges for sampled current values. Histograms are what give you p95 and p99; exponential histograms give good accuracy with fewer buckets. Temporality matters when exporting: Prometheus-style backends expect cumulative values, while some vendors prefer delta, and the exporter can be configured either way. Cardinality is the main cost driver: each unique combination of attribute values is a separate time series. http.route=/orders/:id is fine; the raw URL /orders/123 is a disaster. Use views to drop or rename attributes and to change histogram buckets without touching instrumentation code.

09Overhead, safety and rollout

Instrumentation is not free, but it is usually small: a few percent of CPU, mostly from creating spans and exporting them. Keep it low by using the batch span processor (never the simple processor in production), sampling, and disabling noisy instrumentations such as file system or DNS spans. Exporting must never block requests: if the backend is down, the SDK drops data after its queue fills rather than slowing the app. Scrub sensitive data at two levels: avoid capturing it in code, and remove headers, query strings and SQL parameters in the Collector. Roll out service by service, starting with the edge and the most critical path, and compare latency before and after. Treat the Collector as production infrastructure: health checks, resource limits, horizontal scaling and alerts on dropped spans.