OpenTelemetry for a Node.js API in an afternoon: traces, metrics and sane sampling

When a Node.js API gets slow, most teams reach for logs first. Logs tell you what happened inside one process. They do not tell you that a checkout request spent 40 milliseconds in your code, 900 in a downstream inventory service and 300 waiting for a connection from an exhausted Postgres pool.

OpenTelemetry fixes that, and the Node.js SDK has matured to the point where a useful setup takes an afternoon rather than a sprint. This post is the setup we use for client APIs: automatic instrumentation for HTTP, Express or Fastify, Postgres and Redis; a few custom spans and metrics where they earn their place; and a sampling strategy that keeps the useful traces without an eye-watering bill.

It assumes Node.js 20 or newer, OpenTelemetry JS SDK 2.x, and a backend that accepts OTLP (Grafana Tempo, Jaeger, Honeycomb, Datadog, and most others do).

The moving parts

OpenTelemetry has a lot of vocabulary. For a single API you only need five pieces.

You can export straight from the app to a backend, but we always put a Collector in between. It lets you change sampling, add attributes, drop noisy spans and switch vendors without redeploying the application.

Step 1: install the packages

npm install @opentelemetry/sdk-node \
  @opentelemetry/api \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/exporter-metrics-otlp-http \
  @opentelemetry/sdk-metrics \
  @opentelemetry/resources \
  @opentelemetry/semantic-conventions

The auto-instrumentations package is a bundle of instrumentations for common libraries. It is convenient but pulls in a lot; once you know which libraries you use, you can swap it for the individual packages.

Step 2: the instrumentation file

The SDK must start before your application imports any library it needs to patch. That means a separate file loaded ahead of your app, not an import at the top of server.ts.

// src/instrumentation.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { resourceFromAttributes } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: process.env.OTEL_SERVICE_NAME ?? 'orders-api',
    [ATTR_SERVICE_VERSION]: process.env.GIT_SHA ?? 'dev',
    'deployment.environment.name': process.env.APP_ENV ?? 'local',
  }),
  traceExporter: new OTLPTraceExporter(),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter(),
    exportIntervalMillis: 30_000,
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': { enabled: false },
      '@opentelemetry/instrumentation-dns': { enabled: false },
      '@opentelemetry/instrumentation-net': { enabled: false },
      '@opentelemetry/instrumentation-http': {
        ignoreIncomingRequestHook: (req) =>
          req.url === '/healthz' || req.url === '/readyz',
      },
    }),
  ],
});

sdk.start();

process.on('SIGTERM', () => {
  sdk.shutdown().finally(() => process.exit(0));
});

A few decisions in there are worth explaining.

Disable fs, dns and net. The filesystem instrumentation creates a span for every file read, which can mean hundreds of spans per request in some frameworks. DNS and raw socket spans are rarely useful when you already have HTTP and database spans. Turning these off is the single biggest reduction in noise.

Ignore health checks. Kubernetes probes hit /healthz every few seconds per pod. Without this hook they will make up a large share of your traces.

Flush on shutdown. Spans are batched in memory. Without sdk.shutdown() on SIGTERM, every deploy loses the last few seconds of data, which is often exactly the data you want when a deploy goes wrong.

In SDK 2.x the old new Resource() constructor was replaced by resourceFromAttributes(). If you are following an older tutorial and it does not compile, that is usually why.

Step 3: load it first

For a TypeScript project compiled to CommonJS, preload the compiled file with --require:

node --require ./dist/instrumentation.js dist/server.js

For native ES modules, use --import instead, and register the ESM loader hook at the top of your instrumentation file with register('@opentelemetry/instrumentation/hook.mjs', import.meta.url) from node:module. Without the hook, ESM imports are not patched and you will see HTTP spans but no database spans, which is a confusing half-working state.

Configuration then comes from standard environment variables, which keeps the code identical across environments:

OTEL_SERVICE_NAME=orders-api
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability:4318
OTEL_TRACES_SAMPLER=parentbased_always_on

The HTTP exporters append /v1/traces and /v1/metrics to the base endpoint automatically.

Step 4: add a few custom spans

Automatic instrumentation shows you the edges: incoming requests, outgoing HTTP calls, queries. It does not show you what your own code is doing between them. Add custom spans around business operations that take meaningful time or that you will want to filter by.

import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('orders-api');

export async function priceCart(cart: Cart) {
  return tracer.startActiveSpan('pricing.calculate', async (span) => {
    span.setAttribute('cart.items', cart.items.length);
    span.setAttribute('cart.currency', cart.currency);
    try {
      const result = await applyRules(cart);
      span.setAttribute('pricing.discount_applied', result.discount > 0);
      return result;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

Keep attribute values low-cardinality where you can, and never put personal data in them. A customer tier is a good attribute; an email address is not. We usually add the tenant or account ID for B2B APIs because it is the first thing support asks for, and it is not personal data in most contexts.

Step 5: metrics that complement traces

The HTTP instrumentation already emits request duration histograms following the semantic conventions (http.server.request.duration). That covers the standard rate, errors and duration dashboards.

Add your own metrics for things that are not requests: queue depth, cache hit ratio, jobs processed.

import { metrics } from '@opentelemetry/api';

const meter = metrics.getMeter('orders-api');
const cacheLookups = meter.createCounter('cache.lookups', {
  description: 'Cache lookups by result',
});

export function recordCacheLookup(hit: boolean) {
  cacheLookups.add(1, { 'cache.result': hit ? 'hit' : 'miss' });
}

The same cardinality rule applies, only more strictly. Every unique combination of attribute values becomes a separate time series, and most backends bill per series.

Sane sampling

Tracing every request is fine at a few requests per second. At a few thousand, you will pay to store millions of near-identical traces of healthy requests. Sampling is how you keep cost under control without losing the traces that matter.

Head sampling in the SDK

The simplest option is to decide at the start of each trace. parentbased_traceidratio with OTEL_TRACES_SAMPLER_ARG=0.1 keeps 10% of new traces and follows the upstream decision when a request arrives with a trace context already attached. Parent-based sampling matters: without it, each service samples independently and you get broken traces.

The weakness is that the decision is made before you know whether the request will fail or be slow. At 10%, you throw away 90% of your errors.

Tail sampling in the Collector

Tail sampling waits until a trace is complete, then decides. The tail_sampling processor in the Collector contrib distribution lets you keep every error and every slow trace, plus a small share of everything else.

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp]

With this setup, the SDK sends everything (parentbased_always_on) to the Collector, and the Collector forwards errors, slow requests and a 5% baseline. For one Series A fintech client, this cut stored trace volume by 91% while keeping every failed request.

There is one catch. Tail sampling needs all spans of a trace to reach the same Collector instance. With a single Collector that is automatic. With several replicas, put a first tier of Collectors running the loadbalancing exporter with routing_key: traceID in front of the sampling tier. Skip this and you will see partial traces with missing spans.

Keep metrics unsampled. They are aggregated in the SDK before export, so their cost does not grow with traffic the way traces do.

What we'd do differently

• Put the Collector in from day one. Every team that exports directly to a vendor ends up adding one later, usually during an incident.
• Disable noisy instrumentations before the first deploy, not after the first bill.
• Add the git SHA as service.version immediately. Being able to split latency by release is the fastest way to catch a regression.
• Write a runbook for "traces are missing": the usual causes are the SDK loading after the app, a missing ESM hook, or a Collector dropping data under memory pressure (watch otelcol_processor_refused_spans).

Checklist

1. SDK loaded with --require or --import before application code.
2. service.name, service.version and environment set on the resource.
3. fs, dns and net instrumentations disabled; health checks ignored.
4. sdk.shutdown() called on SIGTERM.
5. Custom spans around key business operations, with low-cardinality attributes and no personal data.
6. A Collector between the app and the backend.
7. Tail sampling that keeps errors and slow traces, with trace-ID load balancing if you run more than one Collector.
8. Metrics exported unsampled, with attribute cardinality under control.

An afternoon gets you most of the value. The rest is iteration: every incident will show you one span you wish you had, and adding it takes five minutes.