Skip to content

Instrumentation

Every instrumentation below is registered by setupTracing, apart from those that are enabled per option. Each entry lists the attributes this library adds on top of what the instrumentation already emits.

Several instrumentations record metrics as well as spans. Those measurements reach the backend only when a meter provider is registered, which setupTracing does unless enableMetrics is false.

Metric Emitted by
http.server.request.duration HTTP
http.client.request.duration HTTP, Undici
gen_ai.client.token.usage, gen_ai.client.operation.duration AWS SDK, for Bedrock calls
Instrumentation Package Enabled
HTTP @opentelemetry/instrumentation-http Always
Undici (fetch) @opentelemetry/instrumentation-undici Always
Express @opentelemetry/instrumentation-express Always
Connect @opentelemetry/instrumentation-connect Always
Pino @opentelemetry/instrumentation-pino Always
AWS SDK @opentelemetry/instrumentation-aws-sdk Always
IORedis @opentelemetry/instrumentation-ioredis Always
Elasticsearch opentelemetry-instrumentation-elasticsearch Always
Node runtime @opentelemetry/instrumentation-runtime-node enableMetrics
File system @opentelemetry/instrumentation-fs enableFsInstrumentation
DNS @opentelemetry/instrumentation-dns enableDnsInstrumentation

HTTP

Incoming requests to a path starting with /metrics or /healthz produce no span, which keeps scrape and probe traffic out of the trace store.

Attribute Source Set on
http.request.content_type content-type request header Client and server spans
http.request.content_length content-length request header Client and server spans
http.request_id x-request-id request or response header Client and server spans
http.correlation_id x-correlation-id request header Client and server spans
http.response.content_type content-type response header Client and server spans
http.response.content_length content-length response header Client and server spans
peer.service Remote host of an outgoing request Client spans only
db.system.name Remote host of an outgoing request Client spans only

A content length header is parsed as an integer and written only when it is a non-negative number, so a malformed header is ignored rather than recorded as text.

The request hook reads getHeaders() on an outgoing ClientRequest and .headers on an incoming IncomingMessage, so the same attributes appear on both sides of a call.

Undici (fetch)

globalThis.fetch runs on undici, which never touches the http and https modules the HTTP instrumentation patches. Without this instrumentation a fetch call produces no client span and injects no traceparent, the callee starts a new trace, and the two services can never be paired into a service graph edge.

Attribute Source
peer.service The request origin, matched against the known peer list
db.system.name The request origin, matched against the known peer list

Express

The instrumentation creates a span per layer, covering every middleware, every router and the request handler, and it calls the request hook on each one. The hook records attributes on the request handler layer alone, which is the layer carrying the matched route. Middleware and router spans keep the names the instrumentation gives them, such as middleware - expressInit and router - /work, which is what shows where time inside a request went.

Attribute Source
express.route The matched route, when there is one
express.params Route parameters, JSON encoded, when the object is not empty
express.query_keys The names of the query string parameters, sorted
user.id request.user.id, when the application sets one

Query values are not recorded. A query string carries access tokens and personal data, and OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT is unbounded by default, so a value written to a span attribute is exported in full. The key names describe the shape of a request without exporting its contents.

The server span is named METHOD /route, for example GET /work/:id. That naming comes from the HTTP instrumentation, which reads http.route as the response finishes, and the Express instrumentation supplies the route it reads.

Connect

Registered with defaults. The instrumentation configuration accepts only the base options, with no request or ignore hooks, so nothing is added on top of the middleware spans it emits.

Pino

The instrumentation injects trace_id, span_id and trace_flags into every log record by default. The log hook adds service.name, so a log line carries the same service identity as the span it belongs to.

Correlating logs with traces in Grafana relies on those ids being in the log record.

It also sends a copy of each record to the OpenTelemetry logs API, which setupTracing exports over OTLP unless enableLogs is false. The application's own stream still receives every record, so container logs are unchanged and the export is an addition to them.

Field Source
Severity The Pino level, mapped onto the OpenTelemetry severity numbers
Timestamp The record time, converted according to the logger's timestamp function
Body The record message
Trace context The active span, so a record written inside a request carries its trace

A record is sent through pino.multistream, which needs Pino 7 or later. Log sending is skipped on an older version.

AWS SDK

Setting Value
suppressInternalInstrumentation false, so the underlying HTTP calls are still traced
sqsExtractContextPropagationFromPayload true, so a trace continues across an SQS message
Attribute Source
peer.service The AWS service name, lower cased
aws.service The AWS service name, lower cased
aws.request_id The request id from the response

IORedis

requireParentSpan is false, so a Redis command issued outside a request still produces a span.

Attribute Source
peer.service Always redis
db.redis.key The first command argument
db.redis.args_count The argument count, when there is more than one
db.response.type The JavaScript type of the response
db.response.count The response length, when it is an array

Spans are renamed to redis.COMMAND, for example redis.SET. db.system.name, db.operation.name and the server.* attributes come from the instrumentation itself.

The statement serialiser truncates each argument to 100 characters and appends an ellipsis. A Buffer argument is sliced before it is decoded, so a large value is not converted in full only to be discarded.

Node runtime

Registered when enableMetrics is set, which is the default. It produces metrics alone, with no spans, and it is constructed only when metrics are enabled because its collectors begin sampling on construction.

Metric Description
nodejs.eventloop.delay.min, .max, .mean, .stddev, .p50, .p90, .p99 Event loop delay distribution
nodejs.eventloop.utilization Fraction of the loop spent active
nodejs.eventloop.time Time in the loop, split by nodejs.eventloop.state of active or idle
v8js.gc.duration Garbage collection pause duration, by v8js.gc.type
v8js.memory.heap.used, v8js.memory.heap.space.available_size, v8js.memory.heap.space.physical_size Heap occupancy per heap space
v8js.resource.active Active handles and requests, by v8js.resource.type

Event loop saturation slows every operation in a process at once. No span attribute carries it, which is what these metrics are for.

Elasticsearch

Registered with defaults. peer.service for an Elasticsearch call comes from the HTTP or undici hook, which matches elasticsearch in the remote host.

File system

Off by default, enabled with enableFsInstrumentation. The instrumentation patches fs on construction, so it is constructed only when the option is set.

DNS

Off by default, enabled with enableDnsInstrumentation. Lookups of localhost, 127.0.0.1 and ::1 are ignored. The configuration accepts only an ignore list, with no hooks, so nothing is added to its spans.