OpenTelemetry

Synapse speaks OTLP/HTTP for three signals: security event logs, metrics, and request spans. Point it at any OpenTelemetry collector and they arrive on the standard paths — there is no Gen0Sec-specific protocol in the way. Logs and metrics are on by default; spans are not, and Tracing turns them on.

The backend-specific pages (Axiom, Honeycomb) are worked examples of what this page describes generically.

Turning it on

# /etc/synapse/config.yaml
telemetry:
  enabled: true
  otlp:
    endpoint: "https://collector.example.com:4318"
    headers:
      authorization: "Bearer <token>"

  # Sampling covers HTTP access logs, and only those. Errors bypass the sampler.
  access_log:
    sample_rate: 0.1
    always_keep_errors: true

endpoint is a base URL. Synapse appends the OTel-standard paths itself:

Signal	Path
Logs	{endpoint}/v1/logs
Metrics	{endpoint}/v1/metrics
Spans	{endpoint}/v1/traces — only with telemetry.tracing.export_spans

Every request carries User-Agent: Synapse/<version>, so one allowlist entry covers config-poll and telemetry push alike, and the two are distinguishable in a collector's own access log.

Where telemetry goes

Up to three OTLP destinations, and they are not a precedence ladder. Synapse pushes to every one that is configured:

1. The OTEL_EXPORTER_OTLP_* environment variables — the standard OTel knobs. An endpoint set here is a destination of its own, not an override of the file.
2. The telemetry.otlp block above.
3. The platform — {base_url}/v1/telemetry, where logs and metrics share one path. This push is gated by platform.telemetry_sending_enabled, which defaults to true, not by whether the other two set an endpoint.

So a stray OTEL_EXPORTER_OTLP_ENDPOINT does not quietly replace your config file: it adds a second place the data goes. For a reader who cares where telemetry lands, item 3 is the one to check — an agent with platform.api_key set is already pushing to the platform whether or not you configure a collector.

Headers follow their own destination. OTEL_EXPORTER_OTLP_HEADERS reach the environment's destination only, and telemetry.otlp.headers reach that block's only. A URL that two of them name is pushed to once, as the earlier, carrying the headers of both — the earlier's winning a clash.

The local file sink

{logging.log_directory}/telemetry.log — /var/log/synapse/telemetry.log with the default directory. Three things about it are easy to get wrong:

• It is off by default. The sink exists only when logging.file_logging_enabled is true, and that key defaults to false. On a default config there is no telemetry.log at all.
• It is not a fallback. When it is on, block events are written to it and pushed to every OTLP destination, rather than the file catching what is left when none is configured.
• It carries block events, not the whole log signal. This is the part worth knowing before you treat telemetry.log as a local mirror: access logs, IDS alerts and the fingerprint events take a different emit path that writes to OTLP only. With the sink on and no collector configured, telemetry.log fills with block events while those others go nowhere on the telemetry path at all. Their local counterparts are events.log, eventbridge.log and the access log, not this file.
• telemetry.enabled: false takes it with them. The switch skips telemetry initialisation entirely — no OTLP push, no file sink, no instruments. Block events still reach the blocking log by their own path, so the host is not silent: that is events.log when logging.file_logging_enabled is on, and stdout when it is not.

Omitting the block does not disable telemetry

telemetry.enabled defaults to true, so a config with no telemetry: section at all still exports — to the platform if platform.api_key is set. If you want nothing leaving the host, set enabled: false explicitly; leaving the section out does the opposite.

Metrics

Metric	Type	Unit
synapse.kernel.drops	Counter	{packet}	Packets dropped by the in-kernel BPF firewall, partitioned by layer
synapse.block.events.access_rules	Counter	{event}	Static IP / CIDR / country / ASN blocklists
synapse.block.events.smart_firewall	Counter	{event}	JA4-family fingerprint matches, kernel and userland
synapse.block.events.ids	Counter	{event}	Thalamus Suricata-style signatures
synapse.block.events.waf	Counter	{event}	HTTP-layer wirefilter rules
synapse.block.events.threat_intel	Counter	{event}	IP-reputation hits
synapse.block.events.captcha	Counter	{event}	Challenge issued or failed
synapse.block.events.rate_limit	Counter	{event}	Per-route or per-fingerprint rate hits
synapse.ban.active	Gauge	{ban}	Bans currently held, observed on the collection interval
synapse.ban.records_dropped	Counter	{record}	Ban-ledger records dropped
synapse.telemetry.emit_latency	Histogram	ms	Cost of the emit path itself: file write, OTLP enqueue and metrics record

Attributes

Two attribute keys carry the dimensions, and they are attached to different metrics:

Attribute	On	Values
synapse.action	every block.events.* counter	block, drop, allow, log, notice, ratelimit, captcha
synapse.layer	synapse.kernel.drops, synapse.ban.active	the enforcing layer

So the layer is encoded two different ways depending on the metric, and this is the thing that trips up a first query: block.events.* is one counter per layer, with the action as its attribute. A total across all enforcement is therefore a sum over the metric family, not a single series filtered by a layer label:

sum(rate({__name__=~"synapse_block_events_.*"}[5m]))

Whereas synapse.kernel.drops and synapse.ban.active are single series that you do split by synapse.layer:

sum by (synapse_layer) (rate(synapse_kernel_drops[5m]))

synapse.telemetry.emit_latency carries explicit bucket boundaries from 0.01 ms to 100 ms. It measures Synapse's own observability overhead, which is the number to look at before blaming telemetry for a latency regression.

Metrics export on a periodic reader. Override the interval with the standard OTEL_METRIC_EXPORT_INTERVAL environment variable; it applies to every configured destination.

Identifying an agent

Every record carries service.instance.id, set to Synapse's derived agent ID — a stable hash of the agent name that survives restarts, which is what you want for a long-running dashboard.

Without an agent ID, the instance ID is the PID

The fallback is the process PID, which changes on every restart. That is useful for spotting restarts and useless for a dashboard grouping by instance — a restart looks like one agent disappearing and a different one appearing.

Sampling

One sampler, and it covers HTTP access logs only — one record per proxied request is the highest-volume stream a proxy produces.

Key	Default	What it does
telemetry.access_log.sample_rate	1.0	Probability in [0.0, 1.0] of keeping a non-error access-log record. 1.0 keeps every one
telemetry.access_log.always_keep_errors	true	4xx and 5xx responses bypass the sampler

Leave always_keep_errors on; sampling away errors is how a sampled stream becomes misleading rather than merely incomplete. SYNAPSE_ACCESS_LOG_SAMPLE_RATE overrides the sample rate when it is set.

There is no sampler on the general log stream — every other event Synapse exports is exported in full. Metrics are not sampled either: they are aggregated at the source, so the export interval, not a sample rate, controls their volume.

The access-log sampler also suppresses spans

A request's spans are exported only if the access-log sampler kept that request's record, or another hop's spans hang off it (a trusted caller sampled the trace, or Synapse sent its own span upstream as their parent). On a default config neither of those holds — it takes a trusted_proxies entry or trust_lb_tls_headers, and propagation other than passthrough — so access_log.sample_rate: 0.1 really does cost about nine spans in ten, whatever tracing.sample_rate says. Sample the access log and you are sampling traces with it.

Two corollaries, both of which look like a tracing bug and are not:

• Errors keep their spans, because always_keep_errors keeps their access-log record. The nine-in-ten figure is for non-error traffic.
• An endpoint with access logging switched off exports no spans, at any sample rate. That includes health-check probes.

telemetry.otlp takes no sampling keys

sample_rate and always_keep_errors live under telemetry.access_log, and nowhere else. Set either one under telemetry.otlp and it is silently ignored: the key lands in the Unused config options in <path> warning at startup, and nothing is sampled. An earlier version of this page showed them in both places.

Tracing

Every log record a request produces carries its trace context whatever these settings say. They decide only what reaches the upstream, and whether spans are exported.

Key	Default	What it does
telemetry.tracing.export_spans	false	Export a server span per proxied request, with a child per timed stage, at /v1/traces
telemetry.tracing.propagation	passthrough	What the proxy does with the traceparent / tracestate headers it forwards upstream
telemetry.tracing.sample_rate	1.0	Ratio of the traces Synapse starts that are sampled (parentbased_traceidratio, so a caller's own decision is followed)

propagation takes three values:

• passthrough (the default) — forward the headers as the client sent them, so Synapse's hop is not part of the trace the upstream continues.
• propagate — send Synapse's own context: the client's trace, or a new one, with Synapse's span as the upstream's parent. The client's tracestate is kept only when its trace is continued.
• off — strip both, so no trace context reaches the upstream.

OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG override sample_rate. All three keys are read once, at startup.

Spans do not go to the platform

Of the three destinations above, spans reach the first two only. The platform destination takes logs and metrics and refuses traces, logging no spans to <url>: it takes logs and metrics only. So an agent configured with nothing but platform.api_key exports no spans at all, however export_spans is set — give it an OTEL_EXPORTER_OTLP_ENDPOINT or a telemetry.otlp.endpoint to send them to.

Limits worth knowing

• The queue is bounded. If a collector is unreachable, records buffer and then drop at enqueue once the queue fills, with a warning logged. Telemetry backpressure never becomes request backpressure, which is the right trade — but it does mean a long collector outage is a gap, not a backlog.
• Batching adds delay. Records flush on a size threshold or a time cap, whichever comes first, so there is a small lag between a decision and its appearance downstream.
• Spans are off by default, not absent. See Tracing for the three keys that turn them on and decide what reaches the upstream.

See also

• Security Event Export — the event model behind the log stream, and the non-OTLP sinks
• Axiom and Honeycomb — backend-specific walkthroughs
• Configuration — the full telemetry block