Skip to main content

OpenTelemetry

Synapse speaks OTLP/HTTP for two signals: security event logs, and metrics. Point it at any OpenTelemetry collector and both arrive on the standard paths — there is no Gen0Sec-specific protocol in the way.

The backend-specific pages (Axiom, Honeycomb) are worked examples of what this page describes generically.

Turning it on​

# /etc/synapse/config.yaml
telemetry:
  enabled: true
  otlp:
    endpoint: "https://collector.example.com:4318"
    headers:
      authorization: "Bearer <token>"
    # Sample the log stream. Errors bypass the sampler.
    sample_rate: 1.0
    always_keep_errors: true

  # HTTP access logs are the highest-volume stream, so they sample separately.
  access_log:
    sample_rate: 0.1
    always_keep_errors: true

endpoint is a base URL. Synapse appends the OTel-standard paths itself:

SignalPath
Logs{endpoint}/v1/logs
Metrics{endpoint}/v1/metrics

Every request carries User-Agent: Synapse/<version>, so one allowlist entry covers config-poll and telemetry push alike, and the two are distinguishable in a collector's own access log.

Where configuration comes from​

Four sources, highest precedence first. This ordering matters because a stray environment variable silently outranks your config file:

  1. OTEL_EXPORTER_OTLP_* environment variables — the standard OTel knobs, and top precedence, the same as every other OTel SDK.
  2. The telemetry.otlp block above.
  3. Legacy platform fallback — if neither of the above sets an endpoint but platform.api_key and platform.base_url are set, Synapse pushes to {base_url}/v1/telemetry, where logs and metrics share one path.
  4. File sink only — /var/log/synapse/telemetry.log.

telemetry.enabled: false is a master switch: no OTLP push happens regardless of endpoint or environment. The local file sink keeps working.

Omitting the block does not disable telemetry

enabled defaults to true, so a config with no telemetry: section at all still emits — to the platform if platform.api_key is set, otherwise to the file sink. If you want nothing leaving the host, set enabled: false explicitly.

Metrics​

MetricTypeUnit
synapse.kernel.dropsCounter{packet}Packets dropped by the in-kernel BPF firewall, partitioned by layer
synapse.block.events.access_rulesCounter{event}Static IP / CIDR / country / ASN blocklists
synapse.block.events.smart_firewallCounter{event}JA4-family fingerprint matches, kernel and userland
synapse.block.events.idsCounter{event}Thalamus Suricata-style signatures
synapse.block.events.wafCounter{event}HTTP-layer wirefilter rules
synapse.block.events.threat_intelCounter{event}IP-reputation hits
synapse.block.events.captchaCounter{event}Challenge issued or failed
synapse.block.events.rate_limitCounter{event}Per-route or per-fingerprint rate hits
synapse.ban.activeGauge{ban}Bans currently held, observed on the collection interval
synapse.ban.records_droppedCounter{record}Ban-ledger records dropped
synapse.telemetry.emit_latencyHistogrammsCost of the emit path itself: file write, OTLP enqueue and metrics record

Attributes​

Two attribute keys carry the dimensions, and they are attached to different metrics:

AttributeOnValues
synapse.actionevery block.events.* counterblock, drop, allow, log, notice, ratelimit, captcha
synapse.layersynapse.kernel.drops, synapse.ban.activethe enforcing layer

So the layer is encoded two different ways depending on the metric, and this is the thing that trips up a first query: block.events.* is one counter per layer, with the action as its attribute. A total across all enforcement is therefore a sum over the metric family, not a single series filtered by a layer label:

sum(rate({__name__=~"synapse_block_events_.*"}[5m]))

Whereas synapse.kernel.drops and synapse.ban.active are single series that you do split by synapse.layer:

sum by (synapse_layer) (rate(synapse_kernel_drops[5m]))

synapse.telemetry.emit_latency carries explicit bucket boundaries from 0.01 ms to 100 ms. It measures Synapse's own observability overhead, which is the number to look at before blaming telemetry for a latency regression.

Metrics export on a periodic reader. Override the interval with the standard OTEL_METRIC_EXPORT_INTERVAL environment variable; it applies to every configured destination.

Identifying an agent​

Every record carries service.instance.id, set to Synapse's derived agent ID — a stable hash of the agent name that survives restarts, which is what you want for a long-running dashboard.

Without an agent ID, the instance ID is the PID

The fallback is the process PID, which changes on every restart. That is useful for spotting restarts and useless for a dashboard grouping by instance — a restart looks like one agent disappearing and a different one appearing.

Sampling​

Two independent sample rates, because the streams differ in volume by orders of magnitude:

  • telemetry.otlp.sample_rate — the general log stream.
  • telemetry.access_log.sample_rate — one record per proxied HTTP request, which on a busy proxy dominates everything else.

always_keep_errors makes 4xx and 5xx responses bypass the sampler, so a low access-log sample rate still preserves every failure. Leave it on; sampling away errors is how a sampled stream becomes misleading rather than merely incomplete.

Metrics are not sampled — they are aggregated at the source, so the export interval, not a sample rate, is what controls their volume.

Limits worth knowing​

  • The queue is bounded. If a collector is unreachable, records buffer and then drop at enqueue once the queue fills, with a warning logged. Telemetry backpressure never becomes request backpressure, which is the right trade — but it does mean a long collector outage is a gap, not a backlog.
  • Batching adds delay. Records flush on a size threshold or a time cap, whichever comes first, so there is a small lag between a decision and its appearance downstream.
  • No traces. Synapse exports logs and metrics. It is not instrumented for distributed tracing, and a collector expecting spans from it will see none.

See also​