Upgrade Notes

These pages describe Synapse 0.8.8. This page is the delta: what moves under your feet when you upgrade, what a 0.8.8 configuration file can say that an older one could not, and what happens to such a file if you roll back.

0.8.8

Read these before you upgrade

Each one changes what a running instance does with the configuration you already have.

Host matching

The one that can change which origin serves a request, so take it first.

• Host is now matched without regard to case and without a trailing root dot. APP.example.com and app.example.com. are served by the host app.example.com; before they got a 404, or a wildcard host that covered them. A request that used to be served by the wildcard now arrives at the exact host - with that host's header rules, body limits and WAF context, not the wildcard's.
• Expression routes (match_expr) on a wildcard host now fire. Before, they never did.
• A host that serves a bare address - an entry keyed by it, or a default_host: - and has a route for / is served at /. Before, that path always got the probe reply, so a load-balancer health check against an address may now get a different answer. Adopting default_host: with a route for / is enough to change it, with no host keyed by an address anywhere in the file.
• In a v1 file, a literal "*" host key now acts as the default host. Before it matched nothing.
• A name more than one label under a wildcard host that declares listeners: is no longer served on other listeners. Requests that worked - and should not have - are answered as an unconfigured host, or served by default_host: if you have one: a different origin rather than a refusal.
• timeouts.idle: 0 no longer reuses an upstream connection at all.
• The origin of a request with an IPv6-literal Host receives [addr], not the text before the first colon.

GeoIP starts working without a platform key

An instance with no platform.api_key whose configuration names GeoIP databases now opens them at startup, and downloads any database given by URL whose file is missing. That is a new outbound request and a new disk write on first start after the upgrade.

Check this first if your egress is restricted

The shipped config/config.yaml names one: https://github.com/gen0sec/geoip-databases/raw/download/ipinfo_lite.mmdb to /var/lib/synapse/ipinfo_lite.mmdb. On a host that cannot reach GitHub, or whose database path is not writable, this is the item to settle before upgrading.

Until now such an instance loaded no database at all, so its records carried no country or ASN. That is the fix; the download is its consequence.

A firewall rule on ja4h may need rewriting

With ids.post_tls.mode: offload, the JA4H recorded for a request the proxy hands over - in the fingerprint log, and in the store the smart firewall reads - is now the access log's value. It used to be rebuilt from the transaction, with lower-cased header names and an added Host on HTTP/2, so it differed from what the proxy logged. A rule written against the old value has to be given the new one, and while proxies are upgraded one at a time the same client is recorded under either value.

OTLP headers stay with their own destination

Synapse pushes telemetry to every destination that is configured, not to the first alone - the endpoint OTEL_EXPORTER_OTLP_ENDPOINT names, the one telemetry.otlp.endpoint names, and the platform's. That part is not new.

What changed is which headers each one receives. Through 0.8.7 the OTLP SDK attached OTEL_EXPORTER_OTLP_HEADERS (and its per-signal forms) to every exporter it built, replacing the destination's own header of the same name. So a token meant for your collector also went to the platform, and the platform's bearer token was overwritten by it. From 0.8.8 each destination is sent its own headers and no other's:

Destination	Headers it gets
OTEL_EXPORTER_OTLP_ENDPOINT	OTEL_EXPORTER_OTLP_HEADERS, or the signal's own OTEL_EXPORTER_OTLP_{LOGS,METRICS,TRACES}_HEADERS in its place
telemetry.otlp.endpoint	telemetry.otlp.headers only
the platform	the API key, or platform.telemetry_endpoint.headers

If your collector is named in YAML and its credentials are in the environment

That collector stops receiving them. Copy the credentials into telemetry.otlp.headers. Synapse tells you at startup, naming the fix:

OTEL_EXPORTER_OTLP_HEADERS is set, but the environment names no endpoint for its headers (OTEL_EXPORTER_OTLP_ENDPOINT): they are sent to no destination. The headers for telemetry.otlp.endpoint go in telemetry.otlp.headers.

The warning fires when the environment names no endpoint, which is this shape exactly. It stays silent when OTEL_EXPORTER_OTLP_ENDPOINT is also set and the token was meant for that other collector - the headers do have a destination then, just not the YAML one.

Setting both is safe: with no endpoint in the environment the environment's headers are ignored, and where two sources do name the same URL it is pushed to once, carrying the headers of both. On a clash of names there, the earlier destination wins - and the environment's is pushed first, so a stale token in OTEL_EXPORTER_OTLP_HEADERS beats a fresh one in telemetry.otlp.headers.

Two more in the same area:

• OTEL_EXPORTER_OTLP_COMPRESSION no longer stops all export. Setting it, or a per-signal form, used to stop every OTLP exporter from being built - the platform's included - so nothing was exported at all. If your telemetry has been silent since you set it, this upgrade is the repair. The environment's destination now gets the compression it asks for; every other destination is sent uncompressed, as before.
• OTEL_EXPORTER_OTLP_{LOGS,METRICS,TRACES}_ENDPOINT now works. Synapse used to set each signal's endpoint itself, and programmatic configuration beats the environment in the SDK, so these three were read and had no effect.

Two detection paths see more than they did

• SNI on the AF_PACKET path. A tls.sni smart-firewall rule now sees a name on this path, where it had none, so a rule that never matched there can start to. A TLS ClientHello's fingerprint record also carries that name next to its JA4.
• IDS engine (thalamus 0.2.0). Lua scripts in IDS rules run in an allowlist sandbox, with byte and instruction limits that cover loading the script too. A script using anything outside the allowlist, or going over a limit, stops working. An HTTP/1 ranged download is now inspected as one file, so rules on file content can match a download that arrived in ranges.

What a 0.8.8 upstreams.yaml can say that a 0.8.7 one could not

All of these are additive: a file that worked on 0.8.7 keeps working, and none of them changes what an existing route does.

Construct	What it does	On 0.8.7
default_host:	A terminating host serving every name no hosts: entry matches; without it such a request is answered 404. It takes no tls: block - a client asking for an unknown name is shown proxy.default_certificate from config.yaml - and it is tried after exact names and wildcards. A bare * under hosts: is still refused.	whole file refused (unknown key)
match_expr on a v2 route	An expression decides whether the route matches, and it can now read the query string and the headers, not only the path, method and host. A host's expressions are tried in the order of their keys.	whole file refused (unknown key on the route)
tls: { terminate: {} }	A host terminates TLS naming no certificate of its own: it is served over TLS with whatever the listener presents for that name - a certificate found for it in the certificates directory, or the listener's default - and in plain HTTP.	whole file refused: tls.terminate must specify either cert or acme
paths: on a passthrough host	TLS for the name is still passed through, and a plain-HTTP request for it - an ACME HTTP-01 challenge, say - is served by the host's paths:.	whole file refused: passthrough hosts cannot have paths: - synapse does not see L7
health_check: { type: none }	Turns upstream probing off for one route. On a terminating route, leaving health_check out means probed with the global settings, not unprobed - unlike a passthrough host, where no block means no probing.	whole file refused: health_check.type "none" - expected tcp or http

What a rollback does to such a file

Nothing above is ignored by 0.8.7. The v2 parser rejects unknown keys at the root and on a route, and the other three are explicit validation errors, so the whole file fails to load: the proxy keeps running and routes nothing. Two error lines say so - Failed to parse v2 upstreams YAML (kind=…, source=…) with the validation message, then Failed to load upstreams configuration from: <path>

• and a config_reload error metric is recorded, which is the alertable signal. Keep the pre-upgrade file if a rollback is a possibility.

One case is quieter. A match_expr that reads http.request.query or http.request.headers - fields 0.8.8 added - compiles against no such field on 0.8.7. In a v1 file that is warn-and-skip, not a refusal: the file loads and that one route is simply missing, with one warning among the others - Skipping expression route (<host>) "<expr>": compile failed: …. Grep for Skipping expression route after a rollback; nothing else looks like a failure.

A new warning on every v1 upstreams file

Loading an upstreams file in the v1 schema now logs a deprecation warning on every load. The file is still read, and nothing about it stops working. Moving to the v2 schema (version: 2, with hosts: in place of upstreams:) is what closes the warning, and there is still no migration tool - conversion is by hand.

Unchanged, despite what you may have read

• The main config.yaml surface did not move. No key, flag or default was added, removed or changed between 0.8.7 and 0.8.8. The OTLP environment variables did change effect - the header, compression and per-signal endpoint forms alike - and every one of them is described above.
• Passthrough's schema is the same. protocol: http and the per-wire passthrough.tls / passthrough.http blocks work on both releases; their parse types are identical. What 0.8.8 adds is paths: on a passthrough host, in the table above.
• Telemetry fan-out is not new. 0.8.7 also pushed to every configured destination. Documentation describing a precedence order, where one destination suppressed the others, was wrong about both releases.
• match_expr itself is not new. It has worked in a v1 file since before 0.8.8, where it reads the path, method and host. What 0.8.8 adds is match_expr in a v2 file, and the two new fields an expression can read.