Skip to main content

Troubleshooting

Applies to SynapseOS builds from 2026-10-09 (20261009-141152 and later). See What changed.

Find the message you see; each entry says what caused it and what to do.

Where the logs are​

The journal holds every service's output. Two things are written elsewhere: the audit records, in /var/log/audit/audit.log (auditd rotates it itself), and the syslog files the forwarder also writes locally, /var/log/messages, auth.log, kern.log and daemon.log.

journalctl -b                      # this boot, everything
journalctl -b -1                   # the previous boot: what happened before a rollback or reboot
journalctl -u synapse-agent -f     # the data plane, live
journalctl -u synapseos-firewall -u systemd-networkd -b
ausearch -ts boot                  # audit records since boot, from /var/log/audit/audit.log

synapseos-ab-update status and synapseos-synapse status are the two commands to run before anything else when a unit misbehaves after an update.

Boot​

device-mapper: verity: ... data block N is corrupted then Timed out waiting for device /dev/mapper/usr​

Cause. The system partition's content does not match its hash tree. On a unit you imaged yourself this is almost always a writer that skipped blocks it thought were empty (balenaEtcher's "trim unallocated space" does this); on a delivered unit it is a failing disk.

Fix. A unit you imaged: write the image again with a tool that writes every block (dd from the decompressed image). A delivered unit: synapseos-ab-update rollback from the console returns to the other slot if it still verifies; if both fail, the disk is the problem, contact support.

A start job is running for Wait for Network to be Online for two minutes​

Cause. A managed interface has no link or no address, and systemd-networkd-wait-online waits for every managed interface. The unit boots fine afterwards; only the wait is lost.

Fix. Cable or configure every interface that networkctl lists as configuring, or leave it: the delay is bounded and nothing depends on it.

The unit boots but has no address​

Cause. No .network file matched the interface. The default runs DHCP on interfaces named en* and eth* only; an interface with another name, or one you gave a .network file with a wrong MACAddress=, stays unconfigured.

Fix. On the console, networkctl status shows which file applied to which interface (unmanaged means none). Match on the MAC the console shows, then networkctl reload. See Configuring the appliance.

synapseos-ab-update status shows the previous slot running after an update and a reboot​

Cause. The unit rolled back: the new slot did not confirm within its trial boots. The login banner and /etc/os-release do not tell you this; they name the build the unit was delivered with and are not rewritten by updates. status and /usr/lib/os-release name the running one.

Fix. journalctl -b -1 -u synapse-agent shows why the data plane did not come up in the previous boot. See "A new slot keeps rolling back" below.

Access​

Permission denied (publickey)​

Cause. SSH accepts root with a key only, and the key offered is not in /root/.ssh/authorized_keys. As shipped, a password is not accepted over SSH; a unit delivered with password SSH turned on carries a drop-in under /etc/ssh/sshd_config.d/ that says so, and sshd -T | grep -i passwordauthentication shows which applies.

Fix. Use the key delivered with the unit (ssh -i), or add yours from the console. See Logging in.

The console asks for a password and refuses every one​

Cause. No password is set. Root is locked until one is.

Fix. Over SSH, passwd root. Set it before changing addressing, since the console is the way back in. See Logging in.

The serial console shows nothing​

See Serial console: the speed, the port and the driver, in that order.

Updates​

no UPDATE_BASE_URL in /etc/synapseos/update.conf -- offline appliance; use: install <bundle-dir>​

Cause. The unit has no channel configured, so check and update have nowhere to look.

Fix. Put the URL Gen0Sec gave you for this platform in the file, or update from external media. See Updating.

cannot reach <base>​

Cause. The update host is not reachable from the unit: no route, a firewall, DNS, or a proxy the updater was not told about.

Fix. curl -sS -o /dev/null -w '%{http_code}\n' <base> from the unit tells which. A proxy goes in update.conf with export. See Network requirements.

bundle serial N is not newer than the running M. Refusing a downgrade​

Cause. The channel, or the media, holds a version equal to or older than the one running. Equal counts: reinstalling the running version is refused too.

Fix. If that is what you want, synapseos-ab-update update --allow-downgrade (or install <dir> --allow-downgrade). See Updating.

bundle is variant 'X' but this appliance runs 'Y'​

Cause. The update was built for a different variant (hardened versus baseline). Installing it would change the unit's security posture, so the updater refuses, and the --allow-downgrade its message mentions does not override this check.

Fix. Check UPDATE_BASE_URL against what you were given for this platform; this usually means the wrong channel. A unit that is meant to change variant is re-imaged.

SIGNATURE VERIFICATION FAILED or the UKI is not signed by any certificate this device trusts​

Cause. Either the archive was damaged in transit, or it was signed with a key this unit does not trust: a build for another platform, or a unit whose trusted keys predate a key rotation.

Fix. Compare the archive's checksum with the one published beside it. If it matches, the keys are the problem; contact support with the output of synapseos-ab-update status, since a unit in this state cannot take an update on its own.

slot B FAILED verity verification -- not switching to it​

Cause. The slot was written but its content does not match the hash tree it came with: a damaged archive, or faulty media.

Fix. Nothing changed on the unit; the bootloader still points at the running slot. Unpack the archive again from a fresh copy and run install again.

A new slot keeps rolling back​

Cause. The new system booted but Synapse (and the jailer on hardened units) did not reach a running state within 120 seconds, so the slot was never confirmed and the bootloader returned to the previous one after three attempts.

Fix. journalctl -b -1 -u synapse-agent shows what the agent did in the trial boot. The usual causes are a configuration key the new version rejects (compare /etc/synapse/config.yaml with /usr/share/synapse/<new version>/config.yaml) and a pinned Synapse version the new image no longer ships. Fix the configuration, then synapseos-ab-update update again.

A service the release notes mention reports disabled after the update​

See Known limitations: once per unit, systemctl enable --now <unit>.

Synapse​

synapseos-synapse use refuses the version​

Cause. Either the version is not on the unit (list shows what is), or its binary failed the test run use performs before switching: a release that cannot start on this image.

Fix. list first. If the version is there and still refused, the message from the test run says why; the running version is untouched. See Synapse versions.

synapse-agent.service: Failed to create reference to PID from file '/run/synapse-agent.pid': Invalid argument​

Cause. A message systemd prints while the agent writes its PID file; the next line is Started Synapse. It does not indicate a failure.

Fix. None. If the unit is actually not filtering, look at the lines after it.

The proxy was enabled and the agent stopped filtering​

Cause. synapse-proxy.service was started with the shipped unit file, so it read the agent's configuration and took over the kernel packet filter.

Fix. systemctl stop synapse-proxy, then follow Running the proxy from the first step: the proxy needs its own configuration before it starts.

The proxy answers 404 for every host​

Cause. /etc/synapse/upstreams.yaml did not parse, so the proxy has no routes at all. One bad host takes every host down.

Fix. journalctl -u synapse-proxy -n 50 names the line. Every terminating host needs its tls.terminate block. See Running the proxy.

Collecting diagnostics for support​

One command gathers what support asks for first, into a file you can send:

D=/var/tmp/diag-$(hostname)-$(date -u +%Y%m%dT%H%M%SZ); mkdir -p "$D"
synapseos-ab-update status  > "$D/ab-update-status.txt" 2>&1
synapseos-synapse status    > "$D/synapse-status.txt"   2>&1
cat /etc/os-release         > "$D/os-release.txt"
networkctl status           > "$D/networkctl.txt"       2>&1
nft list ruleset            > "$D/nft.txt"              2>&1
journalctl -b    --no-pager > "$D/journal-this-boot.txt"
journalctl -b -1 --no-pager > "$D/journal-previous-boot.txt" 2>/dev/null
cp /var/log/audit/audit.log /var/log/messages "$D/" 2>/dev/null
tar -C /var/tmp -czf "$D.tar.gz" "$(basename "$D")" && rm -r "$D" && echo "$D.tar.gz"

Copy it off with scp, then delete it; the journal in it may contain addresses from your traffic. Send it to [email protected] with the message you saw and when.