# Gen0Sec — full documentation > Active NDR with eBPF/XDP. Kernel-native threat detection at wire speed — without decrypting a packet. Every page concatenated, in the order of the index at /llms.txt. --- # Getting Started Source: https://docs.gen0sec.com/docs/getting-started/ # Getting Started Synapse is a high-performance Active NDR agent built in Rust. On Linux it filters and fingerprints in the kernel via eBPF/XDP. A Windows agent ships too, capturing at L4 through an NDIS filter and able to use eBPF for Windows for XDP filtering. The unified firewall rule backend is still in development there, so pages below describe Linux unless they say otherwise. ## Quick Start ### 1. Get your credentials Two different secrets, from two places in the dashboard: | | Where | For | |---|---|---| | **Repository credential** | **Clients** → **Repository Access** | Downloading the packages | | **API key** | **API keys** | The running agent, as `platform.api_key` | The repository credential is scoped per host — tick APT, RPM, Docker or Releases for the channels you install from. A key without the right scope fails as an authentication error. ### 2. Pick your build Two choices — a role, and whether the ML models come with it: | Package | Role | ML | |---|---|:---:| | `synapse-agent` | Telemetry, firewall enforcement and IDS on the host. Most machines want this. | — | | `synapse-proxy` | Terminates traffic you route through it, inspects it at L7. | — | | `synapse-agent-ml` / `synapse-proxy-ml` | The same, plus bundled models for on-host inference. | Yes | The four conflict, so one at a time — and the systemd unit is named after the package you chose. ### 3. Add the repository Debian and Ubuntu below; RHEL, binary, Docker, Kubernetes and Windows are on the [Installation Guide](../client/installation.mdx). ```bash export GEN0SEC_KEY='' # The credential apt authenticates with sudo install -d -m 0755 /etc/apt/auth.conf.d printf 'machine apt.gen0sec.com login x password %s\n' "$GEN0SEC_KEY" \ | sudo tee /etc/apt/auth.conf.d/gen0sec.conf > /dev/null sudo chmod 600 /etc/apt/auth.conf.d/gen0sec.conf # The key it verifies the repository against, from a keyserver rather than # from a Gen0Sec host — so trust needs no credential of its own. sudo install -d -m 0700 /root/.gnupg sudo gpg --keyserver hkp://keyserver.ubuntu.com:80 --no-default-keyring \ --keyring /usr/share/keyrings/gen0sec.gpg \ --recv-keys C1631499081A7146332D2038798FF49A6E9DA146 # The source itself echo "deb [signed-by=/usr/share/keyrings/gen0sec.gpg] https://apt.gen0sec.com stable main" \ | sudo tee /etc/apt/sources.list.d/gen0sec.list > /dev/null ``` ### 4. Install ```bash sudo apt-get update sudo apt-get install -y synapse-agent sudo systemctl enable --now synapse-agent ``` ### 5. Connect it to the platform ```yaml # /etc/synapse/config.yaml platform: api_key: "" ``` ```bash sudo systemctl restart synapse-agent ``` Then check it came up: ```bash systemctl status synapse-agent journalctl -u synapse-agent -f ``` A healthy start logs the interfaces it attached to. ## Modes Agent and Proxy differ in what they can enforce. You select between them by installing the matching package, above. | Feature | Proxy | Agent | |---------|:-----:|:-----:| | **HTTP/HTTPS Reverse Proxy** | Yes | No | | **TLS Termination** | Yes | No | | **Upstreams & Load Balancing** | Yes | No | | **WAF, Rate Limiting, CAPTCHA** | Yes | No | | **Content Scanning (ClamAV)** | Yes | No | | **XDP Firewall & Access Rules** | Yes | Yes | | **Threat Intelligence & GeoIP** | Yes | Yes | | **JA4+ Fingerprinting** | Full | Full on Linux, L4-only on Windows | | **BPF Stats & TCP Fingerprinting** | Yes | Yes | | **File/Syslog Logging** | Yes | Yes | ### Agent Mode (Default) Standalone agent focused on access rules enforcement without HTTP/HTTPS proxy. Ideal for network-level protection where you don't need request proxying. ```yaml mode: "agent" # default, can be omitted ``` Use cases: - Network-level firewall protection without proxying - Access rules enforcement at the edge - Kernel-level IP blocking without HTTP overhead - Integration with existing reverse proxies or load balancers ### Proxy Mode Full-featured reverse proxy with HTTP/HTTPS support, forwarding requests to upstream servers while applying access rules and threat intelligence at the kernel level. ```yaml mode: "proxy" proxy: address_http: "0.0.0.0:80" address_tls: "0.0.0.0:443" upstream: conf: "/etc/synapse/upstreams.yaml" ``` ## Requirements | Requirement | Minimum | Recommended | Notes | |-------------|---------|-------------|-------| | **Kernel** | 4.18+ | 5.4+ | XDP support required | | **glibc** | 2.31+ | 2.35+ | For binary releases | | **Architecture** | x86_64 | x86_64, aarch64 | ARM64 supported | | **Memory** | 128 MB | 512 MB+ | Depends on traffic | | **Disk** | 100 MB | 500 MB+ | For logs and MMDB files | | Dependency | Required | Purpose | |------------|----------|---------| | **libbpf** | Yes | eBPF program loading | | **Redis** | Optional | Shared certificate store across proxy instances; storage defaults to file | | **ClamAV** | Optional | Content scanning | ### Tested Distributions | Distribution | Version | Status | |--------------|---------|--------| | **Ubuntu** | 24.04 LTS | Tested, Recommended | | **Ubuntu** | 22.04 LTS | Tested | | **Ubuntu** | 20.04 LTS | Tested | | **Debian** | 12 (Bookworm) | Tested | | **RHEL/Rocky/Alma** | 9.x | Tested | | **Fedora** | 39+ | Compatible | | **Amazon Linux** | 2023 | Compatible | ## Next Steps - [Synapse Overview](../client) - Full feature overview - [Installation Guide](../client/installation) - Detailed installation options - [Configuration](../client/configuration) - Complete configuration reference - [Access Rules](../features/access-rules) - Configure network access rules - [Threat Detection](../features/threat-detection) - Threat intelligence features ## Resources - [Dashboard](https://app.gen0sec.com) - [Discord Community](https://discord.gg/jzsW5Q6s9q) --- # Use Cases Source: https://docs.gen0sec.com/docs/use-cases/ # Use Cases Synapse is several enforcement layers that share one set of signals. Which layers you turn on depends on what you are trying to stop, so these pages are organised by the problem rather than by the feature. Each one states the scenario, which layers answer it, what to turn on, and — the part most product docs leave out — where the approach stops working. | If you are facing | Start here | |---|---| | Automated login attempts, scraping, carding | [Stop bots and credential stuffing](./stop-bots.mdx) | | Floods that saturate a host before the app sees them | [Absorb a volumetric attack](./absorb-volumetric-attack.mdx) | | Users uploading files into your service | [Block malware in uploads](./block-malware-uploads.mdx) | | An application you cannot patch or redeploy | [Protect an app you cannot change](./protect-legacy-app.mdx) | | An intruder already inside the network | [Detect lateral movement](./detect-lateral-movement.mdx) | | A host you believe is already compromised | [Contain a compromised host](./contain-compromised-host.mdx) | ## Try them without installing anything The **[Security Layer Playground](https://playground.gen0sec.com/)** runs these scenarios against the layers in a browser. You pick an attack, send it, and watch which layer stops it and why — no agent, no cluster, no account. Twelve scenarios ship with it, and they line up with the pages above: | Scenario | Layer that answers it | Page | |---|---|---| | XSS, SQL injection, path traversal, CSRF | WAF | [Protect an app you cannot change](./protect-legacy-app.mdx) | | DDoS attempt | Access rules | [Absorb a volumetric attack](./absorb-volumetric-attack.mdx) | | Malware upload | Content scanning | [Block malware in uploads](./block-malware-uploads.mdx) | | Brute force | WAF rate limiting | [Stop bots and credential stuffing](./stop-bots.mdx) | | Botnet, crypto mining, Tor exit node, VPN/proxy | Threat intelligence | [Threat Detection](../features/threat-detection.mdx) | | Request from a blocked country | Access rules | [Access Rules](../features/access-rules.mdx) | There is a clean request in there too, which is the one worth sending first — knowing what *passes* makes the blocks legible. A live sandbox is available beyond the browser demo, and that one does need an account. ## The one thing to understand first Cost rises sharply as traffic moves up the stack. A packet dropped in [XDP](../features/access-rules.mdx) never allocates a socket buffer; a request refused by the [WAF](../features/waf.mdx) has already been accepted, decrypted and parsed. So the pattern in every page below is the same: **push each decision as far down as it can be made correctly.** Address-level facts belong in the kernel. Anything that needs the request body cannot go there, and pretending otherwise is how people end up with a firewall that blocks their own users. Which layers you have at all depends on the mode you installed — the agent enforces in the kernel, the proxy adds everything that requires terminating the request. See [Modes](../client/index.mdx#modes). --- # Absorb a volumetric attack Source: https://docs.gen0sec.com/docs/use-cases/absorb-volumetric-attack # Absorb a volumetric attack A flood is arriving faster than your application can refuse it. The problem is rarely that the requests are hard to identify — it is that *identifying them at all* costs more than you have left, because by the time a userspace process sees a packet the kernel has already allocated for it. ## Drop it before the cost is paid [Access rules](../features/access-rules.mdx) are enforced by [Hillock](../components/hillock/) at the **XDP hook**, which runs in the driver before routing and before a socket buffer exists. That is the cheapest drop a Linux host can perform. The tradeoff is deliberate: XDP sees an address and a port, not a request. So this layer answers *is this source allowed to talk to me at all*, and nothing more subtle. | | Access rules | Firewall rules | |---|---|---| | Hook | XDP, before routing | TC, after ingress and before egress | | Matches | IP address | IP, port range, protocol, TCP flags | | Direction | Inbound | Inbound and outbound | | Use it for | Absorbing a known-bad source at volume | Expressing a specific allowed conversation | ## Where the list comes from You are not expected to write it by hand. [Threat detection](../features/threat-detection.mdx) distributes known-bad sources as a database the agent pulls and installs into the kernel maps. ASN and country are resolved on the platform side and arrive already expanded, so the datapath never does a lookup. `--show-firewall-rules` reports what the kernel actually holds, which is the thing to check when a rule appears not to work. ## Turning it on ```yaml # /etc/synapse/config.yaml firewall: mode: "auto" # auto, xdp, nftables, iptables, none platform: threat: enabled: true ``` `auto` picks XDP where the kernel supports it and falls back to nftables, then iptables. The rule you wrote does not change — only the machinery under it. See [Firewall Rules](../features/firewall-rules.mdx). :::tip[See it happen first] The **[playground](https://playground.gen0sec.com/)** has a *DDoS Attempt* scenario and a geo-blocked *Germany Request*, both stopped by access rules — the same layer described here, without attaching anything to a NIC. ::: ## Where this stops working - **XDP is not universally available.** It needs a driver that supports it and a kernel with BTF. Without them you land on nftables or iptables, which still work and still drop early, but not at driver level. - **An address list cannot stop a distributed attack from addresses you have never seen.** This absorbs known-bad and repeat sources cheaply; it is not a substitute for upstream capacity when the flood is genuinely novel and genuinely large. - **Layer 7 floods look like valid traffic here.** A slow flood of well-formed requests from unremarkable addresses passes XDP untouched and has to be handled at the proxy — see [Stop bots](./stop-bots.mdx). - **Restarting the agent has a brief window.** The XDP link is pinned and re-attachment is not instantaneous; plan restarts accordingly. ## See also - [Access Rules](../features/access-rules.mdx) — what XDP matches and where it runs - [Firewall Rules](../features/firewall-rules.mdx) — port, protocol and TCP flag matching - [Hillock](../components/hillock/) — the kernel firewall doing the enforcing - [Threat Detection](../features/threat-detection.mdx) — where the source list comes from --- # Block malware in uploads Source: https://docs.gen0sec.com/docs/use-cases/block-malware-uploads # Block malware in uploads Your service accepts files — attachments, avatars, documents, imports. Anything a user uploads is content you did not write, arriving inside a request your application is about to trust. ## Scan the upload path, and only the upload path [Content scanning](../features/content-scanning.mdx) hands the request body to ClamAV before it is forwarded upstream. A signature hit refuses the request with 403, so infected content never reaches your storage, your queue or a downstream worker. It is proxy-mode only, for the obvious reason: something has to have the decrypted body. The scanner is armed in configuration and **aimed by a rule**. Enabling it on its own scans nothing: ```yaml proxy: content_scanning: enabled: true clamav_server: "localhost:3310" max_file_size: 10485760 unscannable: block ``` Then write a [WAF](../features/waf.mdx) rule with the `content_scanning` action to select the traffic: ```text starts_with(http.request.path, "/upload") ``` Everything outside that path is untouched and costs nothing — which is the whole point. Scanning every request body would put a ClamAV round trip in front of every API write. :::tip[If nothing is being scanned, look for the rule] `enabled: true` with no `content_scanning` rule inspects nothing, and the agent says so on every load: *content scanning is enabled but no WAF rule uses the content_scanning action*. ::: ## Decide what happens to what you cannot scan An upload can be selected by your rule and still be uninspectable — over `max_file_size`, or chunked past it. `unscannable` is that decision, and it defaults to refusing: | | | |---|---| | `block` *(default)* | Refuse with **413**. The rule asked for a guarantee that could not be provided. | | `pass` | Forward unscanned, logged. Oversized uploads keep working; they reach the backend uninspected. | Choosing `pass` is legitimate — just make it deliberately, because it is the setting that quietly turns a scanned endpoint into a partially scanned one. ## Where this stops working :::tip[See it happen first] The **[playground](https://playground.gen0sec.com/)** has a *Malware Upload* scenario that walks the file through content scanning and shows the refusal, with no ClamAV to stand up. ::: - **ClamAV is signature-based.** It catches known malware. It does not catch a targeted payload written for you. - **The scanner fails open.** If ClamAV is unreachable the request proceeds and the failure is logged. Alert on scan failures; a quietly dead scanner looks identical to a clean one. - **Only what your rule selects is scanned.** That is the design, but it means a second upload endpoint added later is unprotected until the rule covers it. - **Latency scales with body size**, plus a round trip, on the selected traffic. - **You need a ClamAV you run and keep updated.** A stale signature database is a quiet failure. ## See also - [Content Scanning](../features/content-scanning.mdx) — the full behaviour, memory budget and verdict cache - [WAF](../features/waf.mdx) — the rule engine and the `content_scanning` action - [Security Event Export](../features/event-export.mdx) — records of what was refused --- # Contain a compromised host Source: https://docs.gen0sec.com/docs/use-cases/contain-compromised-host # Contain a compromised host You have a host you no longer trust. The instinct is to pull its network cable, but that destroys the volatile evidence you need and takes the service down with it. What you usually want instead is narrower: stop it reaching anything new, keep watching it, and leave the machine running while you investigate. ## Cut egress, keep visibility [Firewall rules](../features/firewall-rules.mdx) match in **both directions**. Where access rules answer *is this source allowed to reach me*, firewall rules answer *is this conversation allowed* — and they can stop a host reaching out, not just being reached. Enforcement is stateless and runs in the kernel at the TC hook, so nothing sits in the application's path and a compromised userspace process cannot bypass it. | Matches on | | |---|---| | Address | Source and destination, with a CIDR prefix | | Port | Source and destination ranges | | Protocol | TCP, UDP, or any | | TCP flags | A mask and the value it must equal | | Direction | `ingress` or `egress` | | Action | `allow` or `drop` | ## The containment shape Deny egress by default, then allow only what the investigation needs — your log collector, your management plane — and leave ingress monitoring untouched so you keep seeing what the host receives. :::caution[Order decides] Evaluation stops at the first matching rule, lowest `order` first. An `allow` placed below a broad `drop` never runs. Give the exceptions a lower `order` value than the deny they are meant to escape. ::: Only active rules are loaded into the kernel, so disabling a rule removes it from the datapath rather than leaving it resident and inert. ## Rules survive the host The same policy can be written once and enforced by whatever the host offers — **XDP/eBPF → nftables → iptables** — which is what lets one containment policy cover a fleet that is not uniform. ## Where this stops working - **Stateless means exactly that.** There is no connection tracking, so a rule permitting a reply has to exist in its own right. This surprises people who have written stateful firewall rules for years. - **Matching stops at the transport header.** You can cut a destination and a port. You cannot express *block this TLS SNI* here — that is evaluated on the host by [Amygdala](../components/amygdala/), which then installs the resulting block. - **A local-only compromise is untouched.** Egress rules do nothing about something that never leaves the box. - **Containment is not eradication.** This buys time and limits blast radius; it is not a substitute for rebuilding the host. ## See also - [Firewall Rules](../features/firewall-rules.mdx) — every dimension, and the TCP flag semantics - [Access Rules](../features/access-rules.mdx) — the cheaper, inbound-only counterpart - [Detect lateral movement](./detect-lateral-movement.mdx) — finding the host in the first place - [Live Traffic View](../features/live-traffic.mdx) — watch what it is still trying to reach --- # Detect lateral movement Source: https://docs.gen0sec.com/docs/use-cases/detect-lateral-movement # Detect lateral movement An intruder is already inside. The perimeter told you nothing, because nothing crossed it — the traffic that matters is one workload reaching another, and on a Kubernetes cluster that looks like an ephemeral IP talking to another ephemeral IP. The reason east-west detection is hard is not visibility. It is that raw flows between pod addresses are unreadable: by the time you investigate, the pod is gone and the address has been reused. ## Give the flows names The [service graph](../features/service-graph.mdx) resolves addresses to identity — which workload, which namespace, which app — so a flow reads as *checkout talked to the payments database* rather than as two IPs. That mapping is built by the [operator](../client/synapse-operator.mdx) from the cluster's own Pods and uploaded for agents to consume: | Producer | Builds | From | |---|---|---| | Identity producer | pod IP → workload, namespace, app | Cluster Pods | | Edge producer | the allow-list of intended conversations | Cluster NetworkPolicy | ## Compare observed against declared The second producer is what turns visibility into detection. Your NetworkPolicy already describes which workloads are *supposed* to talk to each other; compiling it into declared edges means an observed flow can be checked against it. A conversation nobody declared is the signal. It is not proof of compromise — but it is a short, reviewable list, which is more than a flow log gives you. ## Turning it on The producers are operator flags, exposed as chart values. Both need an API key with `identity:write`: ```yaml operator: identityProducer: enabled: true edgeProducer: enabled: true apiKeySecret: name: synapse-operator-api-key ``` Deploy the agent as a DaemonSet so every node observes its own traffic — see [Helm Charts](../client/helm-charts.mdx). :::caution[Check your chart version] The producers need `pods`, `namespaces` and `networkpolicies` in the operator's ClusterRole. Charts before synapse-stack 0.6.0 granted none of them, so both producers fail on a default install. [Synapse Operator](../client/synapse-operator.mdx) has the detail and the workaround. ::: ## Where this stops working - **Identity is as good as your labels.** Workloads that are not labelled meaningfully resolve to names that do not help an investigation. - **A cluster with no NetworkPolicy has no declared edges**, so there is nothing to compare against and the edge producer contributes nothing. - **The baseline is periodic.** The full upload is a cold-start and resync source; freshness between uploads comes from event-driven deltas. A pod that lives and dies inside a gap may not resolve. - **This is detection, not prevention.** Enforcement still happens in the kernel, through the [firewall](../features/firewall-rules.mdx). Seeing an undeclared conversation does not stop it. - **Multiple clusters need `--cluster-id`.** Without it they share one path and the last writer wins. ## See also - [Service Graph](../features/service-graph.mdx) — identity, declared edges, querying it - [Synapse Operator](../client/synapse-operator.mdx) — the producers and their RBAC - [Contain a compromised host](./contain-compromised-host.mdx) — what to do once you have found one - [Security Event Export](../features/event-export.mdx) — getting the flows to your SIEM --- # Protect an app you cannot change Source: https://docs.gen0sec.com/docs/use-cases/protect-legacy-app # Protect an app you cannot change The application is a vendor appliance, an unmaintained internal service, or something with a change freeze on it. You cannot add a library, you cannot patch it this quarter, and a CVE has just been published against the stack it runs on. The answer is to move the security boundary off the application and in front of it. ## Terminate, inspect, forward In proxy mode Synapse sits in the request path: it terminates TLS, applies policy, and forwards to the unchanged application over its existing port. The application needs no modification and does not need to know it is behind anything. What you gain at that point: | | | |---|---| | [WAF](../features/waf.mdx) | Refuse requests by path, method, header, body or threat score | | [TLS termination](../client/configuration.mdx) | Your certificates, or SNI passthrough to a backend that holds them | | [Load balancing](../features/load-balancing.mdx) | Weighted backends and health checks in front of one fragile service | | [Content scanning](../features/content-scanning.mdx) | Inspect selected uploads before they reach the app | | [Event export](../features/event-export.mdx) | Structured records of what was refused and why | ## Virtual patching The practical use of the WAF here is narrow and specific: block the exact request shape that reaches the vulnerability, and leave everything else alone. That buys time until the real fix ships. ```text http.request.method == "POST" and starts_with(http.request.path, "/admin") ``` Rules are [wirefilter](../features/waf.mdx) expressions with a documented field list, so the rule you write is checked at load rather than failing open at runtime. :::caution[First match wins] Evaluation short-circuits at the first matching rule. An allow placed below a broad block never runs — the block has already decided. Order exceptions above the rules they are meant to escape. ::: ## Turning it on ```yaml # /etc/synapse/config.yaml mode: "proxy" proxy: address_http: "0.0.0.0:80" address_tls: "0.0.0.0:443" upstream: conf: "/etc/synapse/upstreams.yaml" ``` Upstreams are a separate hot-reloaded file, so backends change without a restart. The v2 schema and the migration table are in [Configuration](../client/configuration.mdx). On Kubernetes, the same thing is driven from native Ingress objects instead — see [Kubernetes Ingress](../client/kubernetes-ingress.mdx). :::tip[See it happen first] Four of the **[playground](https://playground.gen0sec.com/)** scenarios are exactly this case — *XSS Attack*, *SQL Injection*, *Path Traversal* and *CSRF Attack* — an unmodified application behind a WAF that refuses the request shape. ::: ## Where this stops working - **Only traffic through the proxy is protected.** Anything that reaches the application by another route — a second interface, an internal caller, a management port — bypasses every rule here. Pair it with [firewall rules](../features/firewall-rules.mdx) that make the proxy the only path in. - **A WAF rule is a shape, not a fix.** It blocks the exploit you described. A variant that reaches the same bug differently gets through. - **Terminating TLS means holding certificates.** If that is unacceptable, passthrough is available, but then the WAF and content scanning have nothing to read. - **You have added a hop.** Health checks and timeouts now matter; they are configured under `proxy.upstream`. ## See also - [WAF](../features/waf.mdx) — the expression language and every field it exposes - [Load Balancing & Proxy](../features/load-balancing.mdx) — upstreams, weights, health checks - [Configuration](../client/configuration.mdx) — TLS, upstreams, and what reloads without a restart - [Kubernetes Ingress](../client/kubernetes-ingress.mdx) — the same policy from Ingress objects --- # Stop bots and credential stuffing Source: https://docs.gen0sec.com/docs/use-cases/stop-bots # Stop bots and credential stuffing Someone is driving your login form with a script. The requests are well-formed, the user agent says Chrome, the addresses rotate through a residential proxy pool, and rate limiting alone either misses them or starts catching real users. The reason this is hard with conventional tooling is that every field a bot controls, it lies about. What it cannot easily change is *how its TLS and TCP stack behaves* — and that is what Synapse matches on. ## What identifies the client [JA4+](../client/ja4-plus.mdx) fingerprints the connection itself: the TLS ClientHello, the TCP options and timing, the HTTP header order. A Go program, a headless Chrome, a curl build and a real browser produce visibly different fingerprints even when every header is identical. That gives you a handle that survives address rotation, because the fingerprint travels with the client, not with its IP. Acting on it is the [Smart Firewall](../features/smart-firewall.mdx): a fingerprint match installs a kernel block, so a scanner that changes address every request is still stopped by the thing that does not change. ## The layers, cheapest first | Layer | Decides on | Cost | |---|---|---| | [Access rules](../features/access-rules.mdx) | Address, ASN, country — pre-resolved on the platform side | Kernel drop, before routing | | [Threat detection](../features/threat-detection.mdx) | Known-bad addresses and fingerprints from the feed | Kernel drop | | [ML models](../features/machine-learning.mdx) | Is this fingerprint malicious, and how serious | On-host inference | | [Smart Firewall](../features/smart-firewall.mdx) | The fingerprint itself — block by who the client is | Kernel, after evaluation | | [WAF](../features/waf.mdx) | The request itself — path, method, headers, body | After decryption | | WAF `rate_limit` | How *often* this client is asking | After decryption | | WAF `challenge` | A CAPTCHA, when you are unsure | Full round trip | The fingerprint classifier is the piece that does the work here. It answers *is this client's JA4+ fingerprint malicious* with a probability, and the [reported accuracy](../features/machine-learning.mdx) is high — but read the section on using the output responsibly before you wire it straight to a block. ## Turning it on Fingerprinting runs in both modes; the classifier needs an `-ml` build, and CAPTCHA and the WAF need proxy mode. ```yaml # /etc/synapse/config.yaml mode: "proxy" classifier: enabled: true proxy: captcha: provider: "turnstile" # hcaptcha, recaptcha, turnstile, prosopo ``` Then express the policy as a [WAF](../features/waf.mdx) rule. The scheme exposes threat intelligence alongside the request, so a rule can combine both: ```text threat.score > 80 and threat.advice == "block" ``` Full field list and the operators available are on the [WAF](../features/waf.mdx) page. Rate limiting is a WAF action rather than a separate subsystem, which matters here: the same rule that identifies the client decides what happens to it. A credential-stuffing rule usually wants `rate_limit` rather than `block` — the login endpoint is legitimate, it is the *rate* that is not. Counters are per rule and per client address, with three settings on the rule: `requests` allowed per `period` seconds, then `duration` seconds at 429 once the threshold is crossed. `duration` is the penalty, not the window. :::tip[See it happen first] The **[playground](https://playground.gen0sec.com/)** has a *Brute Force* scenario, plus *Botnet Traffic*, *Tor Exit Node* and *VPN/Proxy Traffic* — four ways an automated client arrives, and which layer catches each. Send the *Clean Request* first so you know what a pass looks like. ::: ## Where this stops working - **A fingerprint is not an identity.** Every user of a given browser build shares one. Blocking a fingerprint outright blocks a population, which is why the classifier returns a probability and why a challenge is often the better response than a drop. - **Shared egress addresses carry shared reputation.** A CGNAT or corporate NAT address represents many people. Prefer the fingerprint, or a combination of signals, before blocking an address outright. - **A determined adversary can mimic a fingerprint.** This raises the cost of automation; it does not make it impossible. - **Rate limiting behind a load balancer counts the wrong address.** The bucket keys on the L4 peer, so an LB that rotates source addresses scatters one attacker across buckets that never reach the threshold, and an LB that collapses clients onto a few addresses makes real users limit each other. Set [`proxy.trusted_proxies`](../client/configuration.mdx#client-address-behind-a-proxy) before relying on a rate-limit rule in that topology. - **The agent sees less than the proxy.** Without terminating TLS you get the client-side fingerprints but not the request, so no WAF and no CAPTCHA. ## See also - [JA4+](../client/ja4-plus.mdx) — what each fingerprint captures - [Machine Learning Models](../features/machine-learning.mdx) — the classifier and its limits - [WAF](../features/waf.mdx) — the expression language and its fields - [Live Traffic View](../features/live-traffic.mdx) — watch fingerprints arrive while you tune --- # Synapse - Active NDR Firewall & Reverse Proxy Source: https://docs.gen0sec.com/docs/client/ # Synapse Synapse is a high-performance reverse proxy and firewall built with Rust, featuring: - **XDP-based packet filtering** for ultra-low latency protection at kernel level - **Multi-backend firewall** with automatic fallback (XDP > nftables > iptables > userland) - **Dynamic access rules** with automatic updates from Gen0Sec API - **JA4+ fingerprinting** complete suite: JA4, JA4H, JA4T, JA4L, JA4S, JA4X - **TLS termination** with your own certificates, or SNI passthrough to a backend that holds them - **Threat intelligence integration** with Gen0Sec API and Threat MMDB - **Content scanning** with ClamAV, aimed by a WAF rule at the traffic you choose - **Upstream routing** from a hot-reloaded file, with weighted backends and health checks - **Weighted load balancing** with hot-reloadable configuration :::info[Platform support] XDP and eBPF need Linux with kernel 4.18 or newer. Without them Synapse still runs, falling back to nftables and then iptables. A Windows agent ships as well, capturing at L4 — see [Installation](./installation.mdx#windows). ::: ## Install it Synapse installs from the Gen0Sec package repositories, as a container, or into Kubernetes. The mode you want is chosen by which package you install. The repositories are authenticated and signed, and there are two credentials involved, so the steps are worth following in order rather than pasting a command. Every channel — Debian, RHEL, binary tarball, Docker, Kubernetes and Windows — is on the **[Installation Guide](./installation.mdx)**. ## Modes Synapse runs as an **agent** or a **proxy**, and the package you install decides which — `synapse-agent` or `synapse-proxy`. Both enforce in the kernel; the proxy adds everything that requires terminating the request. ### Proxy only | Capability | Agent | Proxy | |---|:--:|:--:| | HTTP/HTTPS reverse proxy | — | Yes | | TLS termination | — | Yes | | Upstreams & load balancing | — | Yes | | WAF ([wirefilter](../features/waf.mdx)) | — | Yes | | Rate limiting | — | Yes | | CAPTCHA | — | Yes | | [Content scanning](../features/content-scanning.mdx) | — | Yes | | Request logging to the platform | — | Yes | ### Both modes Kernel enforcement is identical either way — the agent is not a cut-down build: - **XDP packet filtering**, with fallback to nftables, then iptables - **[Access rules](../features/access-rules.mdx)**, including dynamic updates from the platform - **[Firewall rules](../features/firewall-rules.mdx)** on address, port and protocol - **[Threat intelligence](../features/threat-detection.mdx)** and GeoIP - **Inline IDS**, with kernel-level blocking on a verdict - **BPF statistics** and TCP fingerprinting - **File and syslog logging**, and multiple network interfaces ### Where they differ in degree **[JA4+ fingerprinting](./ja4-plus.mdx)** is the one capability that is not simply on or off. The proxy captures the full suite natively from the TLS terminator. A Linux agent captures the same family through libssl uprobes and a TC-egress collector, so coverage depends on the host's libssl being uprobe-attachable. A Windows agent is L4-only. **Threat intelligence** applies at the request level in proxy mode, and at the network level in agent mode. ## Features ### Multi-backend Firewall - **Automatic fallback** - XDP/BPF > nftables > iptables > userland - **Dynamic access rules** - Allow/block lists auto-updated from Gen0Sec API - **BPF map enforcement** - Rules enforced at kernel level via XDP - **IPv4/IPv6 dual-stack** - Separate rule sets, zero-downtime updates - **BPF statistics** - Packet counters, dropped IP tracking (XDP only) ### Threat Intelligence - **IP reputation scoring** - Automatic scoring via Gen0Sec API - **Threat MMDB** - Local MaxMind database for offline lookups (auto-updated) - **GeoIP MMDB** - Country, ASN, and city-level geolocation (auto-updated) - **Bot detection** - Advanced detection and mitigation - **Redis caching** - Cached threat data for performance ### JA4+ Fingerprinting - **JA4** - TLS client fingerprinting from ClientHello - **JA4H** - HTTP header fingerprinting - **JA4T** - TCP fingerprinting from SYN packet options - **JA4L** - Latency fingerprinting from packet timing - **JA4S** - TLS server fingerprinting from ServerHello - **JA4X** - X.509 certificate fingerprinting ### WAF (Wirefilter Expressions) - **Expression engine** - Filter by request method, path, headers, and more - **Actions** - Allow, block, challenge or rate limit, based on expression matches - **Centralized management** - Expressions fetched from Gen0Sec API ### TLS Management - **Your own certificates** - Provide the certificate and key; Synapse terminates with them - **SNI passthrough** - Forward the connection encrypted when the backend must hold the key - **HTTPS enforcement** - Force HTTPS with HTTP upgrade responses - **On Kubernetes** - cert-manager issues and renews, as documented in [Kubernetes Ingress](./kubernetes-ingress.mdx) ## Architecture ### Components - **Multi-Backend Firewall** - XDP/nftables/iptables/userland packet filtering - **HTTP/TLS Servers** - HTTP traffic handling and HTTPS connection management - **Internal Services Server** - Unified server for CAPTCHA and certificate management - **Reverse Proxy** - Request forwarding to upstream services - **Upstreams Manager** - Routing with service discovery, weighted load balancing, and hot-reloading - **Threat Intelligence** - Gen0Sec API and Threat MMDB integration - **GeoIP Manager** - Country, ASN, and city-level geolocation via MMDB - **Access Rules Engine** - Dynamic IP allow/block lists with periodic API updates - **BPF Statistics Collector** - Kernel-level packet processing tracking - **TCP Fingerprint Collector** - SYN fingerprint extraction and analysis - **Fingerprint Engine** - Complete JA4+ suite: JA4, JA4S, JA4T, JA4TS, JA4H, JA4L, JA4SSH, JA4X, JA4D, JA4D6 - **CAPTCHA Engine** - Multi-provider CAPTCHA validation - **File/Syslog Loggers** - Rotating file-based and centralized syslog logging - **Event Queue** - Unified batch processing for logs, statistics, and events - **Redis Cache** - Certificates, threat intel, CAPTCHA, and content scan results ### Performance - **Ultra-low latency** - XDP filtering operates in kernel space - **High throughput** - Rust-based implementation with async I/O - **Memory efficient** - Minimal footprint with efficient caching - **Scalable** - Multiple network interfaces and concurrent connections ## Documentation - [**Installation Guide**](./installation) - Get Synapse up and running - [**Configuration**](./configuration) - Complete configuration reference - [**Daemon Mode**](./daemon-mode) - Run Synapse as a background service - [**JA4+ Fingerprinting**](./ja4-plus) - Complete fingerprinting suite - [**Testing Guide**](./testing) - Manual testing and validation ## Need Help? - Join our [Discord community](https://discord.gg/jzsW5Q6s9q) --- # Configuration Examples Source: https://docs.gen0sec.com/docs/client/config-examples # Configuration Examples Five files you can copy whole. Each one is complete: a key that is not in it keeps the default from [Configuration](./configuration.mdx), and nothing here depends on a companion snippet elsewhere on the page. :::info[Where these come from] They are adapted from `config/config.yaml` and `config/config_windows.yaml` as shipped with 0.8.8, not written from scratch. Two kinds of edit were made: keys the shipped samples set but the schema does not accept are removed, and the values are the ones a deployment would actually choose rather than the sample's "show every option" defaults. [What these leave out](#what-these-leave-out-and-why) says exactly what went and why, because the difference is visible the moment you compare a file here with the one in the package. ::: ## 1. Minimal agent An agent on one machine: kernel firewall on, fingerprints collected, nothing else. ```yaml # /etc/synapse/config.yaml mode: "agent" network: iface: "auto" firewall: mode: "auto" platform: api_key: "YOUR_API_KEY" logging: level: "info" file_logging_enabled: true log_directory: "/var/log/synapse" ``` Every one of those lines could be dropped and synapse would still start: | Line | What it does for you | | --- | --- | | `mode: "agent"` | Already the default. Spelled out because the other value, `"proxy"`, is what the next examples need, and a file that names its mode is one you can diff against them | | `network.iface: "auto"` | Attaches only to physical uplinks - loopback, `veth*`, CNI and tunnel devices are skipped, so an agent on a Kubernetes node does not land on the CNI datapath | | `firewall.mode: "auto"` | Picks the best available backend: XDP, then nftables, then iptables, then nothing | | `platform.api_key` | Without it synapse starts anyway and logs `Gen0Sec API key not provided. Some features may not work.` - threat intelligence, fleet bans and telemetry are the features in question | | `logging.file_logging_enabled: true` | **Off by default.** Without it every log line, including the access log on a proxy, goes to the console instead of `{log_directory}` | :::tip[The fingerprint stream needs nothing turned on] `logging.eventbridge_log.enabled` defaults to `true`, so `{log_directory}/eventbridge.log` is written by an agent and a proxy alike with no configuration at all. See [Testing](./testing.mdx) for what lands in it and how to read it. ::: ## 2. Agent with per-interface features A gateway with three NICs: a WAN link that gets everything, a LAN link that only gets the firewall, and a management NIC that is deliberately named with nothing enabled so no future change can start attaching to it. ```yaml # /etc/synapse/config.yaml mode: "agent" network: # `interfaces` replaces iface/ifaces - leave both of those unset. interfaces: wan: match: { name: eth1 } features: [firewall, ids, smart_firewall, fingerprint] require_native: true lan: match: { pattern: "enp3s*" } features: [firewall] mgmt: match: { mac: "52:54:00:12:34:56" } # No `features`: matched, listed in the startup summary, never attached. firewall: mode: "auto" amygdala: enabled: true ids: enabled: true rule_paths: - "/etc/synapse/rules/*.rules" address_vars: HOME_NET: "[10.0.0.0/8,172.16.0.0/12,192.168.0.0/16]" EXTERNAL_NET: "!$HOME_NET" # Alert only until the ruleset has been tuned on real traffic. enforce_block: false platform: api_key: "YOUR_API_KEY" logging: level: "info" file_logging_enabled: true log_directory: "/var/log/synapse" module_levels: synapse_idp: debug ``` Five things about this file that are easy to get wrong: - **A per-interface feature still needs its global switch - and the load fails without it.** The block says *where* a feature runs, not *whether*, and it will not let you say "here" for a feature that is globally off: `ids` or `ids_egress` with `ids.enabled` false, `smart_firewall` with `firewall.amygdala.enabled` false, and `firewall` with `firewall.mode: none` each stop the start with a named error. That is why both switches are set above. `fingerprint` is the one feature checked against nothing. - **`match: { name: … }` and `match: { mac: … }` must match a link on this host**, or the start fails with `matches no interface on this host`. So two lines above are the ones to edit before this file runs anywhere: `name: eth1` and the `mac:`. A `pattern:` that matches nothing is only a warning, which is why `lan` is written that way. - **With this block set there is no netfilter fallback.** `firewall.mode: auto` falls back XDP → nftables → iptables only when the block is absent; with it, no XDP means **no kernel firewall at all** and an error line saying so, because the netfilter backends are host-wide and would enforce on the `mgmt` NIC this file deliberately leaves out. - **`require_native: true` refuses the interface** whose driver offers only generic (SKB) XDP, rather than attaching there and running degraded. On a NIC without native XDP that is a link synapse does not protect - which is the point, but it has to be the choice you meant. - **`ssl_uprobe` and `tls_offload` are not valid here.** They are features in the schema but not per-interface ones, and naming either fails the load. The five that belong in this block are `firewall`, `ids`, `ids_egress`, `smart_firewall` and `fingerprint`. [Per-Interface Features](./interfaces.mdx) has the matching rules, the validation table and what each feature costs. ## 3. Proxy terminating TLS A reverse proxy holding its own certificates, with routing in a separate file. ```yaml # /etc/synapse/config.yaml mode: "proxy" network: iface: "auto" firewall: mode: "auto" ids: enabled: true rule_paths: - "/etc/synapse/rules/*.rules" enforce_block: false post_tls: # Inspect decrypted HTTP inside the proxy process. mode: inline platform: api_key: "YOUR_API_KEY" proxy: address_http: "0.0.0.0:80" address_tls: "0.0.0.0:443" # Directory scanned for certificates, and the logical name served when no # loaded certificate matches the requested hostname. certificates: "/etc/synapse/certs" default_certificate: "example.com" tls_grade: "medium" upstream: conf: "/etc/synapse/upstreams.yaml" healthcheck: method: "HEAD" interval: 2 # Per-client connection cap, counted per client address (per /64 for IPv6). # It is ON by default at 512, so this line is documentation rather than a # change: `0` turns it off, and a number raises or lowers it. max_downstream_connections_per_client: 512 logging: level: "info" file_logging_enabled: true log_directory: "/var/log/synapse" ``` ```yaml # /etc/synapse/upstreams.yaml version: 2 proxy: fingerprints: forward: true header_names: g0s headers: response: - "Strict-Transport-Security: max-age=31536000; includeSubDomains" hosts: example.com: tls: terminate: # A logical NAME, not a path: the file is /etc/synapse/certs/example.com.crt # with its key beside it as example.com.key. cert: example.com paths: "/": upstream: "10.0.0.10:8080" force_https: true "/api/": upstream: "10.0.0.11:8080" timeouts: connect: 5 read: 60 ``` :::warning[`cert:` takes a name, not a path] `cert: /etc/synapse/certs/example.com.crt` looks right and is not: the value is matched against the logical names of the certificates found in `proxy.certificates`, so a path matches nothing and the host silently falls back to the default certificate with a warning. A logical name comes from one of two layouts: | On disk | Logical name | | --- | --- | | `example.com.crt` with `example.com.key` beside it | `example.com` | | `…/live/example.com/fullchain.pem` with `privkey.pem` beside it | `example.com` | Anything else in the directory is skipped, not reported as an error - including a lone `example.com.pem`, because the flat layout is matched on `.crt` and its sibling `.key`. [Upstreams & Routing](./upstreams.mdx) has the rest of the certificate behaviour. ::: Three notes on the main config: - **`mode: "proxy"` is load-bearing.** The schema default is `"agent"`, whatever the comment in the shipped sample says, and an agent binds no listener however complete the `proxy:` section is. - **Behind a load balancer, set `proxy.trusted_proxies`.** The per-client cap keys on the socket peer address, so every connection arriving from one load balancer shares a single key and the 512 becomes a cap on the whole site. A peer listed in `trusted_proxies` is exempt from the per-client cap; it still counts toward `proxy.max_downstream_connections`. - **The internal server is on by default**, on `127.0.0.1:9180`, serving `/health` and `/metrics`. It is also reachable through the public listeners, for the reason in [Testing](./testing.mdx#health-check). Certificates are not obtained by synapse here - see the [ACME warning](./configuration.mdx#proxy-proxy-mode) before reaching for `proxy.acme`. ## 4. TLS passthrough for a database Postgres on 5432, passed through without terminating, with the real client address handed to the backend. The proxy also serves HTTP on 80 so a redirect or an ACME HTTP-01 solver has somewhere to live. ```yaml # /etc/synapse/config.yaml mode: "proxy" firewall: mode: "auto" platform: api_key: "YOUR_API_KEY" proxy: # Named listeners, instead of address_http / address_tls. Setting both forms # is refused: there is no precedence between them, so rather than dropping # one, the proxy logs the refusal and binds nothing. listeners: - name: web address: "0.0.0.0:80" mode: http - name: pg address: "0.0.0.0:5432" mode: tls # Required by any `mode: tls` listener, passthrough included. See below. certificates: "/etc/synapse/certs" upstream: conf: "/etc/synapse/upstreams.yaml" logging: level: "info" file_logging_enabled: true log_directory: "/var/log/synapse" ``` ```yaml # /etc/synapse/upstreams.yaml version: 2 hosts: db.example.com: # Only this listener serves the host, so it is never answered on :80. listeners: [pg] passthrough: tls: upstreams: - { addr: "10.0.0.20:5432", weight: 2 } - { addr: "10.0.0.21:5432", weight: 1 } lb: # `round_robin` ignores `weight:` entirely. Weighted entries need # this method, or the weights are config that does nothing. method: weighted_round_robin health_check: type: tcp # Seconds a tunnel may carry nothing before both sockets are closed. # 0 means no bound. This is not `timeouts.idle`, which is the upstream # connection pool's keepalive and would tear down live sessions. idle_timeout: 3600 max_connections: 500 # Prepend a PROXY protocol v2 header so the database sees the real client. proxy_protocol_v2: true ``` :::note[A passthrough host is matched by SNI, one label deep] On a `tls` listener the host is taken from the ClientHello SNI, so a client that sends no SNI never reaches this host. A wildcard key matches one label on the passthrough router where the HTTP table matches at any depth - `*.example.com` covers `db.example.com` and not `a.b.example.com`. [Upstreams & Routing](./upstreams.mdx) has the matching ladder. ::: Two things about this file that are not obvious from it: - **A `mode: tls` listener needs `proxy.certificates` even when nothing terminates.** Passthrough never reads a certificate, but the TLS acceptor is built before the router knows that, and with the key unset the proxy task **panics** - leaving a live process that has bound neither port and logs nothing pointing at the cause. A directory that is missing or holds no PEM only warns, so the value above is enough for a pure-passthrough proxy. It is also the line to resist deleting. - **"Refused" here means the proxy does not start, not that synapse exits.** Setting `listeners` alongside `address_http`/`address_tls` logs `the proxy will not start` and resolves to an empty listener set, which `start` treats as "proxy disabled". The process loads, stays up and serves nothing - so a unit file watching the exit code sees success. ## 5. Windows agent Windows builds carry no proxy and no BPF, so this is agent-only: NDIS capture, the Windows IDS engine, and logs in `ProgramData`. ```yaml # C:\ProgramData\Gen0Sec\Synapse\config.yaml mode: "agent" network: iface: "auto" capture: enabled: true # On Windows `auto` resolves to NDIS, with a line in the log saying so. mode: auto firewall: mode: "auto" ids: enabled: true rule_paths: - "C:\\ProgramData\\Gen0Sec\\Synapse\\rules\\*.rules" enforce_block: false platform: api_key: "YOUR_API_KEY" threat: enabled: true # Without `url` the database is never downloaded or refreshed: the refresh # worker is registered only for a non-empty url, and the initial load from # `path` then warns and leaves the subsystem empty. url: "https://download.gen0sec.com/v1" path: "C:\\ProgramData\\Gen0Sec\\Synapse" geoip: enabled: true country: url: "https://github.com/gen0sec/geoip-databases/raw/download/ipinfo_lite.mmdb" path: "C:\\ProgramData\\Gen0Sec\\Synapse\\ipinfo_lite.mmdb" logging: level: "info" file_logging_enabled: true log_directory: "C:\\ProgramData\\Gen0Sec\\Synapse\\logs" windows_event_log: enabled: true source: "Synapse" ``` - **`ids.enabled: true` runs a different engine here.** On Windows it starts the `thalamus_ids_windows` worker rather than the XDP one, and it does not consult `ids.capture_mode` - which is why that key, set to `xdp` in the shipped Windows sample, is left out above. - **There is no `daemon:` section here on purpose.** Daemonizing is a Unix path: `daemon.enabled: true` on Windows logs `Daemonization is not supported on this platform, ignoring daemon config` and `daemon.working_directory` is never read. The service is installed with `synapse.exe --service install`, and the service manager starts it with `--daemon` itself. - **Backslashes need doubling** inside double quotes in YAML. `"C:\ProgramData\…"` is a parse error on `\P`; single quotes (`'C:\ProgramData\…'`) would also work. - `logging.windows_event_log` defaults to **off**, so the line above is what puts events in the Windows event log. `logging.etw` is separate and also off by default. ## What these leave out, and why The shipped samples set keys the schema does not accept. They are not harmless: each one lands in the startup warning and nothing else happens. | Left out | Why | | --- | --- | | `logging.bpf_stats` and `logging.tcp_fingerprint` | Not in the schema at 0.8.8. The statistics they look like they control are produced anyway - see [Testing](./testing.mdx) | | `logging.{ssh,latency,tls,http}_fingerprint` and `logging.fingerprint_log`'s per-type siblings | Same: six per-type sections, none of them read. The one that is real is `logging.eventbridge_log`, which `fingerprint_log` is an accepted alias for | | `proxy.acme` | Still in the schema and still ships, but it is not a supported way to get a certificate - see the [ACME warning](./configuration.mdx#proxy-proxy-mode) | | `proxy.content_scanning` | In the schema, off by default, and needs a ClamAV server; it is not part of a starting configuration | | `capture.pfring` | No published package carries the `pfring-capture` build feature, so the block cannot take effect | ## Checking a file before you ship it There is no validate-and-exit flag at 0.8.8. The check is the first start: ```bash sudo /usr/bin/synapse --config /etc/synapse/config.yaml 2>&1 | grep -E 'Unused config options|Deprecated|not provided' ``` `Unused config options in : …` is the line that catches a typo, a key at the wrong nesting level, or a section that was removed from the schema. It names the **subtree** once rather than every key beneath it, so `ml` in that list means the whole `ml:` section was ignored - on a build without the classifier, that is expected rather than a mistake. A key that is spelled correctly but *does nothing* will not appear there, because it parsed. Those are the ones to watch for, and they are called out where they exist: `proxy.allow_connect_method_proxying`, which is forced back to `false` with an error line; a v2 `internal:` list, which is parsed, collision-checked and then dropped; `lb.method` on a terminating host, where it only takes effect on passthrough; and a `weight:` under `round_robin`, as above. ## Next Steps - [Configuration](./configuration.mdx) - every main-config section, with code defaults beside the values the shipped sample sets - [Configuration Interfaces](./configuration-interfaces.mdx) - the CLI flags and environment variables that override any of this, and which wins - [Upstreams & Routing](./upstreams.mdx) - the whole `upstreams.yaml` schema - [Per-Interface Features](./interfaces.mdx) - `network.interfaces` in full - [Testing](./testing.mdx) - confirming a running agent is doing what the file says --- # Configuration Reference Source: https://docs.gen0sec.com/docs/client/configuration # Configuration Reference Synapse reads configuration from three places. They are applied in this order, and each one overwrites what came before — so the **last one wins**: 1. **YAML configuration file** — via `synapse -c /etc/synapse/config.yaml` 2. **Command-line flags** — e.g. `synapse --mode proxy` 3. **Environment variables** — e.g. `export MODE=agent` :::warning[Environment variables beat command-line flags] This is the opposite of what most tools do, and of what earlier revisions of this page claimed. `MODE=agent synapse --mode proxy` runs in **agent** mode: the environment is applied after the flags and assigns unconditionally. Two settings do not follow the rule: - **Log level** — `--log-level` wins over `LOGGING_LEVEL`, which wins over `logging.level`, which wins over `RUST_LOG`. - **`daemon.event_socket` and `daemon.control_socket`** — the YAML value wins over `--event-socket` and `--control-socket`. There is no environment variable for either. ::: :::warning[Environment variable prefix] The `AX_` (and `AX_ARXIGNIS_`) environment-variable prefix is **deprecated**. Use the unprefixed variable names (e.g. `API_KEY`, not `AX_API_KEY`). The old prefixes still work but log a deprecation warning and will be removed in a future release. ::: This page is the schema: what each key means and what it defaults to. The interfaces around it — every command-line flag, every environment variable, and what each package actually contains — are on [Configuration Interfaces](./configuration-interfaces.mdx). ## Operating Modes ```yaml mode: "agent" # or "proxy" ``` - **agent** (default) - Access rules, fingerprinting, IDS, and monitoring only. The reverse proxy (pingora) is disabled. - **proxy** - Full reverse proxy functionality plus everything in agent mode. ### Terminal Mode (TUI) Synapse ships a real-time `ratatui` terminal UI that displays live JA4+ fingerprints, traffic, and threat data. Works on Linux and Windows. ```bash # Embedded: TUI runs in the same process as the agent/proxy synapse --mode agent --terminal # Standalone: attach to a running daemon over its unix socket (Linux only) synapse --terminal ``` ## YAML Configuration ### Basic Example ```yaml mode: "agent" network: iface: "auto" # auto-attach physical uplinks only ip_version: "auto" # processes both IPv4 and IPv6 firewall: mode: "auto" # auto, xdp, nftables, iptables, none platform: api_key: "your-api-key" base_url: "https://api.gen0sec.com/v1" telemetry_sending_enabled: true # Threat intelligence and GeoIP are OFF in this config. They need their own # blocks with a `url` or `path` — see "Platform" below for the working shape. logging: level: "info" ``` ### Feature Toggles Two different defaults matter here. The **code default** is what applies when you omit the key — which is the common case in containers, Helm values and hand-written configs. The **shipped value** is what `config/config.yaml` in the release sets, and it turns several things on that are off by default. Where the two differ, the file wins for anyone who installed from a package. | Feature | YAML path | Environment variable | Default when omitted | As shipped | |---------|-----------|----------------------|----------------------|------------| | **Packet Capture** | `capture.enabled` | `CAPTURE_ENABLED` | `true` | `true` | | **Capture backend** | `capture.mode` | `CAPTURE_MODE` | `auto` | `auto` | | **Inline IDS (Thalamus)** | `ids.enabled` | `IDS_ENABLED` | `false` | `true` | | **IDS blocking** | `ids.enforce_block` | `IDS_ENFORCE_BLOCK` | `false` | `true` | | **ML Fingerprint Classifier** | `ml.fingerprint_classifier.enabled` | `CLASSIFIER_ENABLED` | `false` | `false` | | **Threat Intelligence** | `platform.threat.enabled` | `THREAT_MMDB_ENABLED` | `true` † | `true` | | **GeoIP** | `platform.geoip.enabled` | `GEOIP_ENABLED` | `true` † | `true` | | **Telemetry Sending (API)** | `platform.telemetry_sending_enabled` | `TELEMETRY_SENDING_ENABLED` | `true` † | `true` | | **Unified Fingerprint Log** | `logging.fingerprint_log.enabled` | `LOGGING_EVENTBRIDGE_LOG_ENABLED` | `true` | `true` | | **File Logging** | `logging.file_logging_enabled` | `LOGGING_FILE_ENABLED` | `false` | `true` | | **Syslog** (Linux) | `logging.syslog.enabled` | `LOGGING_SYSLOG_ENABLED` | `false` | `false` | | **ETW** (Windows) | `logging.etw.enabled` | `LOGGING_ETW_ENABLED` | `false` | `false` | | **Windows Event Log** | `logging.windows_event_log.enabled` | `LOGGING_WINDOWS_EVENT_LOG_ENABLED` | `false` | `false` | | **Content Scanning** | `proxy.content_scanning.enabled` | `CONTENT_SCANNING_ENABLED` | `false` | `false` | | **CAPTCHA Protection** | (enabled when keys are set) | `CAPTCHA_SITE_KEY`, `CAPTCHA_SECRET_KEY` | disabled | disabled | | **ACME (Auto TLS)** — *deprecated* | `proxy.acme.enabled` | `ACME_ENABLED` | `false` | `true` | | **Internal Services** | `proxy.internal_services.enabled` | `INTERNAL_SERVICES_ENABLED` | `true` | `true` | | **PROXY Protocol** | `proxy.protocol.enabled` | `PROXY_PROTOCOL_ENABLED` | `false` | `true` | | **HTTP/2 cleartext (h2c)** | `proxy.h2c` | `PROXY_H2C` | `false` | `false` | :::caution[† An absent block is not the same as an absent key] Omitting one key gives you that key's default from the table. Omitting the **whole block** can give you something different, because the parent falls back to its type's own `Default` — and the per-key defaults inside that block never run. The three rows marked † are `true` only when their own block is present: - **`platform.threat.enabled`** and **`platform.geoip.enabled`** need a `threat:` or `geoip:` block to be on. A `platform:` block that sets only `api_key` — which is what most hand-written configs and the [Basic Example](#basic-example) look like — leaves threat intelligence and GeoIP **off**. `enabled: true` on its own is still not enough to load anything. At startup each subsystem is initialised only if the flag is set **and** it has somewhere to fetch from: a non-empty `threat.url` or a `threat.path`, and for GeoIP a `url` or `path` under `country`, `asn` or `city`. Neither is derived from `platform.base_url`, so a block holding nothing but `enabled: true` initialises nothing and logs nothing you would recognise as a failure. The [Platform](#platform) section below has the shape that works. - **`platform.telemetry_sending_enabled`** is `true` whenever a `platform:` block exists at all, and `false` if `platform:` is absent entirely. `proxy.captcha` fails the other way round: with no `proxy:` block you get provider `hcaptcha` and TTLs of 7200/300, but a `proxy:` block that merely leaves out `captcha:` gives you an empty provider and TTLs of `0`. Set the keys you care about explicitly rather than relying on a parent block's fallback. ::: :::info[`telemetry_sending_enabled` (renamed)] The platform log/telemetry toggle is now `platform.telemetry_sending_enabled` (formerly `log_sending_enabled`), and its environment override is `TELEMETRY_SENDING_ENABLED`. The old `LOG_SENDING_ENABLED` is still read as a fallback, and the YAML alias `log_sending_enabled` still parses. ::: ## Core Sections ### Network ```yaml network: # "auto" attaches XDP ONLY to physical UP uplink interfaces. Loopback and # CNI/virtual/tunnel devices (veth*, lxc*, cilium*, docker*, bridges, # vxlan/tunnels, etc.) are excluded, so it never clobbers a CNI datapath # (e.g. Cilium) on Kubernetes nodes. bond*/VLAN uplinks (eth0.100) are kept. iface: "auto" # Explicit list overrides `iface`; entries are attached as-is (bypasses the # "auto" filtering above — use this to opt in to a virtual interface). ifaces: [] # "auto" (default) processes both IPv4 and IPv6. ip_version: "auto" ``` To enable features per interface rather than everywhere — the firewall on one link, the IDS on another, nothing at all on the management port — use `network.interfaces`, which replaces `iface` and `ifaces`. See [Per-Interface Features](./interfaces.mdx). ### Packet Capture Controls the backend used for JA4+ fingerprint packet capture. ```yaml capture: enabled: true # Backend: # auto — Auto-detect: XDP > af_packet_recv > pfring (default) # xdp — eBPF/XDP BPF map polling (highest performance) # af_packet — AF_PACKET TPACKET_V3 mmap ring buffer (zero-copy) # af_packet_recv — AF_PACKET plain recvfrom() (works on all kernels, # including 4.19 where TPACKET_V3 poll() is broken) # pfring — PF_RING zero-copy (requires pf_ring.ko + libpfring, # build with --features pfring-capture, which no shipped # package enables) # ndis — NDIS lightweight filter (Windows only) # af_xdp — AF_XDP socket (Windows only) mode: auto # interface: auto # "auto" = all interfaces; or a specific iface snaplen: 65535 # max bytes per packet (af_packet/af_packet_recv/pfring) filter: "tcp" # tcpdump-syntax BPF filter (af_packet, pfring) af_packet: # only used when mode: af_packet block_size: 4194304 # ring block size (power of 2, default 4MB) block_count: 4 # number of blocks; total ring = block_size * block_count block_timeout_ms: 100 promiscuous: true pfring: # only used when mode: pfring zero_copy: true # cluster_id: 1 # multi-queue NIC load balancing (optional) ``` Environment overrides: `CAPTURE_MODE`, `CAPTURE_SNAPLEN`, `CAPTURE_FILTER`, `PFRING_ZERO_COPY`, `PFRING_CLUSTER_ID`. **Collector hygiene.** `capture.dedup_ttl_secs` (default 600) suppresses re-processing a source already seen, so an established ban is not re-emitted on every poll. `capture.collector_stale_secs` (default 900) evicts idle entries from the fingerprint map. Keep the stale age comfortably above the dedup window, or live flows get dropped from the map while still inside their suppression period. ### Firewall ```yaml firewall: # auto: XDP > nftables > iptables > none # xdp / nftables / iptables: force a backend # none: disable kernel firewall, userland enforcement only mode: "auto" ``` **Blocklist capacity.** `firewall.max_ip_filter_entries_v4` and `_v6` size the kernel maps, 50,000 each by default. :::caution[Raising it needs the pins gone] The maps are pinned and reused across restarts, and a pinned map keeps the size it was created with. A raised value takes effect only once the pins are removed — `synapse --force-stop`, or a reboot. A plain restart silently keeps the old size. ::: ### Inline IDS (Thalamus) Synapse embeds the **Thalamus** IDS engine for inline, signature-based detection on captured traffic. ```yaml ids: enabled: true # shipped value; the code default is false capture_mode: xdp # xdp | afpacketrecv (alias: afpacket) rule_paths: - "/etc/synapse/rules/*.rules" address_vars: {} # Suricata-style address variable definitions port_vars: {} # Suricata-style port variable definitions snaplen: 512 poll_timeout_ms: 100 flow_timeout_secs: 120 max_flows: 200000 cleanup_interval_secs: 30 stats_log_interval_secs: 5 enforce_block: true # ban the offending SOURCE via the firewall # (false = alert only). The per-flow kernel drop is # a separate key, enforce_edge_drop. # Shipped value; the code default is false, and the # shipped comment itself recommends false until the # rules are tuned. ``` :::note[Rule loading] Thalamus loads **one rule per line** — there is no `\` line-continuation. A rule split across multiple lines silently loads as zero rules. Validate offline before deploying. ::: ### ML Classifier Optional ONNX-based JA4+ fingerprint classification (ships with the `-ml` package variant). ```yaml ml: fingerprint_classifier: enabled: false model_path: "" # path to the ONNX model file cache_size: 10000 # max cached predictions (0 = no cache) threshold: 0.5 # malicious probability threshold (0.0–1.0). # Shipped value; the code default is 0.52. # Platform-managed models. `enabled: true` is enough — the model is # downloaded and the remaining paths default. traffic_classifier: enabled: false flow_classifier: enabled: false penalty: enabled: false jepa_embedder: enabled: false ``` :::note[The top-level keys are deprecated] `classifier`, `traffic_classifier`, `flow_classifier` and `penalty` were once top-level. They are still accepted, but nest them under `ml:` instead — and note that the fingerprint model is now `ml.fingerprint_classifier`, not `classifier`. ::: ### Platform ```yaml platform: api_key: "" base_url: "https://api.gen0sec.com/v1" # Send access logs / telemetry to the platform (formerly log_sending_enabled) telemetry_sending_enabled: true # WARNING: may capture PII, tokens, or API responses in logs include_response_body: false max_body_size: 1048576 # bytes; do not raise above the Basic plan maximum # Threat intelligence MMDB threat: enabled: true url: "https://download.gen0sec.com/v1" path: "/var/lib/synapse" headers: null refresh_secs: 300 # 5 minutes # GeoIP MMDB — ships with the IPinfo Lite bundle (single combined DB with # both country + ASN). Synapse auto-detects the IPinfo schema, so the DB # configured under `country` serves both country and ASN lookups; `asn` # and `city` are unused and their refresh workers don't run. geoip: enabled: true country: url: "https://github.com/gen0sec/geoip-databases/raw/download/ipinfo_lite.mmdb" path: "/var/lib/synapse/ipinfo_lite.mmdb" headers: null asn: { url: "", path: "", headers: null } # unused with IPinfo Lite city: { url: "", path: "", headers: null } # unused with IPinfo Lite refresh_secs: 28800 # 8 hours ``` :::info[GeoIP changed to IPinfo Lite] Synapse now defaults to a single combined **IPinfo Lite** database (`ipinfo_lite.mmdb`) instead of the split MaxMind GeoLite2 country/ASN/city files. To switch back to GeoLite2, point `country`/`asn` at the respective `GeoLite2-*.mmdb` URLs. ::: ### Platform integrations Four `platform.*` subsections govern what the agent exchanges with the platform beyond threat intelligence. `ban_ledger`, `fleet_bans` and `service_graph` are **off at every level** — omitting the key and writing the block but omitting `enabled` both leave them off; the shipped `config.yaml` turns the first two on. `platform.identity` is the exception: it is the same type as `platform.threat`, so writing an `identity:` block turns it on even if you never say `enabled: true`, and the † note above applies to it. ```yaml platform: # Upload this agent's bans so the fleet can corroborate them. # Shipped as true; the code default is false. ban_ledger: enabled: true interval_secs: 60 # Download the corroborated result. Shipped as true; the code default is # false. The refresh is a cheap version check and the artifact is only # fetched when the version changes. # `enforce: true` blocks what it contains instead of only reporting it. fleet_bans: enabled: true refresh_secs: 60 advisory: true advisory_dedup_secs: 300 enforce: false # East-west workload identity. Off by default: it needs a cluster-side # producer publishing the identity MMDB before there is anything to fetch. identity: enabled: false url: "" path: "/var/lib/synapse" # Report the observed service graph. Off by default. service_graph: enabled: false interval_secs: 300 ``` **`ban_ledger` and `fleet_bans` are the two halves of one loop** (see [Fleet Bans](../features/fleet-bans.mdx)) — you upload what this agent blocked, and pull back what the fleet corroborated. As shipped they share a 60-second cadence (each defaults to 300 when left out), so neither leg is the slow one. Turning off the upload while leaving the download on is valid; you consume the fleet's conclusions without contributing. **`fleet_bans.enforce` is off by default.** Turned on (it needs `fleet_bans.enabled`), the agent installs each fleet ban in the kernel IP filter (Linux, unless `firewall.mode: none`) and its proxy refuses fleet-banned clients; see [Enforcing the set](../features/fleet-bans.mdx#enforcing-the-set) for what is and is not enforced. Turn it on per agent once advisory mode has shown the set is safe for its traffic. It is read at startup, so turning it off takes a restart, which lifts what the agent installed. **`identity` and `service_graph` both ship disabled** because each needs something on the other side. Identity needs a producer publishing the MMDB; enabling it without one just produces periodic failures. See [Service Graph](../features/service-graph.mdx). ### Logging ```yaml logging: level: "info" # error, warn, info, debug, trace file_logging_enabled: true # shipped value; the code default is false log_directory: "/var/log/synapse" # Windows: C:\ProgramData\Gen0Sec\Synapse\logs max_log_size: 104857600 # 100MB log_file_count: 10 # rotated files are gzip-compressed # Syslog (Linux/Unix only; silently ignored on Windows). # Requires file_logging_enabled: true — the syslog appender is only built as part # of file logging, so syslog alone emits nothing. syslog: enabled: false facility: "daemon" identifier: "synapse" levels: # per-log-type priority error: "err" app: "info" access: "info" # proxy mode only # Event Tracing for Windows — Windows equivalent of syslog. # Events emit to ETW provider "Gen0Sec.Synapse". etw: enabled: false # Classic Windows Event Log (Event Viewer → Windows Logs → Application) windows_event_log: enabled: false source: "Synapse" # Unified per-connection fingerprint/BPF event log (agent mode). fingerprint_log: enabled: true # file: /var/log/synapse/eventbridge.log # default: {log_directory}/eventbridge.log ``` :::warning[Per-type fingerprint logging has been removed] The six per-type sections — `bpf_stats`, `tcp_fingerprint`, `ssh_fingerprint`, `latency_fingerprint`, `tls_fingerprint` and `http_fingerprint` — are no longer in the configuration schema at all. The shipped `config.yaml` still carries them, but the parser does not read them: they appear in the `Unused config options` warning at startup and change nothing. `fingerprint_log` (a.k.a. `eventbridge_log`) replaces all six, emitting every fingerprint and BPF event through one unified JSON log. ::: When `file_logging_enabled` is true, Synapse writes (and rotates/compresses) `error.log`, `app.log`, and — in proxy mode — `access.log` (JSON). ### Daemon / Service Linux runs as a Unix daemon; Windows registers with the Service Control Manager (`synapse --service install` / `sc start synapse`; `sc control synapse paramchange` triggers a config reload). ```yaml daemon: enabled: true # shipped value; the code default is false pid_file: "/var/run/synapse.pid" # Linux only working_directory: "/var/lib/synapse" # shipped value; the code default is "/" on # Linux, C:\ProgramData\Gen0Sec\Synapse on Windows user: root # Linux only group: root # Linux only chown_pid_file: true # Linux only # Sockets used by standalone terminal clients # event_socket: "/var/run/synapse-events.sock" # Windows: TCP, e.g. 127.0.0.1:19199 # control_socket: "/var/run/synapse-control.sock" ``` ### Proxy (proxy mode) :::caution[`proxy.redis.prefix` in YAML is overwritten] The key prefix is reset to `g0s:synapse` on every start unless you pass `--redis-prefix`, so a value set only in `config.yaml` does not survive. Pass the flag, or set `REDIS_PREFIX`, which is applied after it and overrides both. ::: :::warning[ACME is deprecated] The embedded ACME client is no longer a supported way to obtain certificates, and it is not documented as a feature anywhere else. It still ships, and the `config.yaml` we ship still sets `enabled: true` even though the code default is `false`, so the keys are listed here for anyone who has it running — but do not build on it. Provide your own certificate and key, or pass the connection through to a backend that holds them. On Kubernetes, use cert-manager — see [Kubernetes Ingress](./kubernetes-ingress.mdx#tls-with-cert-manager). ::: ```yaml proxy: address_http: "0.0.0.0:80" address_tls: "0.0.0.0:443" certificates: "/etc/synapse/certs" # cert files: {NAME}.crt, {NAME}.key tls_grade: "medium" # high, medium, unsafe default_certificate: "default" # fallback cert (file stem) # Enable HTTP/2 cleartext on the plaintext listener (gRPC over plaintext, # --http2-prior-knowledge clients). Prior knowledge only: the first 24 bytes are # peeked and anything that is not the h2 preface is served as HTTP/1.1, so # turning this on does not disturb HTTP/1 clients. `Upgrade: h2c` is a different # mechanism and is not implemented. Forced off on the TLS listener, where ALPN # decides. h2c: false # Unsupported. CONNECT always answers 405, whatever this is set to — setting it # to true logs an error and forces it back to false at startup. allow_connect_method_proxying: false # Max requests per connection before closing (frees per-connection memory). # null = no limit (default). keepalive_request_limit: null redis: url: "redis://127.0.0.1:6379/0" prefix: "g0s:synapse" # ssl: # ca_cert_path: "/path/to/ca.crt" # client_cert_path: "/path/to/client.crt" # optional, mutual TLS # client_key_path: "/path/to/client.key" # optional, mutual TLS # insecure: false upstream: conf: "/etc/synapse/upstreams.yaml" healthcheck: method: "HEAD" # HEAD, GET, POST interval: 2 # seconds # Shipped value; the code default is false. Keep it on ONLY when something in # front of Synapse actually sends a PROXY header — otherwise the first bytes of # every connection fail to parse. protocol: enabled: true timeout_ms: 1000 captcha: site_key: null secret_key: null jwt_secret: null # openssl rand -base64 48 provider: "hcaptcha" # hcaptcha, recaptcha, turnstile, prosopo token_ttl: 7200 cache_ttl: 300 acme: enabled: true # shipped value; the code default is false port: 9180 # binds to 127.0.0.1 only email: null storage_type: null # "file" or "redis" (defaults by whether redis is set) storage_path: "/var/lib/synapse/acme" development: false # true = Let's Encrypt staging redis_url: null # overrides global redis.url for ACME storage # Enabling the scanner scans nothing on its own — a WAF rule with action # `content_scanning` is what selects the traffic. content_scanning: enabled: false clamav_server: "localhost:3310" max_file_size: 10485760 # 10MB # A selected request whose body cannot be inspected: block (413) or pass. unscannable: block # Ceiling on body memory held across all in-flight scans. Over it, scans # fall back to streaming rather than refusing — load shedding, not policy. scan_memory_budget: 268435456 # Reuse a verdict for an identical body. 0 disables. Failures never cached. verdict_cache_ttl_secs: 300 verdict_cache_entries: 8192 internal_services: enabled: true # auto-disabled in agent mode port: 9180 bind_ip: "127.0.0.1" # CIDRs whose L4 peer is a trusted upstream reverse proxy. Empty by default. trusted_proxies: [] ``` ### Client address behind a proxy Two things key off the client address: the [WAF](../features/waf.mdx) rate-limit bucket, and `ip.src` in the wirefilter context. Both default to the **L4 socket peer** — which, when Synapse sits behind a load balancer, is the load balancer. That breaks quietly rather than loudly: - An LB that **rotates its source addresses** scatters one client across many rate-limit buckets, so no bucket reaches the threshold and the rule never fires. - An LB that **collapses many clients onto a few addresses** puts unrelated users in one bucket, where they rate-limit each other. `proxy.trusted_proxies` fixes both. List the CIDRs your proxies connect from, and for connections arriving from those ranges Synapse takes the client address from the first `X-Forwarded-For` token instead of the socket peer: ```yaml proxy: trusted_proxies: - "35.191.0.0/16" # GCP LB health checks and GFE - "130.211.0.0/22" # GCP LB health checks - "10.10.0.0/20" # an internal NLB range ``` :::note[Why a list, and not just trusting the header] `X-Forwarded-For` is client-supplied and trivially forged. A client connecting **directly** is not inside any listed range, so its header is ignored and the socket peer wins. Trust is granted to the hop, never to the header — which is why an empty list is the safe default and why widening these ranges beyond your actual proxies hands clients a way to spoof their own address. ::: PROXY protocol v2 (`proxy.protocol.enabled`) solves the same problem for L4 proxies that speak it. Use `trusted_proxies` when the upstream does not — a GCP Global External Application Load Balancer, for instance. ## Environment Variables ```bash # Application mode & runtime export MODE="agent" # or "proxy" export MULTI_THREAD="false" # enable multi-threaded runtime export WORKER_THREADS="4" # worker threads when MULTI_THREAD=true # Redis export REDIS_URL="redis://127.0.0.1:6379/0" export REDIS_PREFIX="g0s:synapse" export REDIS_SSL_CA_CERT_PATH="..." export REDIS_SSL_CLIENT_CERT_PATH="..." export REDIS_SSL_CLIENT_KEY_PATH="..." export REDIS_SSL_INSECURE="false" # Network & capture export NETWORK_IFACE="auto" export NETWORK_IFACES="eth0,eth1" export NETWORK_IP_VERSION="both" # ipv4, ipv6, both - "auto" is refused here export CAPTURE_MODE="auto" # xdp, af_packet, af_packet_recv, pfring, ndis, af_xdp export CAPTURE_SNAPLEN="65535" export CAPTURE_FILTER="tcp" export PFRING_ZERO_COPY="true" export PFRING_CLUSTER_ID="1" export FIREWALL_MODE="auto" # Gen0Sec Platform export API_KEY="your-api-key" export BASE_URL="https://api.gen0sec.com/v1" export TELEMETRY_SENDING_ENABLED="true" # platform.telemetry_sending_enabled # (LOG_SENDING_ENABLED still works) # Logging export LOGGING_LEVEL="info" export LOGGING_FILE_ENABLED="true" export LOGGING_DIRECTORY="/var/log/synapse" export LOGGING_MAX_FILE_SIZE="104857600" export LOGGING_FILE_COUNT="10" export LOGGING_SYSLOG_ENABLED="false" # Linux export LOGGING_SYSLOG_FACILITY="daemon" export LOGGING_SYSLOG_IDENTIFIER="synapse" export LOGGING_ETW_ENABLED="false" # Windows export LOGGING_WINDOWS_EVENT_LOG_ENABLED="false" # Windows export LOGGING_WINDOWS_EVENT_LOG_SOURCE="Synapse" # Content scanning export CONTENT_SCANNING_ENABLED="true" export CLAMAV_SERVER="localhost:3310" export CONTENT_MAX_FILE_SIZE="10485760" # `scan_expression`, `scan_content_types` and `skip_extensions` are gone — they # are not environment variables and no longer YAML keys either. Selecting which # traffic to scan is now a WAF rule with action `content_scanning`. # CAPTCHA export CAPTCHA_SITE_KEY="your-site-key" export CAPTCHA_SECRET_KEY="your-secret-key" export CAPTCHA_JWT_SECRET="your-jwt-secret" export CAPTCHA_PROVIDER="hcaptcha" # hcaptcha, recaptcha, turnstile, prosopo export CAPTCHA_TOKEN_TTL="7200" export CAPTCHA_CACHE_TTL="300" # Internal services & PROXY protocol export INTERNAL_SERVICES_ENABLED="true" export INTERNAL_SERVICES_PORT="9180" export INTERNAL_SERVICES_BIND_IP="127.0.0.1" export PROXY_PROTOCOL_ENABLED="true" export PROXY_PROTOCOL_TIMEOUT="1000" # Daemon export DAEMON_ENABLED="true" export DAEMON_PID_FILE="/var/run/synapse.pid" export DAEMON_WORKING_DIRECTORY="/var/lib/synapse" export DAEMON_USER="root" export DAEMON_GROUP="root" export DAEMON_CHOWN_PID_FILE="true" ``` ## Upstreams Configuration Routing lives in its own file, `upstreams.yaml`, parsed and reloaded independently of this one. It declares which hosts Synapse serves, how TLS is handled for each, and which backends a path forwards to. The whole schema — every key, every default, and what each kind of bad file does — is on its own page: **[Upstreams & Routing](./upstreams.mdx)**. What `config.yaml` contributes is the pointer to that file and the settings that apply to whatever backends it finds: | YAML path | Environment variable | Default when omitted | As shipped | What it does | |-----------|----------------------|----------------------|------------|--------------| | `proxy.upstream.conf` | `UPSTREAM_CONF` | `""` | `/etc/synapse/upstreams.yaml` | Path to the upstreams file. Empty means no routing table is loaded | | `proxy.upstream.healthcheck.method` | `UPSTREAM_HEALTHCHECK_METHOD` | `HEAD` | `HEAD` | HTTP method the backend probe uses | | `proxy.upstream.healthcheck.interval` | `UPSTREAM_HEALTHCHECK_INTERVAL` | `2` | `2` | Seconds between probes | | `proxy.upstream.dns_cache_ttl_secs` | — | `60` | `60` | Seconds a resolved backend hostname is cached | The first three also take a flag: `--upstream-conf`, `--upstream-healthcheck-method` and `--upstream-healthcheck-interval`. `dns_cache_ttl_secs` is YAML-only — there is no flag and no environment variable for it. :::note[The health-check interval is global, not per route] `healthcheck.method` and `.interval` are the probe settings for every terminating backend. A v2 route's own `health_check:` block chooses the probe **type** and path; it cannot set an interval, and the parser rejects one. Only a passthrough host carries its own interval. See [Health checks](./upstreams.mdx#health-checks). ::: When a backend is named by hostname, the resolved address is cached for `dns_cache_ttl_secs` rather than resolved on every connection. On Kubernetes that matters more than it sounds: without it, every upstream connection makes a blocking call to the cluster resolver, which becomes the floor on peer-select latency. Raise it where Service ClusterIPs are stable; lower it if you rely on DNS-level failover. Unset or `0` both mean the built-in 60 seconds. ## SIGHUP Configuration Reload Synapse supports runtime configuration reload via `SIGHUP` signal (Windows: `sc control synapse paramchange`): ```bash kill -HUP $(cat /var/run/synapse.pid) ``` A `SIGHUP` does not re-read `config.yaml` in one place. The signal is broadcast, and every subscriber that acts on it is listed here: - the daemon applies `logging.level` and the `platform` log-sender settings, and re-opens the GeoIP databases from disk; - the upstreams watcher re-reads the upstreams file; - the rules worker re-reads its rules file, or re-fetches the rules from the platform when an API key is set; - the **XDP** IDS worker re-reads `config.yaml` and hot-applies `ids.enforce_block` and the engine fields. It is the only one of the IDS workers that does: with `ids.capture_mode: afpacketrecv`, or when `xdp` is configured and XDP turns out to be unavailable — the agent falls back to the AF_PACKET worker and logs one line about it — nothing in `ids.*` reloads at all. On Windows the NDIS worker does not reload either; - on an `-ml` package, four model reconcilers re-read it and apply each model's `enabled` and `pool_size`. Between them they cover the table below: no other key is applied in full. The restart table that follows names the ones operators most often expect to be live, with the one key that is applied in part; it is not the full remainder. ### Hot-reloadable Settings | Setting | Config Path | Notes | |---------|-------------|-------| | Log level | `logging.level` | Switches log verbosity instantly | | Telemetry sending | `platform.telemetry_sending_enabled` | Enable/disable API log sending | | API key | `platform.api_key` | Update platform credentials | | Base URL | `platform.base_url` | Change API endpoint | | Include response body | `platform.include_response_body` | Toggle body capture in access logs | | Max body size | `platform.max_body_size` | Adjust body size limit for logs | | Upstreams | `proxy.upstream.conf` | Re-reads upstreams YAML file | | Security rules | Local file or API | Re-fetches access rules, WAF rules, and IDS rules | | GeoIP databases | `platform.geoip.*` | Re-opens the files from disk, at the paths the process started with. Nothing is downloaded, and a changed path is not picked up | | IDS rules and tuning | `ids.rule_paths`, `ids.managed_rules_path`, `ids.address_vars`, `ids.port_vars`, `ids.flow_timeout_secs`, `ids.max_flows` | XDP IDS worker only. Rebuilds and swaps the signature engine; in-flight flow state is reset. A managed-rules bundle rewritten in place is picked up too, by content. With `ids.post_tls.mode: offload` the node-wide L7 consumer runs its own engine beside this one and keeps the ruleset it started with, so a reload leaves the two halves on different rules | | IDS enforcement | `ids.enforce_block` | XDP IDS worker only, as above — there a `SIGHUP` can start or stop dropping traffic. On the AF_PACKET worker it needs a restart | | ML models, enablement and pool | `ml.penalty.*`, `ml.traffic_classifier.*`, `ml.flow_classifier.*`, `ml.jepa_embedder.*` — each one's `enabled` and `pool_size`, and the penalty model's `calibration_a` / `calibration_b` | `-ml` packages. Enabling a model loads it once its file is on disk; until then enabling only starts the fetch, and a later reload loads it. Disabling unloads it | ### Settings That Require Restart | Setting | Config Path | Reason | |---------|-------------|--------| | Listen addresses | `proxy.address_http`, `proxy.address_tls` | Port bindings set at startup | | Network interface | `network.iface`, `network.ifaces` | XDP attached at startup | | Per-interface features | `network.interfaces` | Links resolved and XDP attached at startup | | Firewall mode | `firewall.mode` | BPF programs loaded at startup | | Capture backend | `capture.*` | Capture ring/sockets set up at startup | | Inline IDS capture | `ids.enabled`, `ids.capture_mode`, `ids.snaplen`, and the `ids.*` keys not named in the hot-reload table above | Capture and engine set up at startup. On a host running the AF_PACKET IDS worker this row is the whole of `ids.*`. `ids.bypass.*` is the half-case: the poll loop follows a new `enabled` at once, but the kernel-side depth caps are written only at startup, so give it a restart | | Fingerprint classifier | `ml.fingerprint_classifier.*` | ONNX model loaded at startup. It is the one model with no reconciler; the other four hot-reload, above | | Runtime threads | `multi_thread`, `worker_threads` | Tokio runtime created at startup | | Daemon settings | `daemon.*` | Process daemonization is one-time | | Redis connection | `proxy.redis.*` | Connection pool created at startup | | CAPTCHA config | `proxy.captcha.*` | Provider initialized once | | Content scanning | `proxy.content_scanning.*` | Scanner initialized once | | Certificate paths | `proxy.certificates` | Loaded at startup | | ACME settings | `proxy.acme.*` | ACME manager created at startup | | Internal services | `proxy.internal_services.*` | Server bound at startup | | Fingerprint log | `logging.fingerprint_log.*` | Appender built at startup | | File / syslog / Windows logging | `logging.file_logging_enabled`, `logging.syslog.*`, `logging.etw.*`, `logging.windows_event_log.*` | Appenders built at startup | | IP version | `network.ip_version` | BPF filter compiled at startup | | PROXY protocol | `proxy.protocol.*` | Listener configured at startup | ## Internal Services Endpoints When `proxy.internal_services.enabled` is `true`, the following endpoints are available: | Endpoint | Method | Purpose | |----------|--------|---------| | `/health` | GET | Health check | | `/cgi-bin/captcha/verify` | POST | CAPTCHA verification | | `/.well-known/acme-challenge/*` | GET | ACME HTTP-01 challenges | | `/cert/expiration` | GET | Check all certificate expiration status | | `/cert/expiration/:domain` | GET | Check specific certificate status | | `/cert/renew/:domain` | POST | Manually trigger certificate renewal | ## Next Steps - [Configuration Examples](./config-examples) - Five complete files to copy: agent, per-interface features, proxy with TLS termination, database passthrough, Windows - [Daemon Mode](./daemon-mode) - Run as a background service - [JA4+ Fingerprinting](./ja4-plus) - Complete fingerprinting suite - [Testing Guide](./testing) - Validate your configuration --- # Configuration Interfaces Source: https://docs.gen0sec.com/docs/client/configuration-interfaces # Configuration Interfaces Four things configure synapse, and they are not interchangeable: 1. **`config.yaml`** - the full schema, documented on [Configuration](./configuration.mdx). 2. **Command-line flags** - a subset of the schema, plus a handful of one-shot commands. 3. **Environment variables** - a different subset, plus variables that exist nowhere in the YAML. 4. **The package you installed** - not every build contains every subsystem. A key for a subsystem your package does not carry still parses, and then does nothing. This page covers how the first three combine, lists all of them, and ends with what each package actually contains. :::info[Verified against 0.8.8] Every flag, variable, default and behaviour here was read out of the v0.8.8 source. The counts are exact for that release: 84 flags, of which 63 write a configuration key; 129 environment variable names in the merge path, of which 116 are canonical and 13 are fallbacks; and 51 further variables read at runtime outside the merge. The four per-signal rows under [OpenTelemetry](#opentelemetry) stand for three names each. ::: ## Which source wins Configuration is assembled in three steps, in this order: 1. the YAML file is parsed (`--config`, or built-in defaults if no file is given); 2. command-line flags are merged over it; 3. environment variables are applied over that. Step 3 assigns unconditionally. So the order of precedence is: **environment variable** → **command-line flag** → **`config.yaml`** → built-in default. :::warning[The environment beats the command line] This is the opposite of the usual convention, and the opposite of what synapse's own in-repo `docs/CONFIGURATION.md` says. If a flag appears to be ignored, look for the environment variable that writes the same key - a stray `FIREWALL_MODE` in a unit file or a container spec silently wins over `--firewall-mode` on the command line. There is no diagnostic for this. Nothing logs that an environment variable replaced a flag's value. ::: Two settings do not follow that order. ### Exception 1: the log level The log level is resolved separately, and the flag wins: **`--log-level`** → **`logging.level`** (YAML, or `LOGGING_LEVEL` over it) → **`RUST_LOG`** → `info`. `RUST_LOG` is only consulted when neither the flag nor `logging.level` is set, and only a bare level is accepted: `error`, `warn`, `info`, `debug`, `trace`, `off`. A per-module directive such as `RUST_LOG=synapse=debug,hyper=warn` is not passed through - it does not match one of the six names, so the level degrades to `info` with no warning. Synapse then overwrites `RUST_LOG` in its own environment with the level it resolved, so anything reading the variable later sees the resolved value rather than what you set. ### Exception 2: the event and control sockets `daemon.event_socket` and `daemon.control_socket` in the YAML win over `--event-socket` and `--control-socket` **in the process that serves those sockets** - the daemon. The flags only apply there when the YAML leaves the key unset, and neither key has an environment variable. The client side does not consult the YAML at all. `synapse --terminal` connects to whatever `--event-socket` says, default included, so moving the socket with `daemon.event_socket` alone leaves the terminal looking at the old path. Pass `--event-socket` to the terminal as well, or keep the socket where the default puts it. ### Three flags overwrite the YAML even when you do not pass them Most flags are merged only when you actually pass them. These three are assigned unconditionally, so the flag's own default lands on top of whatever the YAML said: | Flag | Key it overwrites | Value written when the flag is absent | | --- | --- | --- | | `--arxignis-include-response-body` | `platform.include_response_body` | `false` | | `--arxignis-max-body-size` | `platform.max_body_size` | `1048576` | | `--daemon-chown-pid-file` | `daemon.chown_pid_file` | `true` | Setting any of these three in `config.yaml` has no effect. Use the flag or, where one exists, the environment variable. ### `proxy.redis.prefix` cannot be set from the YAML Seven flags are merged only when their value differs from a sentinel, which is how a flag with a built-in default avoids clobbering the YAML. For six of them the sentinel is the flag's own default and the mechanism works. For `--redis-prefix` the two disagree: the flag defaults to `g0s:synapse` and the comparison is against `ax:synapse`, so the test always passes and `proxy.redis.prefix` from the YAML is always replaced by `g0s:synapse`. To use a different prefix, pass `--redis-prefix` or set `REDIS_PREFIX`. Both work. ### What a bad value does An environment variable that cannot be parsed into the key's type is handled in one of four ways. All four are silent for all but three variables: | Behaviour | What happens | Variables | | --- | --- | --- | | **used as given** | the string is stored unvalidated | 55 | | **built-in default** | the key is reset to the code default, discarding the YAML value | 35 | | **ignored** | the assignment is skipped and the YAML value survives | 20 | | **cleared** | the key is set to "unset", discarding the YAML value | 6 | `IDS_MAX_FLOWS=lots` resets `ids.max_flows` to the code default even if the YAML set it; `CAPTURE_ENABLED=yes` is ignored entirely, because only `true` and `false` parse as a boolean; `GEOIP_COUNTRY_HEADERS=junk` clears `platform.geoip.country.headers`, so a YAML header map is dropped rather than kept. The **Unparseable value** column in the tables below says which applies to each variable. Only three variables log anything when their value does not parse, and the tables mark them "and it warns": `CAPTURE_MODE` and `IDS_CAPTURE_MODE`, which warn and then use the built-in default, and `NETWORK_IP_VERSION`, which warns and then changes nothing. Its message claims it is "using default 'both'", but the code makes no assignment, so the YAML value actually survives. :::warning[A misspelt `FIREWALL_MODE` silently disables what you configured] `firewall.mode` is one of the 35. Any value that is not `auto`, `xdp`, `nftables`, `iptables` or `none` resets the key to `auto`, with nothing logged - so `FIREWALL_MODE=xpd` in a unit file replaces a deliberate `xdp` or `none` in your `config.yaml` with mode selection you did not ask for. The same shape applies to `capture.mode` and `ids.capture_mode`, which at least warn. ::: ## Command-line flags `synapse --help` lists these. 84 flags exist; 63 of them write a configuration key. ### Flags that write a configuration key The key is the YAML path on [Configuration](./configuration.mdx), which is where its meaning is documented. "Overrides the YAML" says when the merge applies the flag. | Flag | Value | Flag default | YAML key it writes | Overrides the YAML | | --- | --- | --- | --- | --- | | `--redis-url` | string | `redis://127.0.0.1/0` | `proxy.redis.url` | unless it is `redis://127.0.0.1/0` | | `--redis-prefix` | string | `g0s:synapse` | `proxy.redis.prefix` | unless it is `ax:synapse` | | `--iface` | string | `auto` | `network.iface` | unless it is `auto` | | `--ifaces` | list | — | `network.ifaces` | when non-empty | | `--arxignis-api-key` | string | — | `platform.api_key` | when passed | | `--arxignis-base-url` | string | `https://api.gen0sec.com/v1` | `platform.base_url` | unless it is `https://api.gen0sec.com/v1` | | `--arxignis-telemetry-sending-enabled`
also accepted: `--arxignis-log-sending-enabled` | `true`/`false` | — | `platform.telemetry_sending_enabled` | when passed | | `--arxignis-include-response-body` | switch | `false` | `platform.include_response_body` | **always, even when absent** | | `--arxignis-max-body-size` | integer | `1048576` | `platform.max_body_size` | **always, even when absent** | | `--firewall-mode` | string | — | `firewall.mode` | when passed | | `--captcha-site-key` | string | — | `proxy.captcha.site_key` | when passed | | `--captcha-secret-key` | string | — | `proxy.captcha.secret_key` | when passed | | `--captcha-jwt-secret` | string | — | `proxy.captcha.jwt_secret` | when passed | | `--captcha-provider` | `hcaptcha`/`recaptcha`/`turnstile`/`prosopo` | — | `proxy.captcha.provider` | when passed | | `--proxy-protocol-enabled` | switch | `false` | `proxy.protocol.enabled` | when given | | `--proxy-protocol-timeout` | integer | `1000` | `proxy.protocol.timeout_ms` | unless it is `1000` | | `-d`, `--daemon` | switch | `false` | `daemon.enabled` → `true` | when given | | `--daemon-pid-file` | string | `/var/run/synapse.pid` (Linux), empty (Windows) | `daemon.pid_file` | unless it is the built-in default | | `--daemon-working-dir` | string | `/` (Linux), `C:\ProgramData\Gen0Sec\Synapse` (Windows) | `daemon.working_directory` | unless it is the built-in default | | `--daemon-user` | string | — | `daemon.user` | when passed | | `--daemon-group` | string | — | `daemon.group` | when passed | | `--daemon-chown-pid-file` | switch | `true` | `daemon.chown_pid_file` | **always, even when absent** | | `--mode` | string | — | `mode` | when passed | | `--network-ip-version` | string | — | `network.ip_version` | when passed | | `--threat-mmdb-url` | string | — | `platform.threat.url` | when passed | | `--threat-mmdb-path` | string | — | `platform.threat.path` | when passed | | `--threat-mmdb-refresh-secs` | integer | — | `platform.threat.refresh_secs` | when passed | | `--file-logging-enabled` | `true`/`false` | — | `logging.file_logging_enabled` | when passed | | `--log-directory` | string | — | `logging.log_directory` | when passed | | `--max-log-size` | integer | — | `logging.max_log_size` | when passed | | `--log-file-count` | integer | — | `logging.log_file_count` | when passed | | `--syslog-enabled` | `true`/`false` | — | `logging.syslog.enabled` | when passed | | `--syslog-facility` | string | — | `logging.syslog.facility` | when passed | | `--syslog-identifier` | string | — | `logging.syslog.identifier` | when passed | | `--proxy-address-http` | string | — | `proxy.address_http` | when passed | | `--proxy-address-tls` | string | — | `proxy.address_tls` | when passed | | `--proxy-certificates` | string | — | `proxy.certificates` | when passed | | `--proxy-tls-grade` | string | — | `proxy.tls_grade` | when passed | | `--proxy-default-certificate` | string | — | `proxy.default_certificate` | when passed | | `--upstream-conf` | string | — | `proxy.upstream.conf` | when passed | | `--upstream-healthcheck-method` | string | — | `proxy.upstream.healthcheck.method` | when passed | | `--upstream-healthcheck-interval` | integer | — | `proxy.upstream.healthcheck.interval` | when passed | | `--geoip-country-url` | string | — | `platform.geoip.country.url` | when passed | | `--geoip-country-path` | string | — | `platform.geoip.country.path` | when passed | | `--geoip-asn-url` | string | — | `platform.geoip.asn.url` | when passed | | `--geoip-asn-path` | string | — | `platform.geoip.asn.path` | when passed | | `--geoip-city-url` | string | — | `platform.geoip.city.url` | when passed | | `--geoip-city-path` | string | — | `platform.geoip.city.path` | when passed | | `--geoip-refresh-secs` | integer | — | `platform.geoip.refresh_secs` | when passed | | `--acme-enabled` | `true`/`false` | — | `proxy.acme.enabled` | when passed | | `--acme-port` | integer | — | `proxy.acme.port` | when passed | | `--acme-email` | string | — | `proxy.acme.email` | when passed | | `--acme-storage-path` | string | — | `proxy.acme.storage_path` | when passed | | `--acme-storage-type` | string | — | `proxy.acme.storage_type` | when passed | | `--acme-development` | `true`/`false` | — | `proxy.acme.development` | when passed | | `--acme-redis-url` | string | — | `proxy.acme.redis_url` | when passed | | `--content-scanning-enabled` | `true`/`false` | — | `proxy.content_scanning.enabled` | when passed | | `--content-scanning-clamav-server` | string | — | `proxy.content_scanning.clamav_server` | when passed | | `--content-scanning-max-file-size` | integer | — | `proxy.content_scanning.max_file_size` | when passed | | `--internal-services-enabled` | `true`/`false` | — | `proxy.internal_services.enabled` | when passed | | `--internal-services-port` | integer | — | `proxy.internal_services.port` | when passed | | `--internal-services-bind-ip` | string | — | `proxy.internal_services.bind_ip` | when passed | | `-t`, `--terminal` | switch | `false` | `daemon.enabled` → `false` | when given | :::note[One flag is not in every build] `--captcha-provider` exists only in builds that carry the proxy, so it is absent from `synapse --help` on Windows and on the legacy agent. Every other flag is present everywhere, whether or not the subsystem behind it is. ::: ### Flags that do not write a configuration key | Flag | Value | Default | What it does | | --- | --- | --- | --- | | `-c`, `--config` | path | — | The YAML file to load. Without it, synapse starts from built-in defaults plus flags and environment. | | `--security-rules-config` | path | `security_rules.yaml` | Local access and WAF rules, used when no Gen0Sec API key is configured. | | `--log-level` | `error` / `warn` / `info` / `debug` / `trace` | — | Sets the log level. Beats `logging.level` and `RUST_LOG` - see [Exception 1](#exception-1-the-log-level). The flag has no `off`; only `logging.level`, `LOGGING_LEVEL` and `RUST_LOG` accept it. | | `--event-socket` | path | `/var/run/synapse-events.sock` (Linux), `C:\ProgramData\Gen0Sec\Synapse\events.sock` (Windows) | Where the daemon serves events for terminal mode, and where `--terminal` connects. `daemon.event_socket` wins over it in the daemon only - see [Exception 2](#exception-2-the-event-and-control-sockets). | | `--control-socket` | path | `/var/run/synapse-control.sock` (Linux), `C:\ProgramData\Gen0Sec\Synapse\control.sock` (Windows) | Where the daemon serves control commands. `daemon.control_socket` wins over it in the daemon only - see [Exception 2](#exception-2-the-event-and-control-sockets). | | `--service` | `install` / `uninstall` | — | Windows only: register or remove the Windows service, then exit. | | `--upgrade` | switch | — | Signal the running synapse to replace its binary in place, then exit. Linux only; on Windows the flag is accepted and synapse starts normally instead. It needs a build that carries graceful restart, which the legacy agent does not: there the signal has no handler and **stops the process** rather than upgrading it. | | `--show-firewall-rules` | `text` / `json` | — | Print the rules the kernel is actually enforcing, read out of the pinned maps, then exit. Read-only, needs root, and does not need synapse to be running. | | `--firewall-pin-root` | path | `/sys/fs/bpf/hillock` | Where to look for those pinned maps. Only meaningful with `--show-firewall-rules`. | | `--force-stop` | switch | — | Stop synapse and remove its kernel datapath - attached programs and pinned maps outlive the process, so a crashed agent leaves enforcement running with nothing owning it. Needs root; safe to run twice. | | `--force-stop-dry-run` | switch | — | Print what `--force-stop` would remove, and remove nothing. | | `--clear-certificate` | host name | — | Remove one certificate from local storage and from Redis, then exit. Needs a build that carries the proxy; on Windows and on the legacy agent the flag is accepted and synapse starts normally instead. | | `-T`, `--tcpdump` | switch | `false` | Non-interactive packet output instead of the terminal view. Mutually exclusive with `--terminal`. | | `-r`, `--reply` | switch | `false` | In terminal mode, correlate SYN with SYN-ACK. | | `--block-mode` | switch | `false` | In terminal and replay runs, block what is classified as a threat and add the source address to the runtime blocklist. | | `--pcap-file` | path | — | Replay a capture file through the full pipeline: same fingerprints, classification and rule evaluation as live traffic. Not supported with `--daemon`. | | `--pcap-limit` | integer | `0` | Stop after this many packets. `0` reads the whole file. | | `--pcap-start` | `YYYY-MM-DD` or `YYYY-MM-DD HH:MM:SS` (UTC) | — | Skip packets before this timestamp. | | `--pcap-end` | `YYYY-MM-DD` or `YYYY-MM-DD HH:MM:SS` (UTC) | — | Skip packets after this timestamp. | | `--captcha-token-ttl` | integer | `7200` | **Nothing.** Parsed and never read. | | `--captcha-cache-ttl` | integer | `300` | **Nothing.** Parsed and never read. | :::note[`--disable-xdp` does not exist] Synapse's in-repo `docs/CONFIGURATION.md` lists a `--disable-xdp` flag. There is no such flag in the code, at this release or any other. To not attach XDP, set `firewall.mode: none` or pick a non-XDP `capture.mode`. ::: ## Environment variables ### The deprecated prefixes Every variable in the merge path is looked up unprefixed first. If that is unset, a deprecated prefixed form is tried, and using one logs a deprecation warning naming the variable to move to: - all variables: `NAME`, then `AX_NAME`. - the ten platform variables marked ◆ below: `NAME`, then `AX_NAME`, then `AX_ARXIGNIS_NAME`, then `ARXIGNIS_NAME`. An unset variable and an empty one are different: an empty value is a value, and it is assigned. ### Variables that override a configuration key 116 variables, listed by the section of `config.yaml` they write. 13 further names are fallbacks, shown under the variable they stand in for. The key's meaning is documented on [Configuration](./configuration.mdx); this table only maps the variable onto the key. #### Mode and threading | Variable | YAML key | Value | Unparseable value | | --- | --- | --- | --- | | `MODE` | `mode` | string | used as given | | `MULTI_THREAD` | `multi_thread` | `true` / `false` | ignored | | `WORKER_THREADS` | `worker_threads` | integer | ignored | #### Network | Variable | YAML key | Value | Unparseable value | | --- | --- | --- | --- | | `NETWORK_IFACE` | `network.iface` | string | used as given | | `NETWORK_IFACES` | `network.ifaces` | comma-separated list | used as given | | `NETWORK_IP_VERSION` | `network.ip_version` | `ipv4` / `ipv6` / `both` | ignored, and it warns | #### Firewall | Variable | YAML key | Value | Unparseable value | | --- | --- | --- | --- | | `FIREWALL_MODE`
falls back to `NETWORK_FIREWALL_MODE` | `firewall.mode` | `auto` / `xdp` / `nftables` / `iptables` / `none` | built-in default | #### IDS (Thalamus) | Variable | YAML key | Value | Unparseable value | | --- | --- | --- | --- | | `IDS_ENABLED`
falls back to `FIREWALL_IDS_ENABLED` | `ids.enabled` | `true` / `false` | built-in default | | `IDS_RULE_PATHS`
falls back to `FIREWALL_IDS_RULE_PATHS` | `ids.rule_paths` | comma-separated list | used as given | | `IDS_SNAPLEN`
falls back to `FIREWALL_IDS_SNAPLEN` | `ids.snaplen` | integer | built-in default | | `IDS_CAPTURE_MODE`
falls back to `FIREWALL_IDS_CAPTURE_MODE` | `ids.capture_mode` | `xdp` / `afpacketrecv` | built-in default, and it warns | | `IDS_POLL_TIMEOUT_MS`
falls back to `FIREWALL_IDS_POLL_TIMEOUT_MS` | `ids.poll_timeout_ms` | integer | built-in default | | `IDS_FLOW_TIMEOUT_SECS`
falls back to `FIREWALL_IDS_FLOW_TIMEOUT_SECS` | `ids.flow_timeout_secs` | integer | built-in default | | `IDS_MAX_FLOWS`
falls back to `FIREWALL_IDS_MAX_FLOWS` | `ids.max_flows` | integer | built-in default | | `IDS_CLEANUP_INTERVAL_SECS`
falls back to `FIREWALL_IDS_CLEANUP_INTERVAL_SECS` | `ids.cleanup_interval_secs` | integer | built-in default | | `IDS_STATS_LOG_INTERVAL_SECS`
falls back to `FIREWALL_IDS_STATS_LOG_INTERVAL_SECS` | `ids.stats_log_interval_secs` | integer | built-in default | | `IDS_ENFORCE_BLOCK`
falls back to `FIREWALL_IDS_ENFORCE_BLOCK` | `ids.enforce_block` | `true` / `false` | built-in default | | `IDS_ENFORCE_EDGE_DROP`
falls back to `FIREWALL_IDS_ENFORCE_EDGE_DROP` | `ids.enforce_edge_drop` | `true` / `false` | built-in default | #### Capture | Variable | YAML key | Value | Unparseable value | | --- | --- | --- | --- | | `CAPTURE_MODE` | `capture.mode` | `auto` / `xdp` / `af_packet` / `af_packet_recv` / `pfring` / `af_xdp` / `ndis` | built-in default, and it warns | | `CAPTURE_SNAPLEN` | `capture.snaplen` | integer | built-in default | | `CAPTURE_ENABLED` | `capture.enabled` | `true` / `false` | ignored | | `CAPTURE_FILTER` | `capture.filter` | string | used as given | | `PFRING_ZERO_COPY` | `capture.pfring.zero_copy` | `true` / `false` | built-in default | | `PFRING_CLUSTER_ID` | `capture.pfring.cluster_id` | integer | cleared | #### ML fingerprint classifier | Variable | YAML key | Value | Unparseable value | | --- | --- | --- | --- | | `CLASSIFIER_ENABLED` | `classifier.enabled` | `true` / `false` | ignored | | `CLASSIFIER_MODEL_PATH` | `classifier.model_path` | string | used as given | | `CLASSIFIER_CACHE_SIZE` | `classifier.cache_size` | integer | ignored | | `CLASSIFIER_THRESHOLD` | `classifier.threshold` | number | ignored | #### Platform: API, GeoIP and threat feed | Variable | YAML key | Value | Unparseable value | | --- | --- | --- | --- | | `API_KEY` ◆ | `platform.api_key` | string | used as given | | `BASE_URL` ◆ | `platform.base_url` | string | used as given | | `AUTHCHECK_URL` ◆ | `platform.authcheck.url` | string | used as given | | `REMOTE_CONFIG_URL` ◆ | `platform.remote_config.url` | string | used as given | | `AGENT_EVENTS_URL` ◆ | `platform.agent_events.url` | string | used as given | | `DATA_EVENTS_URL` ◆ | `platform.data_events.url` | string | used as given | | `TELEMETRY_URL` ◆ | `platform.telemetry_endpoint.url` | string | used as given | | `TELEMETRY_SENDING_ENABLED` ◆
falls back to `LOG_SENDING_ENABLED` | `platform.telemetry_sending_enabled` | `true` / `false` | ignored | | `INCLUDE_RESPONSE_BODY` ◆ | `platform.include_response_body` | `true` / `false` | built-in default | | `MAX_BODY_SIZE` ◆ | `platform.max_body_size` | integer | built-in default | | `THREAT_MMDB_URL` | `platform.threat.url` | string | used as given | | `THREAT_MMDB_PATH` | `platform.threat.path` | string | used as given | | `THREAT_MMDB_REFRESH_SECS` | `platform.threat.refresh_secs` | integer | ignored | | `GEOIP_COUNTRY_URL` | `platform.geoip.country.url` | string | used as given | | `GEOIP_COUNTRY_PATH` | `platform.geoip.country.path` | string | used as given | | `GEOIP_ASN_URL` | `platform.geoip.asn.url` | string | used as given | | `GEOIP_ASN_PATH` | `platform.geoip.asn.path` | string | used as given | | `GEOIP_CITY_URL` | `platform.geoip.city.url` | string | used as given | | `GEOIP_CITY_PATH` | `platform.geoip.city.path` | string | used as given | | `GEOIP_ENABLED` | `platform.geoip.enabled` | `true` / `false` | ignored | | `THREAT_MMDB_ENABLED` | `platform.threat.enabled` | `true` / `false` | ignored | | `GEOIP_COUNTRY_HEADERS` | `platform.geoip.country.headers` | comma-separated `k=v` | cleared | | `GEOIP_ASN_HEADERS` | `platform.geoip.asn.headers` | comma-separated `k=v` | cleared | | `GEOIP_CITY_HEADERS` | `platform.geoip.city.headers` | comma-separated `k=v` | cleared | | `THREAT_MMDB_HEADERS` | `platform.threat.headers` | comma-separated `k=v` | cleared | | `GEOIP_REFRESH_SECS` | `platform.geoip.refresh_secs` | integer | ignored | #### Logging | Variable | YAML key | Value | Unparseable value | | --- | --- | --- | --- | | `LOGGING_LEVEL` | `logging.level` | string | used as given | | `LOGGING_FILE_ENABLED` | `logging.file_logging_enabled` | `true` / `false` | built-in default | | `LOGGING_DIRECTORY` | `logging.log_directory` | string | used as given | | `LOGGING_MAX_FILE_SIZE` | `logging.max_log_size` | integer | built-in default | | `LOGGING_FILE_COUNT` | `logging.log_file_count` | integer | built-in default | | `LOGGING_SYSLOG_ENABLED` | `logging.syslog.enabled` | `true` / `false` | built-in default | | `LOGGING_SYSLOG_FACILITY` | `logging.syslog.facility` | string | used as given | | `LOGGING_EVENTBRIDGE_LOG_ENABLED` | `logging.eventbridge_log.enabled` | `true` / `false` | ignored | | `LOGGING_SYSLOG_IDENTIFIER` | `logging.syslog.identifier` | string | used as given | | `LOGGING_ETW_ENABLED` | `logging.etw.enabled` | `true` / `false` | built-in default | | `LOGGING_WINDOWS_EVENT_LOG_ENABLED` | `logging.windows_event_log.enabled` | `true` / `false` | built-in default | | `LOGGING_WINDOWS_EVENT_LOG_SOURCE` | `logging.windows_event_log.source` | string | used as given | #### Daemon | Variable | YAML key | Value | Unparseable value | | --- | --- | --- | --- | | `DAEMON_ENABLED` | `daemon.enabled` | `true` / `false` | built-in default | | `DAEMON_PID_FILE` | `daemon.pid_file` | string | used as given | | `DAEMON_WORKING_DIRECTORY` | `daemon.working_directory` | string | used as given | | `DAEMON_USER` | `daemon.user` | string | used as given | | `DAEMON_GROUP` | `daemon.group` | string | used as given | | `DAEMON_CHOWN_PID_FILE` | `daemon.chown_pid_file` | `true` / `false` | built-in default | #### Proxy | Variable | YAML key | Value | Unparseable value | | --- | --- | --- | --- | | `REDIS_URL` | `proxy.redis.url` | string | used as given | | `REDIS_PREFIX` | `proxy.redis.prefix` | string | used as given | | `REDIS_SSL_CA_CERT_PATH` | `proxy.redis.ssl.ca_cert_path` | string | used as given | | `REDIS_SSL_CLIENT_CERT_PATH` | `proxy.redis.ssl.client_cert_path` | string | used as given | | `REDIS_SSL_CLIENT_KEY_PATH` | `proxy.redis.ssl.client_key_path` | string | used as given | | `REDIS_SSL_INSECURE` | `proxy.redis.ssl.insecure` | `true` / `false` | ignored | | `CONTENT_VERDICT_CACHE_TTL_SECS` | `proxy.content_scanning.verdict_cache_ttl_secs` | integer | ignored | | `CONTENT_SCAN_MEMORY_BUDGET` | `proxy.content_scanning.scan_memory_budget` | integer | ignored | | `CONTENT_SCANNING_ENABLED` | `proxy.content_scanning.enabled` | `true` / `false` | built-in default | | `CLAMAV_SERVER` | `proxy.content_scanning.clamav_server` | string | used as given | | `CONTENT_MAX_FILE_SIZE` | `proxy.content_scanning.max_file_size` | integer | built-in default | | `CAPTCHA_SITE_KEY` | `proxy.captcha.site_key` | string | used as given | | `CAPTCHA_SECRET_KEY` | `proxy.captcha.secret_key` | string | used as given | | `CAPTCHA_JWT_SECRET` | `proxy.captcha.jwt_secret` | string | used as given | | `CAPTCHA_PROVIDER` | `proxy.captcha.provider` | string | used as given | | `CAPTCHA_TOKEN_TTL` | `proxy.captcha.token_ttl` | integer | built-in default | | `CAPTCHA_CACHE_TTL` | `proxy.captcha.cache_ttl` | integer | built-in default | | `INTERNAL_SERVICES_ENABLED` | `proxy.internal_services.enabled` | `true` / `false` | built-in default | | `INTERNAL_SERVICES_PORT` | `proxy.internal_services.port` | integer | built-in default | | `INTERNAL_SERVICES_BIND_IP` | `proxy.internal_services.bind_ip` | string | used as given | | `INTERNAL_SERVICES_SERVE_ACME_CHALLENGE` | `proxy.internal_services.serve_acme_challenge` | `true` / `false` | built-in default | | `PROXY_PROTOCOL_ENABLED` | `proxy.protocol.enabled` | `true` / `false` | built-in default | | `PROXY_PROTOCOL_TIMEOUT` | `proxy.protocol.timeout_ms` | integer | built-in default | | `PROXY_ADDRESS_HTTP` | `proxy.address_http` | string | used as given | | `PROXY_ADDRESS_TLS` | `proxy.address_tls` | string | used as given | | `PROXY_CERTIFICATES` | `proxy.certificates` | string | used as given | | `PROXY_H2C` | `proxy.h2c` | `true` / `false` | ignored | | `PROXY_ALLOW_CONNECT_METHOD_PROXYING` | `proxy.allow_connect_method_proxying` | `true` / `false` | ignored | | `PROXY_KEEPALIVE_REQUEST_LIMIT` | `proxy.keepalive_request_limit` | integer | cleared | | `PROXY_TLS_GRADE` | `proxy.tls_grade` | string | used as given | | `PROXY_DEFAULT_CERTIFICATE` | `proxy.default_certificate` | string | used as given | | `PROXY_DEFAULT_CERT_PATH` | `proxy.default_cert_path` | string | used as given | | `PROXY_DEFAULT_KEY_PATH` | `proxy.default_key_path` | string | used as given | | `UPSTREAM_CONF` | `proxy.upstream.conf` | string | used as given | | `UPSTREAM_HEALTHCHECK_METHOD` | `proxy.upstream.healthcheck.method` | string | used as given | | `UPSTREAM_HEALTHCHECK_INTERVAL` | `proxy.upstream.healthcheck.interval` | integer | ignored | | `ACME_ENABLED` | `proxy.acme.enabled` | `true` / `false` | built-in default | | `ACME_PORT` | `proxy.acme.port` | integer | ignored | | `ACME_BIND_IP` | `proxy.acme.bind_ip` | string | used as given | | `ACME_EMAIL` | `proxy.acme.email` | string | used as given | | `ACME_STORAGE_PATH` | `proxy.acme.storage_path` | string | used as given | | `ACME_STORAGE_TYPE` | `proxy.acme.storage_type` | string | used as given | | `ACME_DEVELOPMENT` | `proxy.acme.development` | `true` / `false` | built-in default | | `ACME_REDIS_URL` | `proxy.acme.redis_url` | string | used as given | Two notes on that table: - Setting any one of `REDIS_SSL_CA_CERT_PATH`, `REDIS_SSL_CLIENT_CERT_PATH`, `REDIS_SSL_CLIENT_KEY_PATH` or `REDIS_SSL_INSECURE` creates the `proxy.redis.ssl` block if the YAML has none, which turns Redis TLS on as a side effect. - `CAPTCHA_TOKEN_TTL` and `CAPTCHA_CACHE_TTL` write `proxy.captcha.token_ttl` and `proxy.captcha.cache_ttl`, and both keys are read: the token TTL sets the expiry of the captcha token and the `Max-Age` of the cookie carrying it. The `--captcha-token-ttl` and `--captcha-cache-ttl` flags are the ones that do nothing - they are parsed and never read, so set the keys or the variables instead. ### Variables read at runtime These are not part of the configuration merge and have no YAML equivalent. They are read where they are used, so an invalid value is usually ignored silently at that point. #### Agent identity and status Reported to the Gen0Sec API. `AGENT_TAGS`, `AGENT_IP_ADDRESSES`, `AGENT_IPS`, `AGENT_HEARTBEAT_SECS` and `AGENT_ID` are read only when `platform.api_key` is set and the run is not a PCAP replay. `AGENT_NAME` and `HOSTNAME` are read on every run: they also name the instance in OpenTelemetry resource attributes and the host field on CEF lines. | Variable | Value | Default | What it does | | --- | --- | --- | --- | | `AGENT_NAME` | string | the host name | The name this agent reports. The agent's id is derived from it, so changing the name changes the id. | | `AGENT_TAGS` | comma-separated list | empty | Tags reported with the agent's status. | | `AGENT_IP_ADDRESSES` | comma-separated list | empty | The addresses this agent reports. Not validated as IP addresses. | | `AGENT_IPS` | comma-separated list | empty | Older name for the same thing, used only when `AGENT_IP_ADDRESSES` is unset. | | `AGENT_HEARTBEAT_SECS` | integer seconds | `30` | How often the agent reports status. An unparseable value falls back to 30. | | `AGENT_ID` | — | — | **Nothing.** Setting it logs `AGENT_ID is ignored; agent_id is derived from agent_name.` | | `HOSTNAME` | string | the host name | Used as the agent name when `AGENT_NAME` is unset, and as the host field on CEF and syslog lines. | #### Smart firewall | Variable | Value | Default | What it does | | --- | --- | --- | --- | | `AMYGDALA_PREBLOCK` | comma-separated `kind=value`, optionally `allow:kind=value` | empty | Seeds block and allow rules at startup. `kind` is one of `ja4`, `ja4s`, `ja4t`, `ja4ts`, `ja4l`, `ja4ls`, `ja4h`, `ja4x`, `ja4ssh`; entries with no `=`, or an unknown kind, are skipped with a warning. Setting it also starts the bridge even with `firewall.amygdala.enabled` false. | | `AMYGDALA_REACTOR_SCOPE` | `src_tuple`, `src_ip`, `src_ip:` | `src_ip` with prefix 32 | What a block covers: the full source tuple, the whole source address, or a prefix of it. An unknown value warns and leaves the default; an unparseable prefix falls back to 32. Setting it also starts the bridge. | | `AMYGDALA_SWEEP_INTERVAL_MS` | integer milliseconds | `30000` | How often expired entries are evicted. Values below 1000 are raised to 1000 with a warning; an unparseable value falls back to 30000. | | `DANGEROUS_AFXDP_OWN_MODE` | exactly `1` | unset | Permits an `afxdp_attach_mode` other than `shared`. Without it, any other value is refused and forced back to `shared`. Devmap forwarding can rewrite the source MAC, so use it only on isolated interfaces. | #### Logging and tracing | Variable | Value | Default | What it does | | --- | --- | --- | --- | | `RUST_LOG` | a bare level | `info` | Log level, consulted last - see [Exception 1](#exception-1-the-log-level). | | `SYNAPSE_LOG_FORMAT` | `json` (any case) | text | Writes `app.log`, `error.log` and the console line as JSON. `access.log` is unaffected - it is already JSON. | | `SYNAPSE_ACCESS_LOG_SAMPLE_RATE` | number from 0 to 1 | `telemetry.access_log.sample_rate` | Fraction of access-log events kept. A value outside the range, or an unparseable one, is ignored. | #### OpenTelemetry The endpoint, header and compression variables describe **one** OTLP destination, the environment's. It comes into being as soon as any of the four endpoint variables is set, and sits alongside the destination `telemetry.otlp.endpoint` names and the platform's. With only a per-signal endpoint set, the signals without one are not exported there at all. The order of those three matters for everything below: the environment's, then `telemetry.otlp.endpoint`, then the platform. A URL two of them name is pushed to once, as the earlier, carrying the headers of both — the earlier's winning a clash. The two timeouts are not so confined, and their rows say what each reaches. The last five are not about a destination at all. They shape the resource, the sampler and the metric interval, and are read whether or not any OTLP endpoint is set — though with no destination configured only the sampler still acts, since the file sink carries no resource and the metric interval times a reader that exists only per destination. :::caution[Headers reach one destination, not all of them] `OTEL_EXPORTER_OTLP_HEADERS` is attached to the **first** destination only. The one configured under `telemetry:` gets `telemetry.otlp.headers` and nothing else, and the platform gets its own key. Nothing fails when a token is set in the wrong place; the destination that needed it simply receives requests without it. Synapse does warn at startup for the one case it can see — headers set in the environment while the environment names no endpoint to send them to. Through 0.8.7 these variables reached every exporter, so a configuration that relied on one token covering all three destinations stops working on 0.8.8. See [Upgrade Notes](./upgrade-notes.mdx). ::: | Variable | Value | Default | What it does | | --- | --- | --- | --- | | `OTEL_EXPORTER_OTLP_ENDPOINT` | base URL | unset | Adds an OTLP destination, the first of the three. The signal's path (`/v1/logs`, `/v1/metrics`, `/v1/traces`) is appended, replacing one the URL already ends in. An empty value counts as unset. | | `OTEL_EXPORTER_OTLP_{LOGS,METRICS,TRACES}_ENDPOINT` | URL | unset | Sends that one signal to this URL as given, in place of the general endpoint. | | `OTEL_EXPORTER_OTLP_HEADERS` | comma-separated `k=v` | none | Headers for the **first destination only**, which is where an API token for it goes. Values are percent-decoded; an entry with no `=`, an empty name or an empty value is skipped. | | `OTEL_EXPORTER_OTLP_{LOGS,METRICS,TRACES}_HEADERS` | comma-separated `k=v` | none | That signal's headers, read **in place of** the general variable rather than alongside it. | | `OTEL_EXPORTER_OTLP_COMPRESSION` | `gzip`, `zstd`, `none` | unset, which compresses nothing | Compression for the first destination only; the other two are sent uncompressed either way. Any other value warns at startup and is treated as `none`. | | `OTEL_EXPORTER_OTLP_{LOGS,METRICS,TRACES}_COMPRESSION` | `gzip`, `zstd`, `none` | unset | That signal's compression, in place of the general variable — including when it is set to nothing. | | `OTEL_EXPORTER_OTLP_TIMEOUT` | integer milliseconds | `10000` | Two timeouts at once. It is the per-request timeout of the environment's destination — and, because synapse leaves the SDK's own timeout unset, the deadline the SDK gives the export retry loop at **every** destination. An unparseable value falls back to 10 s for both. | | `OTEL_EXPORTER_OTLP_{LOGS,METRICS,TRACES}_TIMEOUT` | integer milliseconds | unset | That signal's retry deadline, at every destination, read in place of the general variable; an unparseable value falls through to it. No destination's per-request timeout changes. | | `OTEL_METRIC_EXPORT_INTERVAL` | integer milliseconds | `30000` | How often metrics are exported, to every destination. | | `OTEL_SERVICE_NAME` | string | `synapse` | The reported service name. `telemetry.resource.service_name` wins over it. | | `OTEL_RESOURCE_ATTRIBUTES` | comma-separated `k=v`, percent-encoded | none | Extra resource attributes. A bad escape discards the whole variable with a warning; `telemetry.resource.attributes` is applied after and wins. | | `OTEL_TRACES_SAMPLER` | `always_on`, `always_off`, `traceidratio`, `parentbased_always_on`, `parentbased_always_off`, `parentbased_traceidratio` | `parentbased_traceidratio` | Trace sampler. An unknown value warns and uses the default. | | `OTEL_TRACES_SAMPLER_ARG` | number from 0 to 1 | `telemetry.tracing.sample_rate` | The ratio for the `traceidratio` samplers. Ignored by the `always_*` ones. | #### Proxy and ACME | Variable | Value | Default | What it does | | --- | --- | --- | --- | | `REDIS_URL` | Redis URL | `redis://127.0.0.1:6379` | Several ACME storage paths read this variable directly rather than `proxy.redis.url`, so a Redis URL set only in the YAML leaves those paths on localhost. Set the variable too, or set `proxy.acme.redis_url`. | | `SYNAPSE_NODE_IP` | IP address | the local socket address | The node address recorded as the destination for post-TLS IDS events. Intended for the Kubernetes downward API (`status.hostIP`). An unparseable value is ignored. | #### Windows | Variable | Value | Default | What it does | | --- | --- | --- | --- | | `SYNAPSE_WINDOWS_IFINDEX` | positive integer | unset | Interface index for Windows IDS capture. Wins over `SYNAPSE_WINDOWS_IFACE`. | | `SYNAPSE_WINDOWS_IFACE` | interface name or number | unset | Interface for Windows IDS capture, used when the index is unset or zero. | #### Diagnostics Off unless set. The first three are switches: any value turns them on, including an empty one. | Variable | Value | What it does | | --- | --- | --- | | `SYNAPSE_BPF_PROBE` | any value | Runs map-creation probes at startup and logs the results. | | `SYNAPSE_LIBBPF_VERBOSE` | any value | Sends libbpf and verifier output to the log. | | `SYNAPSE_IDS_SELFTEST` | any value | Runs an IDS-to-firewall decision self-test at startup. | | `SYNAPSE_FP_BUS_WRITER` | `1` or `true` | Starts a second, bus-driven fingerprint writer alongside the normal one. A flow can then be logged twice. | | `THALAMUS_TRACE_ENABLED` | `1`, `true`, `yes` or `on` | Traces IDS rule evaluation per packet. The four variables below apply only when it is on. | | `THALAMUS_TRACE_DIR` | path (default `./trace`) | Where the trace files are written. Created if missing. | | `THALAMUS_TRACE_SIDS` | comma-separated rule ids | Trace only these rules. Unset traces all of them. | | `THALAMUS_TRACE_FLOWS` | comma-separated `ip:port-ip:port` | Trace only these flows. | | `THALAMUS_TRACE_PREVIEW_BYTES` | positive integer (default `96`) | How many payload bytes each trace entry records. | #### Set by synapse itself | Variable | What it does | | --- | --- | | `ECDYSIS_RUST_UPGRADE` | Marks a process as the child of a graceful binary upgrade. Set by the upgrade machinery; not an operator setting. | #### Custom builds only `SYNAPSE_EGRESS_IDS` and `SYNAPSE_EGRESS_IFACES` configure the egress IDS, which is in no published package - see [What your package contains](#what-your-package-contains). ### Documented elsewhere, never read `BPF_STATS_ENABLED`, `TCP_FINGERPRINT_ENABLED` and `TOKIO_WORKER_THREADS` appear in synapse's in-repo `docs/ENV_VARS.md`. No Rust file reads any of them. The thread count is `worker_threads` in the YAML, or `WORKER_THREADS`. ## What your package contains Not every subsystem is in every package. A configuration key for one that is missing still parses; what happens then depends on the key, and it is not consistent. | | `synapse-agent`
`synapse-proxy` | `synapse-agent-ml`
`synapse-proxy-ml` | Legacy agent
(not published) | Windows | | --- | --- | --- | --- | --- | | L7 proxy: TLS termination, upstreams, ACME, WAF, captcha | yes | yes | no | no | | eBPF/XDP datapath and XDP capture | yes | yes | no | no | | `AF_PACKET` capture | yes | yes | yes | no | | NDIS capture | - | - | - | yes | | Smart firewall (`firewall.amygdala`) | yes | yes | `iptables`/`nftables` only | no | | Inline JA4 ClientHello enforcement | yes | yes | no | no | | ML scoring (`ml.*`) | **no** | yes | no | no | | Graceful binary upgrade (`--upgrade`) | yes | yes | no | no | | Egress IDS | no | no | no | no | | PF_RING capture | no | no | no | no | - The **`-ml` packages** are the only ones with ML scoring. They are published separately because the ONNX runtime needs glibc 2.38 or newer; the plain packages have a glibc 2.31 floor and the `-ml` packages 2.39. Both come in `x86_64` and `arm64`. - The **agent and proxy packages carry the same binary.** They differ in the unit file and in the configuration they ship. The binary being identical, a `synapse-agent` install can run a proxy: set `mode: proxy` in `config.yaml`. The two packages conflict with each other, so one host runs one of them. - **On the `synapse-proxy` package, `mode:` in `config.yaml` does nothing.** Its unit file carries `Environment="MODE=proxy"`, and `MODE` is an environment variable, so it beats the file - which still says `mode: "agent"`, because both packages ship the same `config.yaml`. This is the rule at the top of the page in its most consequential form: if you need that host to stop proxying, edit the unit or override `MODE`, not the YAML. - The **legacy agent** is the static build for old kernels and old glibc: agent only, `AF_PACKET` capture, `iptables`/`nftables` enforcement. It is **not among the published release artefacts** - it is built and tested per change, so ask support for a build if you need it. The column is here because the keys behave the same way on it. - The **Windows build** is agent only. It captures fingerprints through NDIS; it has no proxy, no eBPF, and no smart-firewall enforcement. - **Egress IDS and PF_RING capture are in no published package.** Both need a build made with non-default options. ### Keys that are accepted and do nothing | Key | Needs | Without it | | --- | --- | --- | | `ml.*` | an `-ml` package, on Linux | **named in a startup warning**: `Unused config options in : ml`. No scoring happens | | `CLASSIFIER_*` | an `-ml` package, on Linux | accepted, no scoring, nothing logged | | `capture.mode: pf_ring` | a custom build | accepted, no capture backend starts, nothing logged | | `firewall.amygdala.*` | Linux; a Windows build with the NDIS bridge, which no published Windows package is | accepted, no enforcement, nothing logged | | `ids.egress_ifaces`, `SYNAPSE_EGRESS_IDS` | a custom build | accepted, no egress IDS, nothing logged | | `network.interfaces` with the `ids_egress` feature | a custom build | **synapse refuses to start**: `network.interfaces enables 'ids_egress', but this build has no egress IDS` | Three outcomes, then, and it is worth knowing which one you are in. The last row refuses to start. The `ml.*` row is named in the `Unused config options` warning synapse logs for any section its build does not recognise - which is also how you find a typo in a section name. The rest are silent, and the only signal is the absence of a log line saying the subsystem started, so check the log on first start rather than assuming the key took effect. The `CLASSIFIER_*` variables are silent where `ml.*` is not, because the unused-key warning comes from parsing the file: nothing inspects the environment for variables that this build cannot use. `capture.mode: auto` never selects PF_RING, so it is safe everywhere: it resolves to XDP when the eBPF programs load, `AF_PACKET` when they do not, and NDIS on Windows. --- # Daemon Mode Source: https://docs.gen0sec.com/docs/client/daemon-mode # Daemon Mode Synapse supports running as a daemon (background service) with proper privilege dropping and process management. ## Features - **Background execution** - Runs as a detached background process - **PID file management** - Creates and manages PID files for process control - **Privilege dropping** - Can drop privileges to a specified user and group after initialization - **Output redirection** - Redirects stdout and stderr to configurable log files - **Working directory** - Configurable working directory for the daemon - **Signal handling** - Proper signal handling for graceful shutdown ## Configuration ### YAML Configuration ```yaml daemon: enabled: false pid_file: "/var/run/synapse.pid" working_directory: "/var/lib/synapse" user: root group: root chown_pid_file: true ``` ### Command Line ```bash synapse --daemon -c /etc/synapse/config.yaml ``` ### Environment Variables ```bash export DAEMON_ENABLED="true" export DAEMON_PID_FILE="/var/run/synapse.pid" export DAEMON_WORKING_DIRECTORY="/" export DAEMON_STDOUT="/var/log/synapse.out" export DAEMON_STDERR="/var/log/synapse.err" export DAEMON_USER="nobody" export DAEMON_GROUP="daemon" export DAEMON_CHOWN_PID_FILE="true" ``` ## Process Management ### Starting the Daemon ```bash synapse --daemon -c /etc/synapse/config.yaml ``` ### Stopping the Daemon ```bash # Using PID file kill $(cat /var/run/synapse.pid) # Or send SIGTERM kill -TERM $(cat /var/run/synapse.pid) # Graceful shutdown with SIGINT kill -INT $(cat /var/run/synapse.pid) ``` ### Checking Status ```bash # Check if process is running ps aux | grep synapse # Or check PID file if [ -f /var/run/synapse.pid ]; then pid=$(cat /var/run/synapse.pid) if ps -p $pid > /dev/null; then echo "Synapse is running (PID: $pid)" else echo "Synapse is not running (stale PID file)" fi else echo "Synapse is not running" fi ``` ### Viewing Logs In daemon mode, logs are split: - **stdout** (`/var/log/synapse.out`) - Application logs (info, debug, warn, error) - **stderr** (`/var/log/synapse.err`) - Panic messages and system errors ```bash # Tail application logs tail -f /var/log/synapse.out # Tail error output tail -f /var/log/synapse.err # View both logs simultaneously tail -f /var/log/synapse.out /var/log/synapse.err ``` ## Security Considerations ### Privilege Dropping When running as daemon with a privileged user (e.g., root) to bind to ports < 1024 or attach XDP programs, it's recommended to drop privileges after initialization: ```bash synapse --daemon -c /etc/synapse/config.yaml ``` This will: 1. Start as root (or privileged user) 2. Bind to privileged ports (80, 443) 3. Attach XDP programs to network interfaces 4. Drop privileges to `nobody:daemon` 5. Continue running as unprivileged user ### File Permissions Ensure proper permissions for daemon files: ```bash # Create log directory sudo mkdir -p /var/log/synapse sudo chown nobody:daemon /var/log/synapse sudo chmod 755 /var/log/synapse # Create PID directory sudo mkdir -p /var/run sudo chmod 755 /var/run # Set up log files sudo touch /var/log/synapse.out /var/log/synapse.err sudo chown nobody:daemon /var/log/synapse.out /var/log/synapse.err sudo chmod 644 /var/log/synapse.out /var/log/synapse.err ``` ## Systemd Integration Create a systemd service file for easier management: :::caution[Only if you did not install from a package] The `synapse-agent`, `synapse-proxy`, `synapse-agent-ml` and `synapse-proxy-ml` packages each ship and own a systemd unit of the same name — if you installed one, use it and skip this section. The unit below is for a manually placed binary. Do not enable both. Two agents on one host contend for the same interface: the XDP link is pinned, so the second one to attach fails, and detaching by hand returns `EBUSY`. ::: ```ini # /etc/systemd/system/synapse.service [Unit] Description=Synapse Reverse Proxy and Firewall After=network-online.target Wants=network-online.target [Service] Type=forking PIDFile=/var/run/synapse.pid ExecStart=/usr/local/bin/synapse --daemon -c /etc/synapse/config.yaml ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure RestartSec=5s # Security settings NoNewPrivileges=true PrivateTmp=true ProtectSystem=strict ProtectHome=true ReadWritePaths=/var/log/synapse /var/run [Install] WantedBy=multi-user.target ``` Manage with systemd: ```bash # Enable service sudo systemctl enable synapse # Start service sudo systemctl start synapse # Stop service sudo systemctl stop synapse # Restart service sudo systemctl restart synapse # View status sudo systemctl status synapse # View logs sudo journalctl -u synapse -f ``` ## Troubleshooting ### Daemon Won't Start Check: 1. Log files for errors: `cat /var/log/synapse.err` 2. Permissions on log directory and PID file location 3. User/group exists: `id nobody` 4. Configuration file is valid: `synapse -c /etc/synapse/config.yaml` (without `--daemon`) ### Permission Denied Errors If you see permission errors: - Ensure log directory is writable by daemon user - Ensure PID file location is writable - Check that user/group specified exists - Verify file system permissions ### Stale PID File If daemon won't start due to existing PID file: ```bash # Check if process is actually running ps -p $(cat /var/run/synapse.pid) # If not running, remove stale PID file sudo rm /var/run/synapse.pid # Then start daemon synapse --daemon -c /etc/synapse/config.yaml ``` ## Usage Examples ### Basic Daemon ```bash synapse --daemon -c /etc/synapse/config.yaml ``` ### Custom Settings ```yaml # /etc/synapse/config.yaml daemon: enabled: true pid_file: "/var/run/synapse.pid" working_directory: "/var/lib/synapse" user: root group: root chown_pid_file: true network: iface: "eth0" platform: api_key: "your-key" ``` ```bash synapse -c /etc/synapse/config.yaml ``` ### With Configuration File ```bash synapse -c /etc/synapse/config.yaml ``` Example config.yaml: ```yaml daemon: enabled: true pid_file: "/var/run/synapse.pid" working_directory: "/var/lib/synapse" user: root group: root chown_pid_file: true network: iface: "eth0" platform: api_key: "your-key" ``` ## Next Steps - [Configuration Reference](./configuration) - Complete configuration options - [Test Your Setup](./testing) - Validate your daemon setup --- # Helm Charts Source: https://docs.gen0sec.com/docs/client/helm-charts # Helm Charts Gen0Sec publishes three charts. Which one you want depends on whether you are deploying the dataplane, the control loop, or both. ```bash helm repo add gen0sec https://helm.gen0sec.com helm repo update helm search repo gen0sec ``` :::note[The repo root is not browsable] `https://helm.gen0sec.com` returns 404 in a browser. That is normal for a chart repository — Helm fetches `/index.yaml`, which is what `helm repo add` reads. ::: ## The three charts | Chart | Deploys | Use it when | |---|---|---| | `synapse` | One Synapse workload — proxy *or* agent | You want a single dataplane workload and manage config yourself | | `synapse-stack` | Proxy + agent + the operator, from one release | **The normal choice.** One release, one namespace, config rollouts handled | | `synapse-operator` | The operator alone | The dataplane is already deployed, or lives elsewhere | `synapse-stack` is an umbrella: it depends on the `synapse` chart **twice**, aliased as `proxy` and `agent`, so each gets its own values tree and its own identity. That aliasing is the single most important thing to understand about it — see [Values live under the aliases](#values-live-under-the-aliases). ## You need an image pull secret The dataplane image is private. A default `helm install` with no pull secret leaves pods in `ImagePullBackOff`, which is the most common first-run failure. The pull secret takes the **repository credential** (Clients → Repository Access, with the `registry:pull` scope), not the platform API key: ```bash export GEN0SEC_KEY='' kubectl create namespace synapse-os --dry-run=client -o yaml | kubectl apply -f - kubectl create secret docker-registry gen0sec-registry \ --docker-server=registry.gen0sec.com \ --docker-username=x \ --docker-password="$GEN0SEC_KEY" \ -n synapse-os ``` Then reference it, and point the image at the registry you have access to: ```yaml proxy: image: repository: registry.gen0sec.com/gen0sec/synapse tag: "0.8.8" imagePullSecrets: - name: gen0sec-registry ``` The **operator** image is a different story — `ghcr.io/gen0sec/synapse-operator` is public and pulls anonymously, so `operator.imagePullSecrets` is only needed if you repoint it at a private mirror. :::caution[`operator.imagePullSecrets` needs synapse-stack 0.5.1+] Earlier charts have no such field on the operator deployments. The value is accepted and silently discarded — the pods just sit in `ImagePullBackOff` with no indication why. On an older chart, put the secret on the ServiceAccount instead, which Kubernetes then applies to every pod using it: ```bash kubectl -n synapse-os patch serviceaccount synapse-operator \ -p '{"imagePullSecrets":[{"name":"gen0sec-registry"}]}' ``` ::: ## The `synapse` chart One workload, from `ghcr.io/gen0sec/synapse` by default. | Value | Default | Notes | |---|---|---| | `kind` | `Deployment` | Set `DaemonSet` for the agent role — one pod per node, XDP on the host NIC. `replicaCount`, `autoscaling` and `strategy` are ignored then. | | `hostNetwork` | `false` | Required to attach XDP to a host interface. `dnsPolicy` then defaults to `ClusterFirstWithHostNet`. | | `securityContext` | `privileged: true` | Plus `SYS_ADMIN`, `NET_ADMIN`, `BPF`, `SYS_RESOURCE`. eBPF needs them; without them the agent falls back off the eBPF path. | | `synapse.config` | a full `config.yaml` | Rendered into a ConfigMap and mounted at `/etc/synapse/config.yaml`. | | `synapse.upstreams` | a file-provider example | Rendered as `upstreams.yaml`. Leave it empty when the operator owns routing. | | `synapse.upstreamsConfigMap` | `""` | Mount `upstreams.yaml` from *another* ConfigMap — the one the operator's UpstreamsResolver writes with ClusterIPs already substituted. | | `env` | `SYNAPSE_LOG_FORMAT: json` | YAML config outranks env vars, so use env only for what the config file does not cover. | The chart's `synapse.config` is a full agent configuration — capture backend, firewall mode, IDS, platform keys, proxy listeners. Every key is explained in [Configuration](./configuration.mdx). :::caution[Leave `service.exposeHealth` off] The health endpoint binds **loopback inside the container**, so a Service port pointing at it never routes — a request to the Pod IP on that port is refused, not answered. On `type: LoadBalancer` it is worse than useless: the cloud controller turns every Service port into a listener *and* opens the load balancer's security group to `0.0.0.0/0` for it. That is an internet-facing listener to nothing, on a port that reads as an administrative endpoint to anyone scanning. ::: ## The `synapse-stack` chart One release, both dataplane workloads, and the operator. ```bash helm upgrade --install synapse-stack gen0sec/synapse-stack \ -n synapse-os --create-namespace \ -f synapse-values.yaml kubectl -n synapse-os rollout status deploy/synapse-proxy ``` By default `proxy.enabled` is **true** and `agent.enabled` is **false** — turn the agent on when you want per-node XDP observation alongside the proxy. Both namespaces default to `synapse-os`: ```yaml global: namespaces: synapse: synapse-os operator: synapse-os ``` Older deployments split these across `synapse-system` / `ingress-synapse` / `synapse`. For anything greenfield, keep them together. ### Values live under the aliases Because the umbrella mounts the same subchart twice, values go under `proxy:` and `agent:` — **never** under `synapse:`. This is the mistake that silently does nothing: ```yaml # Correct proxy: image: tag: "0.8.8" agent: enabled: true kind: DaemonSet hostNetwork: true ``` The chart also sets `nameOverride` and `fullnameOverride` per alias so pods come out as `synapse-proxy-*` and `synapse-agent-*` rather than `-synapse-*`. That naming is load-bearing — the operator's label selector matches on it. :::caution[The label selector must match both workloads] `operator.labelSelector` defaults to the set-based form: ``` app.kubernetes.io/name in (synapse-proxy,synapse-agent) ``` The operator filters its List by this selector. Narrow it to a single name and it silently stops config-hash rolling the *other* workload — no error, just a workload that never picks up config changes. ::: `clamav` and `valkey` are dependencies of the `synapse` subchart and are gated **off** under both aliases here, so aliasing does not deploy two copies of each. ### Turning on the operator's extras The stack chart is where the operator's optional reconcilers are exposed: ```yaml operator: enabled: true netvarsResolver: true # fill IDS HOME_NET/EXTERNAL_NET from the cluster upstreamsResolver: false # substitute Service FQDNs for ClusterIPs perWorkloadConfigHash: true # only roll the workload whose own config changed idsHotReloadHashExclude: false ``` What each one does is on the [Synapse Operator](./synapse-operator.mdx) page. The agent's ConfigMap already carries `synapse.gen0sec.com/resolve-netvars: "true"`, so enabling `netvarsResolver` is all that is needed on the operator side. ### Ingress mode deploys a second operator `operator.ingress.enabled` stands up a **separate** Deployment running `--ingress-mode`, alongside the config-hash one. It has to be separate: ingress mode replaces config-sync within a single process, so one process cannot do both. ```yaml operator: ingress: enabled: true ingressClass: synapse upstreamsOutConfigMap: "synapse-os/synapse-proxy" # required proxy: synapse: upstreams: "" # hand routing to the operator ``` Setting `proxy.synapse.upstreams: ""` is not optional. It makes the chart stop templating that key, which is what stops Helm's three-way merge from clobbering the operator's render on the next upgrade. Routing itself — Ingress objects, path types, TLS — is covered in [Kubernetes Ingress & Gateway API](./kubernetes-ingress.mdx). :::note[Needs synapse-stack 0.6.0 or newer] Before 0.6.0 the stack chart's ClusterRole granted no `networking.k8s.io` permissions, so the ingress-mode Deployment — which shares the same ServiceAccount — had its Ingress list/watch forbidden and rendered nothing. From 0.6.0 the rule is created automatically whenever `operator.ingress.enabled` is set. ::: Add `operator.ingress.gatewayAPI: true` to reconcile Gateway API objects as well. That one value drives both the `--gateway-api` flag and its RBAC, so the permission cannot go missing while the flag is on. It needs the Gateway API CRDs installed. ## The `synapse-operator` chart The operator on its own, for when the dataplane is deployed some other way. ```bash helm upgrade --install synapse-operator gen0sec/synapse-operator -n synapse-os ``` It ships hardened defaults the stack chart does not: non-root `65532`, read-only root filesystem, all capabilities dropped, `RuntimeDefault` seccomp. It also carries the optional extras a long-lived controller wants: | Value | Default | | |---|---|---| | `operator.service.enabled` | `true` | ClusterIP in front of the metrics port | | `operator.serviceMonitor.enabled` | `false` | prometheus-operator ServiceMonitor; only rendered when the CRDs exist | | `operator.pdb.create` | `false` | Auto-enabled above one replica | | `operator.networkPolicy.enabled` | `false` | Opens metrics + healthz ingress, unrestricted egress | | `operator.watchNamespace` | `""` | Restrict to one namespace; needed when running operators side by side | | `operator.image.tag` | `""` | Empty falls back to the chart's `appVersion`, so a chart bump moves the image | :::note[The two charts pin different operator versions] `synapse-operator` leaves `image.tag` empty and follows its own `appVersion`. `synapse-stack` pins an explicit, older operator tag. If you want a specific operator build under the stack chart, set `operator.image.tag` yourself rather than assuming the chart tracks the latest. ::: ## Verifying a deployment ```bash kubectl -n synapse-os get pods kubectl -n synapse-os rollout status deploy/synapse-proxy kubectl -n synapse-os logs deploy/synapse-operator ``` A config change should roll the dataplane on its own: ```bash kubectl -n synapse-os edit configmap synapse-proxy # change any key kubectl -n synapse-os get pod -l app.kubernetes.io/name=synapse-proxy \ -o jsonpath='{.items[0].metadata.annotations.synapse\.gen0sec\.com/config-hash}' ``` If that annotation never changes, the operator is not matching the workload — check `operator.labelSelector` first. ## See also - [Synapse Operator](./synapse-operator.mdx) — modes, reconcilers and the full flag reference - [Kubernetes Ingress & Gateway API](./kubernetes-ingress.mdx) — routing with native Kubernetes objects - [Configuration](./configuration.mdx) — every key inside `synapse.config` - [Installation Guide](./installation.mdx) — the pull secret, and every other channel --- # Installation Guide Source: https://docs.gen0sec.com/docs/client/installation # Installation Guide Synapse is distributed from Gen0Sec's own repositories. They are authenticated, so the first step is always a credential. ## Two credentials, not one Synapse needs two different secrets, issued in two different places. Conflating them is the most common reason an install stalls at the first command. | | Where | What it unlocks | |---|---|---| | **Repository credential** | Dashboard → **Clients** → **Repository Access** (`//repositoryAccess`) | Downloading the software: apt, rpm, the container registry, the release host | | **API key** | Dashboard → **API keys** (`//apiKeys`) | The running agent talking to the platform — `platform.api_key` | The repository credential is minted on the Clients page rather than a settings screen because it is shown once and used immediately by the install commands beside it. ### Repository credential scopes A repository credential carries a scope per host. A key issued for APT alone will not authenticate a `docker login`, which fails as an authentication error rather than a permissions one: | Scope | Host | |---|---| | `apt:pull` | `apt.gen0sec.com` | | `rpm:pull` | `rpm.gen0sec.com` | | `registry:pull` | `registry.gen0sec.com` | | `release:pull` | `releases.gen0sec.com` | Tick the scopes for the channels you actually install from. It is presented as HTTP Basic credentials with the credential as the **password**. The username is ignored — the examples below use `x`. :::note[One host takes Bearer instead] `releases.gen0sec.com`, which serves the tarballs and the Windows installer, accepts `Authorization: Bearer`. apt, rpm and the container registry accept Basic only. ::: ## Which build Two independent choices — a **role** and whether the **ML models** are bundled: | | | |---|---| | **Agent** | Runs on the host itself: telemetry, firewall enforcement and IDS. What most machines want. | | **Proxy** | Terminates traffic you route through it and inspects it at L7 — WAF, rate limits, bot handling. | | **`-ml`** | Adds the bundled models for on-host inference. A substantially larger download. | How that choice is expressed depends on the channel, which is the part that catches people out: | Channel | How the role is chosen | |---|---| | apt / rpm | Four packages — `synapse-agent`, `synapse-proxy`, and an `-ml` variant of each. They conflict, so one at a time. The systemd unit is named after the package. | | Binary | Not in the name. The tarball carries architecture and `-ml` only — there is no `synapse-proxy` tarball. | | Docker | One image. The role comes from `MODE`; `-ml` is a **tag** suffix, not a separate repository. | | Kubernetes | The stack chart mounts the same subchart twice, under the aliases `proxy` and `agent`. | | Windows | One installer. The role is set in the config file. | Where the artefact does not encode the role, configuration does: every package ships the same `/usr/bin/synapse`, and the proxy unit simply sets `MODE=proxy`. **Agent is the default**, so only the proxy needs stating. ## The signing key Every repository and every release artefact is signed with: ``` C1631499081A7146332D2038798FF49A6E9DA146 ``` The public key is fetched from **keyserver.ubuntu.com**, not from a Gen0Sec host. That is deliberate — trust is not bootstrapped from the same place that serves the artefact, and fetching the key needs no credential. ## Debian / Ubuntu Three parts: the credential apt authenticates with, the key it verifies the repository against, and the source itself. ```bash export GEN0SEC_KEY='' # apt reads credentials from auth.conf.d; root-only, since it holds a secret. sudo install -d -m 0755 /etc/apt/auth.conf.d printf 'machine apt.gen0sec.com login x password %s\n' "$GEN0SEC_KEY" \ | sudo tee /etc/apt/auth.conf.d/gen0sec.conf > /dev/null sudo chmod 600 /etc/apt/auth.conf.d/gen0sec.conf sudo install -d -m 0700 /root/.gnupg sudo gpg --keyserver hkp://keyserver.ubuntu.com:80 --no-default-keyring \ --keyring /usr/share/keyrings/gen0sec.gpg \ --recv-keys C1631499081A7146332D2038798FF49A6E9DA146 echo "deb [signed-by=/usr/share/keyrings/gen0sec.gpg] https://apt.gen0sec.com stable main" \ | sudo tee /etc/apt/sources.list.d/gen0sec.list > /dev/null ``` :::note[Why `install -d /root/.gnupg` is in there] gpg needs its own home to exist before it will start dirmngr. On a minimal image it does not, and `--recv-keys` fails with *No dirmngr*. ::: Then install: ```bash sudo apt-get update sudo apt-get install -y synapse-agent # or synapse-proxy, or either -ml sudo systemctl enable --now synapse-agent ``` Built for `amd64` and `arm64`. The suite is `stable` and the component `main` on every distribution — there is no per-codename repository. ## RHEL / Oracle / CentOS / Fedora Both the packages and the repository metadata are signed, so `gpgcheck` and `repo_gpgcheck` can stay on. The credential goes in `username`/`password` rather than in the URL, which keeps it out of logs and out of anything that echoes `baseurl` back. ```bash export GEN0SEC_KEY='' sudo tee /etc/yum.repos.d/gen0sec.repo > /dev/null <" ``` ```bash helm upgrade --install synapse-stack gen0sec/synapse-stack \ -n synapse-os --create-namespace \ -f synapse-values.yaml kubectl -n synapse-os rollout status deploy/synapse-proxy ``` Every chart value is documented in [Helm Charts](./helm-charts.mdx), the control loop in [Synapse Operator](./synapse-operator.mdx), and routing in [Kubernetes Ingress & Gateway API](./kubernetes-ingress.mdx). :::caution[`operator.imagePullSecrets` needs synapse-stack 0.5.1+] Earlier charts have no such field on the operator deployments, so the flag is accepted and silently ignored — and the pods sit in `ImagePullBackOff`. On an older chart, attach the secret to the ServiceAccount instead, which Kubernetes applies to every pod using it: ```bash kubectl -n synapse-os patch serviceaccount synapse-operator \ -p '{"imagePullSecrets":[{"name":"gen0sec-registry"}]}' ``` ::: ## Windows One installer, so the role is chosen in configuration rather than by the package. Run PowerShell 5+ as Administrator: ```powershell $env:GEN0SEC_KEY = "" $headers = @{ Authorization = "Bearer $env:GEN0SEC_KEY" } Invoke-WebRequest -Headers $headers -OutFile synapse-setup.exe ` -Uri https://releases.gen0sec.com/synapse/v/synapse-setup.exe .\synapse-setup.exe ``` An MSI is published alongside it for unattended rollout through Intune, GPO or similar: ```powershell Invoke-WebRequest -Headers $headers -OutFile synapse.msi ` -Uri https://releases.gen0sec.com/synapse/v/synapse.msi msiexec /i synapse.msi /qn ``` Configuration lives at `C:\ProgramData\Gen0Sec\Synapse\config.yaml`. Manage the service with `sc`: ```powershell sc start Synapse sc control Synapse paramchange # reload config without restart synapse.exe --terminal # live TUI dashboard ``` :::note[Windows specifics] XDP/eBPF filtering needs eBPF for Windows v1.1.0+ with test signing — pass `-WithEBPF` to the installer to fetch it automatically. The **ML builds are not published for Windows**: there is no `-ml` installer, so the models are unavailable on this platform. ::: ## Connect it to the platform Whichever channel you used, the running agent needs the **API key** — not the repository credential that fetched the software. It comes from the dashboard's **API keys** page: ```yaml # /etc/synapse/config.yaml platform: api_key: "" ``` Add `mode: "proxy"` above it if you are running a proxy from a channel that does not encode the role — a tarball, a container, or Windows. See [Configuration](./configuration.mdx). ## Manage the service The unit is named after the package you installed: ```bash sudo systemctl status synapse-agent sudo systemctl restart synapse-agent sudo journalctl -u synapse-agent -f ``` A healthy start logs the interfaces it attached to. If it attached to none, see the interface-selection notes in [Access Rules](../features/access-rules.mdx#where-it-runs) — the default deliberately skips virtual and CNI devices. Reload configuration without a restart: ```bash sudo systemctl kill -s HUP synapse-agent ``` Not everything is hot-reloadable — listen addresses, interfaces and firewall mode need a restart. [Configuration](./configuration.mdx#sighup-configuration-reload) lists which is which. ## System requirements | | | |---|---| | OS | Linux, kernel 4.18 or newer for XDP; Windows with eBPF for Windows 1.1.0+ | | Architecture | x86_64 / amd64 and aarch64 / arm64 | | Privileges | `CAP_BPF` and `CAP_NET_ADMIN`, or root | | BTF | `/sys/kernel/btf/vmlinux` present, for the eBPF path | Without a usable eBPF path the agent still runs, falling back to nftables and then iptables. See [Firewall Rules](../features/firewall-rules.mdx). ## Next steps - [Configuration](./configuration.mdx) — every key, and which reload without a restart - [Helm Charts](./helm-charts.mdx) — every value in the three charts - [Synapse Operator](./synapse-operator.mdx) — the Kubernetes control loop - [Kubernetes Ingress](./kubernetes-ingress.mdx) — Synapse as an Ingress and Gateway API controller - [Daemon Mode](./daemon-mode.mdx) — running without systemd --- # Per-Interface Features Source: https://docs.gen0sec.com/docs/client/interfaces # Per-Interface Features Without `network.interfaces`, every feature except egress IDS runs on every interface Synapse picks up from `network.iface` or `network.ifaces`; egress IDS has its own list, `ids.egress_ifaces`, which does nothing unless `SYNAPSE_EGRESS_IDS` is set in the environment as well. With the block, you say which links get which features. The block is **authoritative**. Each entry names the links it matches and the features they get, and **a link no entry matches is attached by nothing the block governs** — no firewall, no IDS, no smart firewall, no XDP program at all. That is the point of the design rather than a side effect of it: an operator who forgets to list the management interface must not find a deny rule on it. Two settings outside the block can still put a program on a link it never names, and one of them can drop traffic. Both are in [What the block does not cover](#what-the-block-does-not-cover), which anyone relying on the paragraph above should read. :::caution[Leaving a link out is how you protect it] Because the block is the complete list, adding it to a working config **narrows** what Synapse attaches to. A link you do not list stops being filtered. There is no "everything else" entry: `pattern: "*"` does match every link, but then any second entry matches one of those links too and the file is refused for the double match. To record a link and leave it alone, give it `features: []`. ::: This page describes Synapse **0.8.8**. The block is Linux-only — see [Linux only](#linux-only) — and changes to it take effect at startup, so an edit plus a `SIGHUP` does not apply it. Restart the agent. ## The shape ```yaml network: interfaces: wan: match: { name: eth1 } features: [firewall, ids, smart_firewall, fingerprint] lan: match: { pattern: "enp3s*" } features: [firewall] oob: match: { mac: "52:54:00:12:34:56" } features: [] # listed, nothing enabled ``` The keys under `interfaces:` are **logical names** you choose — `wan`, `lan`, `oob` above. They appear in the startup log and in error messages, and they are not interface names. Each entry takes three keys: | Key | Required | Default | What it does | |---|---|---|---| | `match` | yes | — | which links this entry applies to: exactly one of `name`, `mac` or `pattern` | | `features` | no | `[]` | the features enabled on the matched links | | `require_native` | no | `false` | refuse to attach rather than fall back to generic XDP | A commented-out example ships in `config/config.yaml`, under the `network:` section. ## Matching a link `match` takes **exactly one** of three keys. A `match` with none, or with two, fails the load. | Key | Form | Matches | |---|---|---| | `name` | `eth1` | the link with exactly this kernel name | | `mac` | `52:54:00:12:34:56` | the link with this hardware address, case-insensitive | | `pattern` | `"enp3s*"` | every link whose name matches this glob — `*`, `?` and `[…]` | Interface names are not stable: udev renames them, predictable naming changes them between distributions, and swapping a NIC moves them. That is why `mac` and `pattern` exist, and why a config meant to survive hardware changes should prefer them to `name`. A `pattern` entry covers **every** link it matches, and they all get that entry's features. A link reporting no hardware address, or an all-zero one — point-to-point and tunnel links do — never matches a `mac`. Three matching rules decide whether the file loads: - a `name` or `mac` that matches **no link on this host** fails the load. It would otherwise silently protect nothing. - a `pattern` that matches nothing is **accepted in silence**. A link matching it may appear later, so it is deliberately not an error — but nothing is printed either, and a pattern that matches nothing looks exactly like one that matches everything you meant. The resolve line below is the only way to tell them apart. - a link matched by **two** entries fails the load, naming both. Each link belongs to one entry, so that its feature set is unambiguous. ## The features Five features can be scoped to a link. **Four** of them have a global switch that is checked: listing one whose switch is off is refused rather than accepted, since it would otherwise parse, log as enabled and do nothing. `fingerprint` is the exception — it is checked against nothing, and what it governs is narrower than its name suggests. | Feature | What it enables on the link | Global switch | If the switch is off | |---|---|---|---| | `firewall` | firewall rules and access-rule IP filtering | `firewall.mode` | load fails when the mode is `none` | | `ids` | IDS on ingress traffic | `ids.enabled` | load fails | | `ids_egress` | IDS on egress traffic, via TC-egress capture | `ids.enabled`, and a build carrying the `egress-ids` feature | load fails; a build without it fails at startup | | `smart_firewall` | the smart firewall's pre-block and reactor | `firewall.amygdala.enabled` | load fails | | `fingerprint` | which link the TC collectors (JA4S, JA4TS, JA4L, JA4SSH) and the single-interface capture run on | `capture.enabled` | **nothing is said**, and those collectors do not run — but read the caution below before concluding the link yields no fingerprints | :::caution[`fingerprint` does not mean "the fingerprints from this link"] Two things follow from it not being checked. The first is the ordinary one: `features: [fingerprint]` with `capture.enabled: false` loads cleanly and logs the feature as enabled, while the TC collectors it selects are skipped — as they are during a PCAP replay. The second matters more, because it runs the other way. **JA4T does not come from this feature at all.** Those fingerprints are read from the XDP program on *every* attached link and written out whenever `logging.eventbridge_log.enabled` is true, which is its default. So a link listed `features: [firewall]` with no `fingerprint` still produces JA4T, and turning `capture.enabled` off does not stop it. The switch for that is `logging.eventbridge_log.enabled`. The shipped `config.yaml` comment names `firewall.mode`, `ids.enabled` and `firewall.amygdala.enabled` — neither of the two switches above. ::: Two more features exist in the schema and are **refused** under an interface, because each runs host-wide and listing it on one link would read as "only here": | Feature | Why it cannot be scoped | Where it is controlled | |---|---|---| | `ssl_uprobe` | it hooks a library, not a link | `capture.enabled` | | `tls_offload` | it sits behind the proxy, after TLS | the IDS post-TLS offload settings | Two more rules on the set itself: a feature listed twice fails the load, and `smart_firewall` requires `firewall` **on the same interface** — it enforces through the firewall datapath and registers the interface for rules on its own, so without `firewall` it would install rules on an interface that asked for none. :::note[The smart firewall runs on exactly one interface] Listing `smart_firewall` on two or more fails at startup, naming them. If `firewall.amygdala.iface` is also set and names a different link, that fails too — drop the key and let the block decide. `firewall.amygdala.backend` must be `auto` or `xdp`; the others are host-wide. ::: ## What it replaces, and what it refuses The block does not merge with the older interface keys, it **replaces** them. Three combinations fail the load: | Also set | Why it is refused | |---|---| | `network.iface` (other than `auto`) or `network.ifaces`, including `--iface`, `--ifaces`, `NETWORK_IFACE` and `NETWORK_IFACES` | two lists of interfaces, one of them authoritative. List every interface under `network.interfaces` instead | | `ids.egress_ifaces` | superseded by the `ids_egress` feature, which names the links in one place | | `firewall.mode: nftables` or `iptables` | those backends install host-wide chains and cannot yet be limited to the listed links, so only `auto` and `xdp` are accepted beside the block. (`none` loads, as long as no entry lists `firewall`) | ## require_native An interface whose driver has no native XDP support can still be attached in generic (SKB) mode, which works but costs performance. `require_native: true` refuses to attach to such a link rather than running there degraded. It is per-entry, so one uplink can insist on native XDP while a secondary link accepts generic. Know what the refusal costs before setting it. Synapse logs an error, **skips that interface and keeps running** — it is not a startup failure. The link then gets no firewall, no ingress IDS and no XDP program, which is the protection it was listed for. On an uplink that is the difference between a slower uplink and an unprotected one. It is not quite an unlisted link, though. The TC-attached collectors go by what the configuration *planned* rather than by what actually attached, so they can still land on a refused link: the JA4 collectors when no interface attached XDP at all, and `ids_egress` always. ## Declaration order and the limit The block is read in **declaration order** rather than sorted, and each entry is assigned a slot index from its position. A configuration may declare at most **64** logical interfaces, and declaring more fails the load with the count. :::note[Reordering entries changes nothing today] The slot is designed to be what scoped kernel rules key on, and the module describes it that way, but at 0.8.8 nothing in the datapath reads it — the scoped IDS and firewall paths key on the kernel's ifindex instead. So reordering the block is currently inert. The 64-entry limit is enforced regardless. ::: ## What a bad file does | The file says | What happens | |---|---| | `match: {}` | load fails: `match` needs one of `name`, `mac` or `pattern` | | two of `name`/`mac`/`pattern` | load fails: `match` takes exactly one of the three | | an unknown key under `match`, such as `ifname` | load fails on the unknown field | | `match: { name: "" }` or `pattern: ""` | load fails; neither may be empty | | a malformed MAC — wrong length, non-hex, seven groups | load fails naming the expected `aa:bb:cc:dd:ee:ff` form | | an invalid glob, such as `[` | load fails naming the pattern | | `features: [waf]`, or any name not in the five | load fails on the unknown feature | | `features: [ssl_uprobe]` or `[tls_offload]` | load fails: host-wide, and the message names where it is controlled | | `features: [firewall, firewall]` | load fails: listed twice | | `features: [smart_firewall]` without `firewall` | load fails | | an empty logical name | load fails | | more than 64 entries | load fails with the count | | `name` or `mac` matching no link on this host | load fails | | `pattern` matching no link | accepted, with **no startup line at all** | | one link matched by two entries | load fails, naming both entries | | the block together with `iface`/`ifaces`, `ids.egress_ifaces`, or a netfilter `firewall.mode` | load fails, as above | | `smart_firewall` on two interfaces | startup fails, naming them | | `ids_egress` on a build without the `egress-ids` feature | startup fails | | the block on a host with no `/sys/class/net` | load fails: `cannot enumerate /sys/class/net` | Every link that does resolve is logged at startup, as `interface 'eth1' -> network.interfaces.wan (features: firewall, ids)`. That line is the quickest check that a `pattern` matched what you meant. ## What stays quiet These do not fail the load or the start, and each has surprised someone: - **`fingerprint` without `capture.enabled`**, and JA4T without `fingerprint`, as above. - **Packet capture may not start at all.** Under the block, a single-interface capture runs on a `fingerprint` interface or not at all — never on a default such as `auto` or `eth0`, either of which could be a link the config deliberately left out. If `capture.interface` names a link with no `fingerprint` feature, or no entry has `fingerprint`, capture is skipped and the reason is logged at info level as `packet capture not started: …` — and only when `capture.enabled` is true. Left unset or `auto`, capture takes the first `fingerprint` interface once physical links have been preferred over virtual ones. - **Only one `ids` link is covered if the IDS falls back to AF_PACKET.** When the XDP IDS is unavailable, the fallback binds to the **first** `ids` interface and logs an error naming the ones it is not covering. Scoping `ids` to three NICs does not mean three are inspected on that path. - **An unavailable XDP is a degraded start, not a failure.** With the block set, `firewall.mode: auto` and no XDP program loaded, Synapse logs an error and does **not** fall back to nftables or iptables — those are the host-wide backends the block cannot scope. It continues with no kernel firewall at all. ## What the block does not cover The guarantee at the top of this page is about the features the block governs. Three things reach an interface by another route, and the first two can reach a link the block never names: - **`capture.observe_interfaces` and `capture.observe_egress_interfaces`.** These are read straight from the configuration; the block is not consulted. The named link gets a BPF program attached, or a capture socket opened, depending on `capture.observe_mode`. And despite the name it is not purely passive: on a smart-firewall `drop` verdict for an overlay flow it has resolved — pod-to-pod identity, or a destination matched from a snooped domain — the observer drops the packet. A link that appears only here can therefore carry a program and lose traffic without being in `network.interfaces` at all. - **`firewall.urpf.interfaces`.** Its keys are kernel interface names and are never cross-checked against the block. uRPF is applied only to links that have an XDP dispatcher, which is every link the block attaches — so it reaches a link whose only feature is `fingerprint`, with no `firewall` requested, at the global `firewall.urpf.mode`, because a name absent from the map falls back to it. A key naming a link the block does not list is simply never applied, and **no line at any level says so** — not even at debug. - **A `require_native` refusal**, which leaves the named link without the protection it was listed for while the TC collectors may still use it — see [require_native](#require_native). If the management interface must stay untouched, leaving it out of `network.interfaces` is necessary and not sufficient: check that it is named in neither `capture.observe_*` nor `firewall.urpf.interfaces`. ## veth, bridges and auto-detection `network.iface: auto` enumerates the host's links and drops the ones that are virtual or not up — loopback, `veth`, bridges and CNI-style prefixes among them. **Under `network.interfaces` that detection never runs**, so none of its exclusions apply: the block's matchers are the only thing that selects a link. One exclusion survives, on the attach path rather than in detection. A `veth` link is skipped unless `firewall.attach_veth` is set — or the block matched it, which is read as the more specific request and attaches it. Matching by `pattern` counts as matching it. :::note[An interface with only `ids_egress` gets no XDP program] The XDP entry program is attached for `firewall`, `ids`, `smart_firewall` and `fingerprint`. TC-egress capture does not need it, so an entry whose only feature is `ids_egress` is in the plan and carries no XDP. ::: ## Linux only `network.interfaces` resolves its matchers by reading `/sys/class/net`, and that read is not conditional on the platform. On Windows the configuration therefore **fails to load** with `network.interfaces: cannot enumerate /sys/class/net`. Use `network.iface` or `network.ifaces` there; the shipped `config_windows.yaml` does. (The exception is `synapse --terminal` run standalone, which ignores a configuration it cannot load and starts on defaults. Every other start path propagates the error.) The features themselves need a Linux build with the `bpf` feature, which is on by default. :::caution[`ids_egress` cannot be used on a released package] It needs the `egress-ids` build feature, and that feature exists only on the inner `synapse-app` crate. There is no root-level one, so a plain `--features egress-ids` is rejected and only the dependency form `--features synapse-app/egress-ids` enables it — and no packaging job passes either. On every published build, `features: [ids_egress]` therefore makes the agent **refuse to start**. Treat it as unavailable unless you build Synapse yourself. ::: ## A worked example A host with a public uplink, two internal NICs and a management port, where the management port must never be touched: ```yaml network: interfaces: uplink: match: { mac: "52:54:00:12:34:56" } # survives a NIC swap features: [firewall, ids, smart_firewall, fingerprint] require_native: true # refuse generic XDP — see the caveat above internal: match: { pattern: "enp3s*" } features: [firewall] firewall: mode: auto amygdala: enabled: true ids: enabled: true capture: enabled: true ``` The management port appears nowhere, which is most of what keeps it clear — the rest is checking it is named in neither `capture.observe_*` nor `firewall.urpf.interfaces`. Note what the file has to get right for the features above to run: `firewall.mode` is `auto` rather than a netfilter backend, and `ids.enabled` and `firewall.amygdala.enabled` are true because `ids` and `smart_firewall` are listed — those three are checked, and the file would be refused without them. `capture.enabled` is true because `fingerprint` is listed, and that one is **not** checked: with it false the file still loads and the TC collectors simply do not run. --- # JA4+ Fingerprinting Source: https://docs.gen0sec.com/docs/client/ja4-plus # JA4+ Fingerprinting Synapse implements the complete JA4+ fingerprinting suite for advanced threat detection and behavioral analysis. ## Implementation Status | Component | Status | Description | |-----------|--------|-------------| | **JA4** | ✅ Complete | TLS ClientHello fingerprinting | | **JA4T** | ✅ Complete | TCP SYN options fingerprinting | | **JA4H** | ✅ Complete | HTTP header fingerprinting | | **JA4L** | ✅ Complete | Latency/RTT measurement | | **JA4S** | ✅ Complete | TLS ServerHello fingerprinting | | **JA4X** | ✅ Complete | X.509 certificate fingerprinting | ## JA4 - TLS Client Fingerprint **Format:** `{proto}{version}{sni}{cipher_count}{ext_count}{alpn}_{cipher_hash}_{ext_hash}` **Example:** `t13d1516h2_8daaf6152771_b186095e22b6` ### What it Detects - TLS version negotiation - Cipher suite preferences - TLS extensions used - ALPN protocols - SNI behavior ### Use Cases - Identify bot traffic - Detect TLS library tampering - Track client applications - Spot impersonation attempts ## JA4T - TCP Fingerprint **Format:** `{window_size}_{tcp_options}_{mss}_{window_scale}` **Example:** `65535_2-4-8-1-3_1460_7` ### Components - **Window size**: TCP window size (65535) - **TCP options**: Ordered list of TCP option kinds (2=MSS, 4=SACK, 8=Timestamps, 1=NOP, 3=Window Scale) - **MSS**: Maximum segment size (1460) - **Window scale**: Window scale factor (7) ### What it Detects - Operating system fingerprinting - Network stack behavior - Client device type - NAT/proxy presence ### Use Cases - Identify botnet traffic - Detect OS spoofing - Track device types - Analyze network topology ## JA4H - HTTP Header Fingerprint **Format:** `{method}{version}{cookie}{referer}{count}{lang}_{headers_hash}_{cookie_names_hash}_{cookie_values_hash}` **Example:** `ge11cr15enus_a1b2c3d4e5f6_123456789abc_def012345678` ### Components - **method**: First 2 chars of HTTP method (`ge` for GET) - **version**: HTTP version (`11` for HTTP/1.1) - **cookie**: `c` if Cookie header present, `n` otherwise - **referer**: `r` if Referer header present, `n` otherwise - **count**: Number of headers (excluding Cookie/Referer) - **lang**: Language code (`enus` for en-US) - Three 12-char SHA256 hashes ### What it Detects - Browser fingerprinting - User-agent spoofing - Header manipulation - Automation tools ### Use Cases - Detect scrapers - Identify bots - Track browser versions - Spot header tampering ## JA4L - Latency Fingerprint **Format:** `{rtt_microseconds}_{ttl}` **Example:** `12500_64` (12.5ms RTT, TTL 64) ### Components - **Client**: `(ACK - SYNACK) / 2` - **Server**: `(SYNACK - SYN) / 2` ### What it Detects - Network distance - Geographic location estimation - Proxy/VPN detection - CDN usage ### Use Cases - Detect proxy chains - Estimate geographic location - Identify VPN usage - Analyze network path ## JA4S - TLS Server Response Fingerprint **Format:** `{proto}{version}{ext_count}{alpn}_{cipher}_{extensions_hash}` **Example:** `t1302h2_1301_a56c5b993250` ### Components - **proto**: `t` for TCP, `q` for QUIC - **version**: TLS version (`13` for TLS 1.3) - **ext_count**: Number of extensions (02) - **alpn**: ALPN protocol (`h2` for HTTP/2) - **cipher**: Cipher suite (1301 = TLS_AES_128_GCM_SHA256) - 12-char extension hash ### What it Detects - Server TLS configuration - Backend infrastructure - Load balancer behavior - CDN fingerprinting ### Use Cases - Map backend infrastructure - Detect server changes - Track TLS configuration - Identify load balancers ## JA4X - X.509 Certificate Fingerprint **Format:** `{issuer_hash}_{subject_hash}_{extensions_hash}` **Example:** `aae71e8db6d7_b186095e22b6_c1a4f9e7d8b3` ### Components Three 12-char SHA256 hashes: - **Issuer**: Issuer RDN OIDs - **Subject**: Subject RDN OIDs - **Extensions**: Certificate extension OIDs ### What it Detects - Certificate authority patterns - Certificate structure - Organization patterns - Certificate types ### Use Cases - Track certificate issuers - Detect certificate changes - Identify certificate types - Map PKI structure ## Where the fingerprints are written Two files, with different shapes. The fingerprint stream is on by default and an agent and a proxy both write it. The collectors that poll the BPF maps need a loaded program as well as the switch, but they are not the only producer: the Linux AF_PACKET capture path and the Windows NDIS path write fingerprints with no BPF at all. The access log is a proxy file, and `logging.file_logging_enabled: true` is what puts it on disk; with file logging off the same record goes to the console, raw, one JSON object per line. `{log_directory}/eventbridge.log` - one event per line, fingerprints at the top level: ```json { "event_type": "fingerprint", "timestamp": "2025-10-29T12:34:56.789Z", "src_ip": "192.168.1.100", "src_port": 54321, "dst_ip": "10.0.0.5", "dst_port": 8443, "ja4": "t13d1516h2_8daaf6152771_b186095e22b6", "ja4t": "65535_2-4-8-1-3_1460_7", "ja4h": "ge11cr15enus_a1b2c3d4e5f6_123456789abc_def012345678" } ``` `{log_directory}/access.log` - one request per line, fingerprints under `fingerprints`: ```json { "event_type": "http_access_log", "timestamp": "2025-10-29T12:34:56.789Z", "http": { "method": "GET", "path": "/api/data" }, "network": { "src_ip": "192.168.1.100", "src_port": 54321 }, "fingerprints": { "ja4": "t13d1516h2_8daaf6152771_b186095e22b6", "ja4t": "65535_2-4-8-1-3_1460_7", "ja4h": "ge11cr15enus_a1b2c3d4e5f6_123456789abc_def012345678", "ja4s": "t1302h2_1301_a56c5b993250", "ja4x": "aae71e8db6d7_b186095e22b6_c1a4f9e7d8b3" } } ``` Neither file has a `client_ip`, a `tls.ja4` or an `http.ja4h` field. Earlier revisions of this page showed all three; a `jq` filter naming one prints `null` rather than failing. Both records omit any fingerprint that was not observed. ## Configuration One unified log carries every fingerprint type and the BPF statistics, and it is on by default: ```yaml # /etc/synapse/config.yaml logging: eventbridge_log: enabled: true # the default # file: /var/log/synapse/eventbridge.log ``` `fingerprint_log` is an accepted alias for `eventbridge_log`. With `file` unset the path is `{log_directory}/eventbridge.log` — not `fingerprints.log`, which is the stream's old name and names no file anything writes. :::caution[`tcp_fingerprint` and the `TCP_FINGERPRINT_*` variables are gone] This page used to show a `logging.tcp_fingerprint` block as "kept for backward compatibility". It is not kept: the section is **absent from the 0.8.8 schema**, along with the other five per-type sections, and setting it has no effect beyond the `Unused config options in ` warning at startup. The six `TCP_FINGERPRINT_*` environment variables this page used to list have no reader anywhere in the code either. The stream's one environment variable is `LOGGING_EVENTBRIDGE_LOG_ENABLED`, which takes `true` or `false` and overrides the key above. ::: ## Analyzing Fingerprints ### Extract Unique Fingerprints ```bash EB=/var/log/synapse/eventbridge.log # Unique JA4 values, most frequent first jq -r 'select(.event_type == "fingerprint" and .ja4) | .ja4' "$EB" | sort | uniq -c | sort -rn # Unique JA4T values jq -r 'select(.event_type == "fingerprint" and .ja4t) | .ja4t' "$EB" | sort | uniq -c | sort -rn # Unique JA4H values jq -r 'select(.event_type == "fingerprint" and .ja4h) | .ja4h' "$EB" | sort | uniq -c | sort -rn # Correlate source address with fingerprints jq -r 'select(.event_type == "fingerprint") | [.src_ip, .ja4t // "-", .ja4h // "-"] | @tsv' "$EB" ``` :::danger[There is no fingerprint statistics endpoint] This page used to document `GET /health/tcp_fingerprint_stats` and the JSON body it returned. **No such route exists**, and none ever shipped: `tcp_fingerprint_stats` appears in no source file and `/metrics` carries no fingerprint metric, so the request answers `404`. The aggregate itself is real - synapse logs it to the console at debug level rather than serving it. See [Counting Fingerprints](./testing#counting-fingerprints) for that command, for the counts from `eventbridge.log` and for the BPF maps. ::: ## Use Cases ### Bot Detection Combine multiple fingerprints to detect bot traffic: ```bash # Group the clients that share a JA4 jq 'select(.event_type == "fingerprint" and .ja4 and .ja4h) | {ip: .src_ip, ja4, ja4h}' /var/log/synapse/eventbridge.log | jq -s 'group_by(.ja4) | map({ja4: .[0].ja4, count: length, ips: map(.ip) | unique})' ``` ### OS Fingerprinting Analyze TCP options to identify operating systems: ```bash # JA4T is {window_size}_{tcp_options}_{mss}_{window_scale} # Print window size, option order and MSS per source address jq -r 'select(.event_type == "fingerprint" and .ja4t) | [.src_ip, (.ja4t | split("_")[0]), (.ja4t | split("_")[1]), (.ja4t | split("_")[2])] | @tsv' \ /var/log/synapse/eventbridge.log | sort -u ``` A JA4T value carries no TTL, so the TTL-based rules of thumb this page used to print (`Linux: TTL=64`, `Windows: TTL=128`) cannot be applied to it. Discriminate on the option order and the MSS: the window size alone does not separate the common stacks, which is why the old table gave `Window=65535` for all three. ### Threat Detection Identify anomalies in fingerprint patterns: ```bash # Find rare fingerprints jq -r 'select(.event_type == "fingerprint" and .ja4) | .ja4' \ /var/log/synapse/eventbridge.log | sort | uniq -c | sort -n | head -20 ``` ## References - [Official JA4+ Repository](https://github.com/FoxIO-LLC/ja4) - [Technical Details](https://github.com/FoxIO-LLC/ja4/tree/main/technical_details) - [Zeek Implementation](https://github.com/FoxIO-LLC/ja4/tree/main/zeek) ## License - **JA4** (TLS Client) - BSD 3-Clause License - **JA4+** methods (JA4S, JA4H, JA4L, JA4T, JA4X) - FoxIO License 1.1 See [JA4 License](https://github.com/FoxIO-LLC/ja4/blob/main/LICENSE) for details. ## Next Steps - [Test JA4+ Features](./testing) - Validate fingerprinting - [Configuration Reference](./configuration) - Detailed settings --- # Kubernetes Ingress & Gateway API Source: https://docs.gen0sec.com/docs/client/kubernetes-ingress # Kubernetes Ingress & Gateway API Synapse can act as a native Kubernetes **Ingress controller** and **Gateway API** implementation. The `synapse-operator` watches your `Ingress`, `IngressClass`, `Gateway`, and `HTTPRoute` resources and programs the Synapse reverse proxy in real time — so Synapse becomes a drop-in replacement for Traefik / ingress-nginx while keeping its eBPF firewall, WAF, and JA4+ fingerprinting on the same data path. It integrates with **cert-manager** for automatic TLS, serves **many domains from one proxy via SNI**, and picks up new or rotated certificates **without a restart**. :::note This page covers the controller. To install the Synapse stack itself, see [Installation](./installation.mdx). ::: ## How it works `synapse-operator` runs in **ingress mode** alongside the Synapse proxy. On every change to the resources it owns it re-renders Synapse's upstream configuration and triggers a hot reload — no proxy restart, no dropped connections. - **Ingress + Gateway API** are reconciled into the same routing table, so you can mix both. - Backends are addressed by in-cluster DNS (`svc.namespace.svc.cluster.local:port`); named Service ports are resolved automatically and a Service change re-renders. - Routing is **deterministic**: sources are processed in a stable order (Ingresses before HTTPRoutes, each sorted by namespace/name) with first-writer-wins on a host+path conflict, so the result is reproducible regardless of watch ordering. ## Deploy the controller Run the operator in ingress mode (it co-locates with the proxy and writes the proxy's upstreams + certificates): ``` --ingress-mode --gateway-api # also reconcile Gateway API (optional) --ingress-class=synapse # spec.ingressClassName this controller serves --certs-out=/certs # project referenced TLS Secrets here (optional) --status-leader-election # see "Scaling" below (multi-replica) ``` Create the `IngressClass`: ```yaml apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: synapse # Optional: make Synapse the default for class-less Ingresses annotations: ingressclass.kubernetes.io/is-default-class: "true" spec: controller: gen0sec.com/synapse ``` The controller needs RBAC to `get/list/watch` ingresses, ingressclasses, services, secrets (and the Gateway API resources when `--gateway-api` is set), plus `update` on their `/status`, `create/patch` on events, and `coordination.k8s.io/leases` when leader election is enabled. ## Routing with Ingress ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: app spec: ingressClassName: synapse rules: - host: app.example.com http: paths: - path: / pathType: Prefix backend: service: name: app port: number: 80 ``` **Class selection** follows Kubernetes precedence: an explicit `spec.ingressClassName` wins; otherwise the legacy `kubernetes.io/ingress.class` annotation; otherwise the default `IngressClass` (when annotated `is-default-class: "true"`). :::note `Exact` path types and HTTPRoute header/method/query matches are approximated or ignored (Synapse routes on host + longest-prefix path); the controller logs a warning and emits a Kubernetes event so the behaviour is never silent. ::: ### Upstream settings via annotations Annotations on the `Ingress` (or `HTTPRoute`) tune the upstream. The `synapse.gen0sec.com/` prefix is canonical; a small `nginx.ingress.kubernetes.io/` compatibility subset is also accepted. | Annotation | Effect | |---|---| | `backend-protocol: HTTP\|HTTPS` | TLS to the upstream | | `http2: "true"` | HTTP/2 to the upstream | | `force-https` / `ssl-redirect: "true"` | redirect HTTP→HTTPS | | `connect-timeout` / `read-timeout` / `write-timeout` / `idle-timeout` | seconds | | `healthcheck: "true"` | upstream health checking | | `disable-access-log: "true"` | drop access logs for this route | | `request-headers` / `response-headers` | comma-separated `Name: value` injection | | `sticky-sessions: "true"` | sticky upstream selection | ## Routing with Gateway API Create a `GatewayClass` whose `controllerName` is `gen0sec.com/synapse`, a `Gateway`, and `HTTPRoute`s bound to it: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: app spec: parentRefs: - name: synapse-gw hostnames: ["app.example.com"] rules: - matches: - path: { type: PathPrefix, value: / } filters: - type: RequestHeaderModifier requestHeaderModifier: set: [{ name: X-From, value: gateway }] backendRefs: - { name: app-v1, port: 80, weight: 90 } - { name: app-v2, port: 80, weight: 10 } ``` - **Weighted `backendRefs`** are honored, including Gateway API semantics where `weight: 0` means *receive no traffic* (the backend is excluded). - `RequestHeaderModifier` / `ResponseHeaderModifier` filters are translated to header injection. - The controller maintains `GatewayClass`, `Gateway`, and `HTTPRoute` status conditions (`Accepted`, `Programmed`, `ResolvedRefs`). ## Smart firewall, WAF & IDS on ingress traffic Because Synapse *is* the proxy, its eBPF **smart firewall**, **WAF**, and signature **IDS/IPS** run on the very same data path as the Ingress/Gateway traffic — no sidecar, no second hop. These are configured on the **Synapse side** (config file + rules), not via Ingress annotations (the controller annotations above only tune the upstream); they then apply to every route the controller programs. ### Smart firewall (eBPF) Set in the Synapse config: ```yaml firewall: mode: "auto" # auto | xdp | nftables | iptables | none ``` `mode` is the only switch. To stop the kernel firewall evaluating rules, set `mode: "none"` — there is no `firewall.enabled` and no `firewall.default_action`, and setting either one is silently ignored. Kernel firewall rules are evaluated in ascending `order`, first match wins (`action`: allow/drop), matching on `src_ip`/`dst_ip` (+`src_ip_prefix`/`dst_ip_prefix`), `src_*_port`/`dst_*_port`, `ip_version`, `ip_protocol`. The JA4/JA4T/JA4H smart-firewall plus threat scoring and ML classification also evaluate ingress clients on this path. See [Firewall Rules](../features/firewall-rules.mdx), [Threat Detection](../features/threat-detection.mdx), and [Machine Learning Models](../features/machine-learning.mdx). ### WAF The WAF inspects ingress HTTP requests using Wirefilter-compatible expression rules (Cloudflare Ruleset-Engine style) with OWASP Top 10 coverage — e.g. matching on `http.request.method`, `http.request.host`, `http.request.path`, `http.request.uri`, `http.request.user_agent`, with `and`/`or`/`not`, `contains`, `matches`, `in`, etc., and an action on match. Rules are managed centrally (per organization), not per-Ingress. See [WAF](../features/waf.mdx) and [Access Rules](../features/access-rules.mdx). ### Intrusion detection (IDS / IPS) Synapse runs a signature IDS/IPS engine (thalamus) that inspects **decrypted, post-TLS HTTP** on the same path as ingress traffic — so it sees real requests even for HTTPS that Synapse terminates. Configured under `ids:` in the Synapse config: ```yaml ids: enabled: true rule_paths: # signature rule files (Suricata / Emerging Threats compatible) - /etc/synapse/rules/emerging-all.rules address_vars: { HOME_NET: "10.0.0.0/8" } # rule $VARS port_vars: { HTTP_PORTS: "80,8080" } enforce_block: false # false = IDS (detect/alert); true = IPS (drop on match) post_tls: mode: inline # inline | offload | off socket: /run/synapse/ids.sock # offload: per-node consumer UDS # flow/capture tuning: snaplen, poll_timeout_ms, flow_timeout_secs, # max_flows, cleanup_interval_secs, stats_log_interval_secs ``` - **`enabled`** turns the engine on. **`rule_paths`** point at signature files; **`address_vars`/`port_vars`** populate rule variables (e.g. `$HOME_NET`, `$HTTP_PORTS`). - **`enforce_block`**: `false` = detect & alert (IDS); `true` = block matched requests (IPS). - **`post_tls.mode`**: `inline` runs the engine in every proxy process (simple, but a horizontally-scaled proxy holds the ruleset in every replica — O(replicas)); `offload` keeps the proxy light and taps decrypted HTTP to an out-of-process per-node consumer over `post_tls.socket` (ruleset cost O(nodes)); `off` disables post-TLS inspection. With `>1` replica, prefer `offload`. - Advanced: `post_tls.enforce` (default off) additionally bans the proxy-observed source IP in the agent firewall on a block detection — guarded against shared/NAT IPs; distinct from `enforce_block`. See [Architecture overview](../architecture/overview.mdx) and [Data flow](../architecture/data-flow.mdx) for where IDS/IPS sits, and [Threat Feeds API](../api/threat-feeds-api.mdx) to feed blocklists. ### Rate limiting The Synapse upstreams routing config (which the controller renders) supports a `global_rate_limit` and per-host / per-path `rate_limit`; these bound request rates on the programmed ingress routes. See [Configuration](./configuration.mdx). :::note Firewall mode, firewall/WAF/IDS rules, and rate limits are Synapse configuration — they are **not** expressed as Ingress annotations. The controller renders routing/upstreams; the firewall, WAF, and IDS apply globally to the proxy and therefore to all ingress traffic. ::: ## TLS with cert-manager ### Automatic certificates (HTTP-01) Annotate the Ingress for cert-manager and declare `spec.tls`. The HTTP-01 challenge is served straight through Synapse: ```yaml metadata: annotations: cert-manager.io/cluster-issuer: letsencrypt-prod spec: ingressClassName: synapse tls: - hosts: [app.example.com] secretName: app-tls rules: - host: app.example.com http: paths: - { path: /, pathType: Prefix, backend: { service: { name: app, port: { number: 80 } } } } ``` ### Multi-domain TLS via SNI, no restart When `--certs-out` is set, the controller projects every referenced TLS Secret (Ingress `spec.tls`, Gateway listener `certificateRefs`) into Synapse's certificate directory. Synapse then selects the right certificate **per SNI hostname**, so a single proxy serves many domains. Adding, rotating (cert-manager renewal), or removing a certificate is picked up **live — no pod restart**; an unreferenced certificate is pruned and that hostname falls back to the default certificate. :::tip This works per-pod and is never gated by leader election, so every replica serves correct certificates immediately. ::: ## Scaling (multiple replicas) Every proxy replica renders its **own** local config and reloads its **own** Synapse — that is never gated. To avoid replicas racing on shared cluster status, enable `--status-leader-election`: only the lease holder writes `Gateway`/`HTTPRoute`/Ingress status, and leadership fails over automatically if the holder dies. ## Observability - **Kubernetes events** on the Ingress/HTTPRoute: `Programmed`, `RouteConflict`, `BackendUnresolved`, `UnsupportedMatch`. - **Prometheus metrics** on the metrics endpoint: renders, changes, reload signals, route conflicts, unresolved backends, projected certs, hosts/routes gauges, and a readiness gauge. - **Readiness** is gated on the first successful render, so the proxy is not advertised ready before it has correct upstreams. ## Migrating from Traefik 1. Deploy `synapse-operator` in ingress mode and create the `synapse` `IngressClass`. 2. Repoint Ingresses by setting `spec.ingressClassName: synapse` (or make Synapse the default class). 3. Move Traefik-specific annotations to the `synapse.gen0sec.com/` equivalents above. 4. cert-manager `Certificate`/`Issuer` resources are unchanged — HTTP-01 via the ingress solver works as-is. 5. Remove Traefik once traffic is verified on Synapse. --- # Synapse Operator Source: https://docs.gen0sec.com/docs/client/synapse-operator # Synapse Operator The operator is a [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) controller that keeps Synapse in step with the cluster around it. It does the jobs that are awkward to do by hand: rolling pods when their config content changes, turning Ingress objects into Synapse routes, and filling in configuration that can only be known at runtime. It is a plain Deployment with no CRDs of its own. It works off native Kubernetes objects — ConfigMaps, Secrets, Ingresses, Pods, NetworkPolicies — and off the labels the Helm charts already apply. Deploying it is covered in [Helm Charts](./helm-charts.mdx). This page is what it does and how to configure it. ## One primary mode, plus add-ons The operator picks **exactly one** primary mode per process, and then layers any number of optional reconcilers on top. | | Config-sync | Ingress / Gateway API | |---|---|---| | Flag | *(default)* | `--ingress-mode` | | Watches | ConfigMaps and Secrets, by label | `Ingress`, plus `HTTPRoute` with `--gateway-api` | | Does | Stamps a config hash on the workload, so Kubernetes rolls it | Renders `upstreams.yaml`, then hot-reloads Synapse | | TLS Secret projection | — | Yes, via `--certs-out` | | Publishes LB status on Ingress | — | Yes, via `--publish-status-address` | The two are mutually exclusive **within a process** — ingress mode replaces config-sync rather than adding to it. To run both, run two Deployments. The `synapse-stack` chart does exactly that when you set `operator.ingress.enabled`. The four reconcilers below are different: each is an independent switch that composes with whichever primary mode is running. ## Config-sync The default. Kubernetes does not restart a pod when a mounted ConfigMap's *content* changes — the file updates in place and the process never notices. The operator closes that gap: it hashes every ConfigMap and Secret matching its label selector, and stamps the hash onto the workload's pod template. A changed hash is a changed pod spec, so Kubernetes rolls it. ``` synapse.gen0sec.com/config-hash: ``` Two flags make this less blunt than it sounds: - **`--per-workload-config-hash`** stamps each workload with a hash of only the config *it* references, resolved through volumes, `envFrom` and `env`. Without it, one combined hash covers every labelled source in the namespace, so changing an agent rule file rolls the proxy too. A workload referencing no labelled source is left alone. - **`--ids-hot-reload-hash-exclude`** leaves the hot-reloadable IDS fields out of the hash entirely — `ids.address_vars`, `enforce_block`, `rule_paths`, `port_vars`, `flow_timeout_secs`, `max_flows`. The agent reloads those in process, so a change to only those does not need a rollout. Other `config.yaml` changes still roll. :::caution[`--ids-hot-reload-hash-exclude` assumes every agent can hot-reload] It is only safe once all agents run a Synapse image that reloads those fields in place. On an older image the field changes in the ConfigMap, no rollout happens, and the agent keeps running the old value indefinitely — silently. ::: `upstreams.yaml` is excluded from the hash by default (`--ignore-configmap-keys`). That is deliberate: in ingress mode the operator itself rewrites that key, and hashing it would make the operator roll the dataplane in response to its own writes. ## Ingress and Gateway API mode With `--ingress-mode`, the operator reconciles class-matched `Ingress` objects — and, with `--gateway-api`, Gateway API `HTTPRoute` objects — into Synapse's `upstreams.yaml`. Two output layouts: | | Flag | How Synapse picks it up | |---|---|---| | **Sidecar** | `--upstreams-out=/shared/upstreams.yaml` | Shared volume; inotify, then `SIGHUP` to the co-located process | | **Central** | `--upstreams-out-configmap=/` | Synapse reloads from its own ConfigMap mount; no `SIGHUP` | In the central layout only the `upstreams.yaml` key is touched — other keys in that ConfigMap are preserved. ### How paths are translated | Source | Becomes | |---|---| | Ingress `Prefix`, Gateway `PathPrefix` | A Synapse longest-prefix path | | Ingress `Exact`, Gateway `Exact` | Approximated as a prefix, plus a warning event — Synapse matches longest-prefix | | Ingress `ImplementationSpecific` with `nginx.ingress.kubernetes.io/use-regex: "true"` | A regex route: `match_expr: http.request.path matches ""` | | Gateway `RegularExpression` | A regex route, same form | | `/.well-known/acme-challenge/*` | `internal_paths`, for the cert-manager HTTP-01 solver | Regex routes are anchored at the start — the leading `^` is supplied by the renderer, since Kubernetes requires the stored path to begin with `/`. :::note[Header, method and query matches are dropped] Synapse's routing model is host plus path. An Ingress or HTTPRoute rule that also matches on headers, method or query parameters cannot be represented, so those conditions are discarded and a warning event is emitted on the object. The route still exists — it is simply broader than what you wrote. ::: Route conflicts resolve first-writer-wins, and both the conflicts and the dropped matches are counted in metrics, so neither is invisible. ### Certificates and status - **`--certs-out`** projects referenced Ingress and Gateway TLS Secrets into Synapse's certificates directory as `.crt` / `.key`. The directory is operator-owned and hot-reloaded, so certificates arrive without a restart. `--certs-out-secret` does the same into a Secret instead, for a proxy in a separate pod, and takes precedence. - **`--publish-status-address`** writes the addresses you give it onto matched Ingresses' `.status.loadBalancer.ingress`, which is what makes `kubectl get ingress` show an address. - **`--status-leader-election`** gates those shared status writes behind a Lease, so with more than one replica only the holder writes. Per-pod render and `SIGHUP` are never gated. - **`--render-once`** renders `upstreams.yaml` and exits — for an initContainer that primes the file before Synapse starts, so the proxy never boots with no routes. ## The composable reconcilers Each of these is an independent flag that runs alongside whichever primary mode is active. All four are off by default. ### UpstreamsResolver — `--upstreams-resolver` Watches ConfigMaps labelled `synapse.gen0sec.com/resolve-upstreams=true`, substitutes each backend Service's DNS name with its ClusterIP, and writes the result to a sibling ConfigMap. The point is the request path: with IP literals, Synapse hands `upstream_peer` an address and never resolves a hostname per request. Point the dataplane at the output with `synapse.upstreamsConfigMap`. `--resolve-backend-cluster-ips` does the same job inside ingress mode's own render, falling back to the FQDN for headless, ExternalName and not-yet-allocated Services. ### NetVarsResolver — `--netvars-resolver` Watches agent config ConfigMaps labelled `synapse.gen0sec.com/resolve-netvars=true` and fills `ids.address_vars.HOME_NET` and `EXTERNAL_NET` from what the cluster actually is: Node IPs, Pod CIDRs, LoadBalancer VIPs and the RFC1918 supernets. This matters more than it looks. With inline IDS blocking (`enforce_block`) and a `HOME_NET` that does not describe the cluster, the IDS can ban an internal address — a node, a pod, or your own load balancer VIP. Deriving it from the cluster is what stops that. A `HOME_NET` you set by hand, or the `synapse.gen0sec.com/home-net` annotation, overrides the auto-discovery. The `synapse-stack` chart already applies the label to the agent's ConfigMap, so enabling the reconciler is all that is needed. ### IdentityProducer — `--identity-producer` Builds a workload-identity MMDB — pod IP to workload, namespace and app — from cluster Pods, and uploads it so agents can pull it for east-west detection. That is what turns a flow between two pod IPs into a flow between two named workloads. `--identity-producer-interval` (default 5 minutes) is the **baseline** re-upload for cold start and resync; per-change freshness comes from event-driven deltas, so it does not need to be frequent. `--cluster-id` scopes the upload so multiple clusters sharing one workspace do not overwrite each other — without it they share a global path, last writer wins. Set it here *and* set the matching `platform.identity.cluster` on every agent in the cluster. ### EdgeProducer — `--edge-producer` Compiles the cluster's NetworkPolicy objects into a declared-edge allow-list — workload to workload, by port — and uploads it for the agent's `edge.*` microsegmentation fields. It turns policy you have already written into something the agent can compare observed traffic against. ::::note[Both producers need an API key] Each requires an API key with `identity:write`, supplied as `SYNAPSE_API_KEY` — the charts wire this from `operator.apiKeySecret`. :::caution[Their RBAC needs synapse-stack 0.6.0 or newer] The identity producer lists **Pods**; the edge producer also lists **Namespaces** and **NetworkPolicies**. Earlier stack charts granted none of these, so both producers failed on a default install. From 0.6.0 the rules are created automatically when you enable the corresponding producer. The standalone `synapse-operator` chart does not expose the producers as values, so running them there through `operator.extraArgs` still needs a ClusterRole of your own. ::: :::: :::caution[`operator.graphProducer` is not implemented] The `synapse-stack` chart exposes a `graphProducer` block that passes `--graph-producer`, `--graph-producer-interval` and `--service-graph-url`. The operator defines none of those flags. Go's flag parser rejects unknown flags and exits, so enabling it puts the operator into `CrashLoopBackOff` with *flag provided but not defined*. Leave it off. ::: ## Flag reference **Common** | Flag | Default | | |---|---|---| | `--metrics-bind-address` | `:8080` | Metrics endpoint | | `--health-probe-bind-address` | `:8081` | Health probes | | `--leader-elect` | `false` | Leader election for the manager | | `--namespace` | *(all)* | Restrict the watch to one namespace | **Config-sync** | Flag | Default | | |---|---|---| | `--label-selector` | `app.kubernetes.io/name=synapse` | Selects config sources and workloads | | `--config-hash-annotation` | `synapse.gen0sec.com/config-hash` | Annotation key for the hash | | `--ignore-configmap-keys` | `upstreams.yaml` | ConfigMap keys excluded from the hash | | `--ignore-secret-keys` | *(none)* | Secret keys excluded from the hash | | `--per-workload-config-hash` | `false` | Hash only what each workload references | | `--ids-hot-reload-hash-exclude` | `false` | Leave hot-reloadable IDS fields out of the hash | **Ingress mode** | Flag | Default | | |---|---|---| | `--ingress-mode` | `false` | Run as an Ingress / Gateway API controller | | `--ingress-class` | `synapse` | The `ingressClassName` this controller serves | | `--upstreams-out` | `/shared/upstreams.yaml` | Rendered file path, sidecar layout | | `--upstreams-out-configmap` | *(none)* | `/` to render into instead, central layout | | `--render-once` | `false` | Render once and exit, for an initContainer | | `--gateway-api` | `false` | Also reconcile Gateway API; needs the CRDs | | `--resolve-backend-cluster-ips` | `false` | Emit `:port` instead of the Service FQDN | | `--cluster-domain` | `cluster.local` | Cluster DNS domain for backend FQDNs | | `--certs-out` | *(off)* | Project TLS Secrets into this directory | | `--certs-out-secret` | *(off)* | Project them into this Secret instead; takes precedence | | `--publish-status-address` | *(none)* | Addresses to publish on matched Ingresses | | `--reload-process-name` | `synapse` | argv0 of the co-located proxy to `SIGHUP` | | `--reload-debounce` | `500ms` | Coalesce `SIGHUP` bursts; `0` signals every render | | `--status-leader-election` | `false` | Only the Lease holder writes shared status | | `--status-leader-election-id` | `synapse-ingress-status` | Lease name | | `--leader-election-namespace` | `$POD_NAMESPACE` | Namespace for that Lease | **Reconcilers** | Flag | Default | | |---|---|---| | `--upstreams-resolver` | `false` | Substitute Service DNS for ClusterIPs | | `--netvars-resolver` | `false` | Fill IDS `HOME_NET` / `EXTERNAL_NET` from the cluster | | `--identity-producer` | `false` | Upload the pod-IP-to-workload MMDB | | `--identity-producer-interval` | `5m` | Baseline re-upload interval | | `--edge-producer` | `false` | Upload NetworkPolicy as a declared-edge allow-list | | `--edge-producer-interval` | `5m` | Baseline re-upload interval | | `--download-api-url` | `https://api.gen0sec.com/v1` | Where the producers upload | | `--cluster-id` | *(none)* | Scopes this cluster's baseline; empty shares a global path | The charts expose the common options as values, including `--gateway-api` as `operator.gatewayAPI` (standalone) or `operator.ingress.gatewayAPI` (stack). Anything without one — `--certs-out`, `--publish-status-address`, `--render-once` and the rest — goes through `operator.extraArgs`. ## Permissions The operator needs cluster-scoped read on the objects it watches, and write on what it stamps: | API group | Resources | Verbs | |---|---|---| | core | `configmaps` | get, list, watch, create, update, patch | | core | `secrets` | get, list, watch, create, update, patch | | core | `nodes`, `services` | get, list, watch | | apps | `deployments`, `daemonsets`, `statefulsets` | get, list, watch, patch, update | | `networking.k8s.io` | `ingresses`, `ingressclasses` | get, list, watch | | `coordination.k8s.io` | `leases` | full — leader election | | core | `events` | create, patch, update | The charts extend this as you turn features on: `ingresses`, `ingressclasses` and `ingresses/status` with ingress mode, the `gateway.networking.k8s.io` resources with `gatewayAPI`, and `pods`, `namespaces` and `networkpolicies` with the producers. Every rule is gated on the same value that enables the feature, so the two cannot drift apart. ## Metrics Exposed on `--metrics-bind-address`, and picked up by a ServiceMonitor if you enable one in the `synapse-operator` chart. | Metric | | |---|---| | `synapse_operator_ready` | `1` once the first successful render completes | | `synapse_operator_render_total` | Render passes attempted | | `synapse_operator_render_errors_total` | Renders that failed on a list or write | | `synapse_operator_render_changed_total` | Renders that actually changed the output | | `synapse_operator_last_render_timestamp_seconds` | When the last good render happened | | `synapse_operator_reload_signals_total` | `SIGHUP`s delivered to Synapse | | `synapse_operator_hosts` / `synapse_operator_routes` | Size of the current render | | `synapse_operator_route_conflicts_total` | Host+path conflicts ignored | | `synapse_operator_unsupported_match_total` | Match features that could not be represented | | `synapse_operator_backend_unresolved_total` | Backends that could not be resolved | | `synapse_operator_certs` / `synapse_operator_cert_errors_total` | TLS Secrets projected, and those missing or unusable | `synapse_operator_unsupported_match_total` and `synapse_operator_route_conflicts_total` are the two worth alerting on — both mean the cluster is asking for routing that Synapse is silently rendering differently. ## Troubleshooting **Config changes never roll the workload.** Check the selector first — `operator.labelSelector` has to match both the proxy and the agent. Then confirm the annotation is moving: ```bash kubectl -n synapse-os get pod -l app.kubernetes.io/name=synapse-proxy \ -o jsonpath='{.items[0].metadata.annotations.synapse\.gen0sec\.com/config-hash}' ``` **Ingresses render nothing.** Confirm `spec.ingressClassName` matches `--ingress-class`, then check for RBAC denials in the operator log — the stack chart does not grant Ingress access. **A route is broader than the Ingress says.** Header, method and query matches are dropped; look for the warning event on the object and check `synapse_operator_unsupported_match_total`. **The operator crash-loops on startup.** Read the first log line. *Flag provided but not defined* means an argument the binary does not know — `operator.graphProducer` is the usual cause, and `operator.extraArgs` the other. ## See also - [Helm Charts](./helm-charts.mdx) — deploying the operator and the dataplane - [Kubernetes Ingress & Gateway API](./kubernetes-ingress.mdx) — routing with native objects - [Configuration](./configuration.mdx) — the config the operator keeps in sync --- # Testing Guide Source: https://docs.gen0sec.com/docs/client/testing # Testing Guide This guide shows how to manually test and validate Synapse's features. ## Prerequisites Install required tools: ```bash apt-get update apt-get install -y curl tcpdump tshark netcat-openbsd bind9-dnsutils jq ``` ## Start Synapse with Logging ### Enable Debug Logging ```bash # Option 1: Environment variable export RUST_LOG=debug ./target/release/synapse --config config.yaml # Option 2: Specific module logging export RUST_LOG=synapse::tcp_fingerprint=debug,synapse::ja4_plus=debug,synapse::access_log=debug ./target/release/synapse --config config.yaml # Option 3: Log to file ./target/release/synapse --config config.yaml 2>&1 | tee synapse.log ``` ### Watch Logs in Real-Time Fingerprints have a file of their own, and at debug level they also appear in the console log, so there are two streams to watch: ```bash # Startup, errors and feature decisions - the stdout log captured above tail -f synapse.log # Fingerprints, one JSON object per line tail -f /var/log/synapse/eventbridge.log | jq . ``` The fingerprint stream is on by default and is written by an agent and a proxy alike. `/var/log/synapse` is `logging.log_directory`; see [JA4+ Fingerprinting](./ja4-plus#configuration) for the switch and the path. :::caution[`TCP_FP` is not a log prefix] This page used to filter the console log with `grep -E "TCP_FP|JA4|Fingerprint"`. Two of those three alternatives work: with `RUST_LOG=debug` set, as above, the proxy logs lines beginning `JA4:`, `JA4T:`, `JA4H:`, `JA4S:` and `JA4X:`, and other components log messages containing `Fingerprint`. `TCP_FP` matches nothing, though - no log line carries that prefix. (`TCP_FP_KEY_SIZE` and similar names do exist in the source, but they are eBPF compile-time constants, not log output.) Treat the console as a debugging aid and `eventbridge.log` as the record: the console lines appear only at debug level, carry one fingerprint type each, and are not JSON. ::: ## Testing TCP Fingerprinting (JA4T) ### Test IPv4 TCP Fingerprinting ```bash # Simple HTTPS request (generates TCP SYN) curl -v https://localhost:8443/ # The JA4T fingerprint, as recorded tail -f /var/log/synapse/eventbridge.log | \ jq 'select(.event_type == "fingerprint" and .ja4t) | {src_ip, src_port, ja4t}' ``` ### Test IPv6 TCP Fingerprinting ```bash # Test with IPv6 localhost curl -v -6 https://[::1]:8443/ # Test with global IPv6 address (if available) curl -v -6 https://[2001:db8::1]:8443/ # A v6 source address lands in the same field as a v4 one tail -f /var/log/synapse/eventbridge.log | \ jq 'select(.event_type == "fingerprint" and (.src_ip | contains(":"))) | {src_ip, ja4t}' ``` ### Capture TCP Options with tcpdump ```bash # Capture and display TCP SYN packets with options sudo tcpdump -i any -nn -vv 'tcp[tcpflags] & tcp-syn != 0' port 8443 # Output shows: # - TTL (from IP header) # - Window size # - MSS option (kind=2) # - Window scale option (kind=3) # - SACK permitted (kind=4) # - Timestamps (kind=8) ``` ### JA4T Format The JA4T format is: `{window_size}_{tcp_options}_{mss}_{window_scale}` Example: `65535_2-4-8-1-3_1460_7` - Window size: 65535 - TCP options: MSS(2), SACK(4), Timestamps(8), NOP(1), Window Scale(3) - MSS value: 1460 - Window scale: 7 ## Testing HTTP Fingerprinting (JA4H) ### Basic HTTP Request ```bash # Simple GET request curl -v https://localhost:8443/test # Expected JA4H format: {method}{version}{cookie}{referer}{count}{lang}_{headers_hash}_{cookie_names_hash}_{cookie_values_hash} # Example: ge11nr05n_a1b2c3d4e5f6_000000000000_000000000000 ``` ### Test with Custom Headers ```bash # With cookies curl -v -H "Cookie: session=abc123; user=john" \ https://localhost:8443/test # With referer curl -v -H "Referer: https://example.com" \ https://localhost:8443/test # With language curl -v -H "Accept-Language: en-US,en;q=0.9" \ https://localhost:8443/test # With multiple headers curl -v \ -H "Cookie: session=abc123" \ -H "Referer: https://example.com" \ -H "Accept-Language: en-US" \ -H "User-Agent: CustomAgent/1.0" \ -H "Accept: application/json" \ https://localhost:8443/test # Expected JA4H: ge11cr08enus___ ``` ### Compare Different Browsers ```bash # Simulate Chrome curl -v -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \ -H "Accept: text/html,application/xhtml+xml" \ -H "Accept-Language: en-US,en;q=0.9" \ -H "Accept-Encoding: gzip, deflate, br" \ https://localhost:8443/ # Simulate Firefox curl -v -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/119.0" \ -H "Accept: text/html,application/xhtml+xml" \ -H "Accept-Language: en-US,en;q=0.5" \ -H "Accept-Encoding: gzip, deflate, br" \ https://localhost:8443/ # Compare JA4H hashes - they should be different ``` ## Testing TLS Client Fingerprinting (JA4) ### Test Default curl ```bash # Default TLS settings curl -v https://localhost:8443/ # Check curl's TLS library curl --version | grep -E "OpenSSL|GnuTLS|NSS" ``` ### Test with Different TLS Versions ```bash # Force TLS 1.2 curl -v --tlsv1.2 --tls-max 1.2 https://localhost:8443/ # Force TLS 1.3 curl -v --tlsv1.3 https://localhost:8443/ # Different fingerprints should be generated ``` ### Capture TLS ClientHello with tshark ```bash # Capture TLS handshake sudo tshark -i any -f "port 8443" -Y "tls.handshake.type == 1" -V # Look for: # - TLS version # - Cipher suites # - Extensions # - ALPN values ``` ## Viewing Fingerprints Fingerprints are readable from two files, and the two are not interchangeable. | File | Written by | On by default | Also needs | One line is | | --- | --- | --- | --- | --- | | `{log_directory}/eventbridge.log` | agent and proxy | **yes**, `logging.eventbridge_log.enabled` | a capture backend that produces fingerprints | one fingerprint observation | | `{log_directory}/access.log` | proxy only | no, needs `logging.file_logging_enabled` | a Unix build with the `proxy` feature | one HTTP request, fingerprints among its fields | The "also needs" column is the part that catches people out: the file appears as soon as the switch is on, whether or not anything is producing fingerprints to put in it. Three backends do produce them, and only the first needs BPF - the periodic collectors that poll the BPF maps, the Linux AF_PACKET capture path, and the Windows NDIS path. So an empty file means "no backend is feeding it", not "BPF is missing". `log_directory` defaults to `/var/log/synapse` on Linux and `C:\ProgramData\Gen0Sec\Synapse\logs` on Windows. ### The fingerprint stream Each line is one event carrying an `event_type` discriminator, and the fingerprint values sit at the **top level** - not nested under `tls` or `http`: ```bash tail -f /var/log/synapse/eventbridge.log | jq 'select(.event_type == "fingerprint")' ``` ```json { "event_type": "fingerprint", "timestamp": "2025-10-29T12:34:56.789Z", "src_ip": "192.168.1.100", "src_port": 54321, "dst_ip": "10.0.0.5", "dst_port": 8443, "ja4": "t13d1516h2_8daaf6152771_b186095e22b6", "ja4t": "65535_2-4-8-1-3_1460_7", "ja4h": "ge11cr08enus_a1b2c3d4e5f6_123456789abc_def012345678" } ``` Only `timestamp`, `src_ip`, `src_port`, `dst_ip` and `dst_port` are always present. Every fingerprint field is omitted when that fingerprint was not observed, so select on the field rather than assuming it is there: ```bash EB=/var/log/synapse/eventbridge.log # Unique JA4 values, most frequent first jq -r 'select(.event_type == "fingerprint" and .ja4) | .ja4' "$EB" | sort | uniq -c | sort -rn # Unique JA4T values jq -r 'select(.event_type == "fingerprint" and .ja4t) | .ja4t' "$EB" | sort | uniq -c | sort -rn # Correlate source address with fingerprints jq -r 'select(.event_type == "fingerprint") | [.src_ip, .ja4t // "-", .ja4h // "-"] | @tsv' "$EB" ``` ### The proxy access log A proxy also records fingerprints per request, but only with file logging on: ```yaml # /etc/synapse/config.yaml logging: file_logging_enabled: true # default false - without it there is no access.log ``` The fingerprints live under `fingerprints`, and the client address under `network`: ```bash tail -f /var/log/synapse/access.log | jq -R -r 'fromjson? | select(.http) | { timestamp: .timestamp, src_ip: .network.src_ip, ja4: .fingerprints.ja4, ja4h: .fingerprints.ja4h, ja4t: .fingerprints.ja4t, method: .http.method, path: .http.path }' ``` :::note[With file logging off, the record goes to the console] `access.log` exists only with `logging.file_logging_enabled: true`, but the record is emitted either way. With file logging off it goes to the console on an appender of its own, printed **raw** - one JSON object per line, with no timestamp or level prefix. What breaks a naive `jq` is the company it keeps: the console also carries ordinary tracing lines, which are not JSON, so `jq` stops at the first of them with a parse error. Filter the stream instead of feeding it whole: ```bash # Non-JSON lines are skipped rather than fatal ./target/release/synapse --config config.yaml 2>&1 | jq -R 'fromjson? | select(.http) | {src_ip: .network.src_ip, ja4: .fingerprints.ja4}' ``` The console target is stderr unless synapse is in daemon mode, which is why the `2>&1` matters. Setting `file_logging_enabled: true` gets the records into `access.log` away from the tracing lines, which is why the recipes above read the file - but keep the `-R 'fromjson?'` guard there too: the proxy writes its own access lines to that same path through a second appender, and nothing guarantees every one of them is JSON. The per-request `ja4h`, `ja4s` and `ja4x` values are available nowhere else on a default proxy: `eventbridge.log` carries fingerprint observations, not request correlation. ::: :::caution[Three field paths this page used to give do not exist] `client_ip`, `tls.ja4` and `http.ja4h` are not fields of an access-log record. A record's top level is `event_type`, `schema_version`, `timestamp`, `request_id`, `http`, `network`, `tls`, `response`, `error`, `remediation`, `geoip`, `upstream`, `performance`, `fingerprints` and `trace`. Every JA4 value is under `fingerprints`, and the client address is `network.src_ip`; `tls.ja4t` exists too and duplicates `fingerprints.ja4t`, but there is no `tls.ja4`. A `jq` filter naming a field that does not exist prints `null` for every line instead of failing, which is how the wrong paths survived on this page for so long. ::: ## Testing Health Endpoints ### Health Check ```bash # The default address is 127.0.0.1:9180 curl http://127.0.0.1:9180/health # OK # HEAD request (for load balancers) curl -I http://127.0.0.1:9180/health ``` The body is the string `OK`, not a JSON object. `proxy.internal_services` is enabled by default on port `9180`, bound to `127.0.0.1`. :::danger[`bind_ip` is not the exposure boundary] A proxy **also publishes `/health` on its public listeners.** Unless your config already defines an `internal_paths` entry for `/health`, synapse adds one pointing at the internal server, and the router consults `internal_paths` before the upstream table and without regard to hostname. So `curl -k https:///health` answers `OK` from any machine that can reach the proxy at all. `bind_ip: 127.0.0.1` restricts only direct access to port `9180`; it is not what decides whether the path is served publicly. **On a v1 upstreams file** you can take the path back: `internal_paths` is a key of the **upstreams file**, not of `config.yaml`, and an entry there for `/health` replaces the default one. Putting `internal_paths:` in `config.yaml` does nothing except add it to the `Unused config options in ` warning at startup. **On a v2 upstreams file there is no override at all.** The v2 reader does not carry `internal_paths` through, so the default `/health` entry stands whatever the file says. On v2, network controls in front of the proxy are the only lever. ::: :::caution[The internal server is a proxy surface] It is started only when `proxy.internal_services.enabled` is true, synapse is **not** in agent mode, and the build is a Unix build with the `proxy` feature - so a Windows build, which ships without that feature, has no internal server either. An agent serves no `/health` at all, and setting the key changes nothing there. If you are testing fingerprinting on an agent, there is no endpoint to call; read `eventbridge.log`. ::: ## Counting Fingerprints :::danger[There is no fingerprint statistics endpoint] This page used to document `GET /health/tcp_fingerprint_stats`, returning a body with `syn_stats.total_syns`, `unique_fingerprints` and a `fingerprints` array. **No such route exists.** The internal server registers `/health`, `/metrics`, `/cgi-bin/captcha/verify`, `/cert/expiration`, `/cert/expiration/{domain}`, `/cert/renew/{domain}`, the opt-in config-stream path and the ACME challenge directory, and nothing else; `tcp_fingerprint_stats` appears in no source file. The request answers `404`, and `/metrics` carries no fingerprint metric either. **The statistics themselves are real** - only the endpoint was imaginary. With the fingerprint stream on and BPF loaded, synapse logs the whole aggregate to the console at debug level, every 60 seconds, as one JSON object: ```bash # Default config: the line goes to the console RUST_LOG=debug ./target/release/synapse --config config.yaml 2>&1 | grep --line-buffered 'TCP Fingerprint Stats JSON:' # With logging.file_logging_enabled: true, it goes to the application log instead grep 'TCP Fingerprint Stats JSON:' /var/log/synapse/app.log ``` That object is the shape this page used to publish, with `syn_stats.total_syns`, `syn_stats.unique_fingerprints`, `total_unique_fingerprints` and a `fingerprints` array whose entries carry `key.{src_ip,src_port,fingerprint}` and `data.{packet_count,ttl,mss,window_size,window_scale}`. It just never had a URL. ::: ```bash EB=/var/log/synapse/eventbridge.log # Observations recorded grep -c '"event_type":"fingerprint"' "$EB" # Distinct JA4T values jq -r 'select(.event_type == "fingerprint" and .ja4t) | .ja4t' "$EB" | sort -u | wc -l # Everything from one source address jq 'select(.event_type == "fingerprint" and .src_ip == "192.168.1.100")' "$EB" # Everything from an IPv6 prefix jq 'select(.event_type == "fingerprint" and (.src_ip | startswith("2001:")))' "$EB" ``` ## Testing with Multiple Clients ### Generate Load ```bash # Script to test with multiple clients for i in {1..10}; do curl -v https://localhost:8443/test?client=$i & done wait # Count the fingerprints those requests produced jq -r 'select(.event_type == "fingerprint" and .ja4t) | .ja4t' \ /var/log/synapse/eventbridge.log | sort -u | wc -l ``` ### Test IPv4 and IPv6 Simultaneously ```bash # Terminal 1: IPv4 requests while true; do curl -4 -s https://localhost:8443/ > /dev/null sleep 1 done # Terminal 2: IPv6 requests while true; do curl -6 -s https://[::1]:8443/ > /dev/null sleep 1 done # Terminal 3: Monitor fingerprints, split by address family while sleep 2; do clear jq -r 'select(.event_type == "fingerprint") | if (.src_ip | contains(":")) then "v6" else "v4" end' \ /var/log/synapse/eventbridge.log | sort | uniq -c done ``` ## Debugging Tools ### Check BPF Maps ```bash # List the maps synapse loaded (requires root) sudo bpftool map show | grep -E "tcp_fingerprint|tcp_syn_stats|banned" # Dump one, by the id from that listing sudo bpftool map dump id ``` ### Monitor Kernel Logs ```bash # XDP and BPF load failures surface here sudo dmesg -T | grep -iE "xdp|bpf" ``` :::note[`trace_pipe` stays empty] The eBPF programs contain no `bpf_printk`, so nothing from synapse ever reaches `/sys/kernel/debug/tracing/trace_pipe`. The recipe this page used to give piped it through `grep TCP_FP`, which could not match. ::: ### Check XDP Attachment ```bash # Should show synapse's programs. TC/TCX attachments list as `sched_cls`, not `xdp`, # so matching only on "xdp" misses a host where just the overlay is attached sudo bpftool prog show | grep -iE 'xdp|sched_cls' # Check which interface ip link show | grep -B 1 "prog/xdp" ``` ## Performance Testing ```bash # Load test with ab (ApacheBench) ab -n 1000 -c 10 https://localhost:8443/ # Monitor CPU usage during collection pidstat -p $(pgrep synapse) 1 10 ``` ## Quick Test Script ```bash #!/bin/bash # quick_test.sh - Comprehensive test script echo "=== Testing TCP Fingerprinting (JA4T) ===" echo "IPv4 test..." curl -4 -s https://localhost:8443/ > /dev/null && echo "✓ IPv4 request sent" echo "IPv6 test..." curl -6 -s https://[::1]:8443/ > /dev/null && echo "✓ IPv6 request sent" echo "" echo "=== Testing HTTP Fingerprinting (JA4H) ===" curl -s -H "Cookie: test=value" -H "Referer: https://test.com" https://localhost:8443/ > /dev/null echo "✓ Request with headers sent" EB=/var/log/synapse/eventbridge.log echo "" echo "=== Counting Fingerprints ===" COUNT=$(jq -r 'select(.event_type == "fingerprint" and .ja4t) | .ja4t' "$EB" | sort -u | wc -l) echo "✓ Distinct JA4T values: $COUNT" echo "" echo "=== Recent Fingerprints ===" jq -c 'select(.event_type == "fingerprint") | {src_ip, ja4t, ja4h}' "$EB" | tail -3 echo "" echo "=== Access Log Sample (proxy with file_logging_enabled only) ===" if [ -r /var/log/synapse/access.log ]; then tail -5 /var/log/synapse/access.log | jq -R -r 'fromjson? | select(.http) | {ja4t: .fingerprints.ja4t, ja4h: .fingerprints.ja4h, src_ip: .network.src_ip}' else echo "- no access.log: logging.file_logging_enabled is off by default, and only a proxy writes one" fi ``` Make executable and run: ```bash chmod +x quick_test.sh ./quick_test.sh ``` ## Validation Checklist After testing, verify: - [ ] JA4T fingerprints appear in `eventbridge.log` - [ ] JA4H fingerprints change with different headers - [ ] JA4 fingerprints change with different TLS settings - [ ] IPv4 and IPv6 both generate fingerprints - [ ] BPF maps contain entries - [ ] No errors in synapse logs - [ ] `eventbridge.log` carries `"event_type":"fingerprint"` lines - [ ] `/health` answers `OK` on `127.0.0.1:9180` (a proxy only) - [ ] Different clients generate different fingerprints - [ ] Same client generates consistent fingerprints ## Common Issues ### No fingerprints appearing ```bash # Check XDP is enabled ./target/release/synapse --config config.yaml 2>&1 | grep -i xdp # Verify interface is correct ip link show # Check BPF program is loaded. TC/TCX attachments list as `sched_cls`, not `xdp` sudo bpftool prog list | grep -iE 'xdp|sched_cls' ``` ### IPv6 not working ```bash # Enable IPv6 sudo sysctl -w net.ipv6.conf.all.disable_ipv6=0 sudo sysctl -w net.ipv6.conf.default.disable_ipv6=0 # Test connectivity ping6 -c 3 ::1 ``` ### No fingerprint records There is no `tcp_fingerprinting` section, and no per-connection threshold to lower: `logging.tcp_fingerprint` and its `min_packet_count` were removed from the schema. The fingerprint stream has one switch, and it is on by default: ```yaml # /etc/synapse/config.yaml logging: eventbridge_log: enabled: true # the default ``` If the stream is empty, the switch is not the thing to adjust. Check, in this order: 1. **Is a capture backend actually producing fingerprints?** The BPF-fed collectors are gated on a loaded program as well as on the switch, so with `firewall.mode: none`, in a container without `CAP_BPF`, or on a build without BPF they contribute nothing. `sudo bpftool prog show | grep -iE 'xdp|sched_cls'` says whether a program is loaded (synapse attaches TC/TCX programs as well as XDP, and those list as `sched_cls`). A missing BPF program is **not** on its own an explanation for an empty file: a Linux AF_PACKET build and a Windows agent both write fingerprints without any BPF at all. If you are on one of those and the file is empty, this is a fault, not the expected outcome. 2. **Is traffic reaching the capture interface at all?** ## Next Steps - [JA4+ Fingerprinting](./ja4-plus) - Learn about fingerprint analysis - [Configuration Reference](./configuration) - Tune your settings --- # Upgrade Notes Source: https://docs.gen0sec.com/docs/client/upgrade-notes {/* Derive this page from the release's own notes, not from a search of the code: `git show v:CHANGELOG.md`, section "Upgrade notes", then verify each entry against the source and record what you checked. The 0.8.8 section of this page was first written from a diff of the two files its author suspected and missed five of the six behaviour changes, because none of them touched those files. */} # Upgrade Notes These pages describe **Synapse 0.8.8**. This page is the delta: what moves under your feet when you upgrade, what a 0.8.8 configuration file can say that an older one could not, and what happens to such a file if you roll back. ## 0.8.8 ### Read these before you upgrade Each one changes what a running instance does with the configuration you already have. #### Host matching The one that can change **which origin serves a request**, so take it first. - `Host` is now matched without regard to case and without a trailing root dot. `APP.example.com` and `app.example.com.` are served by the host `app.example.com`; before they got a 404, or a wildcard host that covered them. A request that used to be served by the wildcard now arrives at the exact host - with that host's header rules, body limits and WAF context, not the wildcard's. - Expression routes (`match_expr`) on a **wildcard** host now fire. Before, they never did. - A host that serves a bare address - an entry keyed by it, **or a `default_host:`** - and has a route for `/` is served at `/`. Before, that path always got the probe reply, so a load-balancer health check against an address may now get a different answer. Adopting `default_host:` with a route for `/` is enough to change it, with no host keyed by an address anywhere in the file. - In a v1 file, a literal `"*"` host key now acts as the default host. Before it matched nothing. - A name more than one label under a wildcard host that declares `listeners:` is no longer served on other listeners. Requests that worked - and should not have - are answered as an unconfigured host, **or served by `default_host:` if you have one**: a different origin rather than a refusal. - `timeouts.idle: 0` no longer reuses an upstream connection at all. - The origin of a request with an IPv6-literal `Host` receives `[addr]`, not the text before the first colon. #### GeoIP starts working without a platform key An instance with no `platform.api_key` whose configuration names GeoIP databases now opens them at startup, and **downloads any database given by URL whose file is missing**. That is a new outbound request and a new disk write on first start after the upgrade. :::caution[Check this first if your egress is restricted] The shipped `config/config.yaml` names one: `https://github.com/gen0sec/geoip-databases/raw/download/ipinfo_lite.mmdb` to `/var/lib/synapse/ipinfo_lite.mmdb`. On a host that cannot reach GitHub, or whose database path is not writable, this is the item to settle before upgrading. ::: Until now such an instance loaded no database at all, so its records carried no country or ASN. That is the fix; the download is its consequence. #### A firewall rule on `ja4h` may need rewriting With `ids.post_tls.mode: offload`, the JA4H recorded for a request the proxy hands over - in the fingerprint log, and in the store the smart firewall reads - is now the access log's value. It used to be rebuilt from the transaction, with lower-cased header names and an added `Host` on HTTP/2, so it differed from what the proxy logged. A rule written against the old value has to be given the new one, and while proxies are upgraded one at a time the same client is recorded under either value. #### OTLP headers stay with their own destination Synapse pushes telemetry to every destination that is configured, not to the first alone - the endpoint `OTEL_EXPORTER_OTLP_ENDPOINT` names, the one `telemetry.otlp.endpoint` names, and the platform's. That part is not new. What changed is which headers each one receives. Through 0.8.7 the OTLP SDK attached `OTEL_EXPORTER_OTLP_HEADERS` (and its per-signal forms) to **every** exporter it built, replacing the destination's own header of the same name. So a token meant for your collector also went to the platform, and the platform's bearer token was overwritten by it. From 0.8.8 each destination is sent its own headers and no other's: | Destination | Headers it gets | |---|---| | `OTEL_EXPORTER_OTLP_ENDPOINT` | `OTEL_EXPORTER_OTLP_HEADERS`, or the signal's own `OTEL_EXPORTER_OTLP_{LOGS,METRICS,TRACES}_HEADERS` in its place | | `telemetry.otlp.endpoint` | `telemetry.otlp.headers` only | | the platform | the API key, or `platform.telemetry_endpoint.headers` | :::warning[If your collector is named in YAML and its credentials are in the environment] That collector stops receiving them. Copy the credentials into `telemetry.otlp.headers`. Synapse tells you at startup, naming the fix: > `OTEL_EXPORTER_OTLP_HEADERS` is set, but the environment names no endpoint for > its headers (`OTEL_EXPORTER_OTLP_ENDPOINT`): they are sent to no destination. > The headers for `telemetry.otlp.endpoint` go in `telemetry.otlp.headers`. The warning fires when the environment names no endpoint, which is this shape exactly. It stays silent when `OTEL_EXPORTER_OTLP_ENDPOINT` is also set and the token was meant for that other collector - the headers do have a destination then, just not the YAML one. ::: Setting both is safe: with no endpoint in the environment the environment's headers are ignored, and where two sources do name the same URL it is pushed to once, carrying the headers of both. On a clash of names there, the **earlier** destination wins - and the environment's is pushed first, so a stale token in `OTEL_EXPORTER_OTLP_HEADERS` beats a fresh one in `telemetry.otlp.headers`. Two more in the same area: - **`OTEL_EXPORTER_OTLP_COMPRESSION` no longer stops all export.** Setting it, or a per-signal form, used to stop every OTLP exporter from being built - the platform's included - so nothing was exported at all. If your telemetry has been silent since you set it, this upgrade is the repair. The environment's destination now gets the compression it asks for; every other destination is sent uncompressed, as before. - **`OTEL_EXPORTER_OTLP_{LOGS,METRICS,TRACES}_ENDPOINT` now works.** Synapse used to set each signal's endpoint itself, and programmatic configuration beats the environment in the SDK, so these three were read and had no effect. #### Two detection paths see more than they did - **SNI on the AF_PACKET path.** A `tls.sni` smart-firewall rule now sees a name on this path, where it had none, so a rule that never matched there can start to. A TLS ClientHello's fingerprint record also carries that name next to its JA4. - **IDS engine** (thalamus 0.2.0). Lua scripts in IDS rules run in an allowlist sandbox, with byte and instruction limits that cover loading the script too. A script using anything outside the allowlist, or going over a limit, stops working. An HTTP/1 ranged download is now inspected as one file, so rules on file content can match a download that arrived in ranges. ### What a 0.8.8 `upstreams.yaml` can say that a 0.8.7 one could not All of these are additive: a file that worked on 0.8.7 keeps working, and none of them changes what an existing route does. | Construct | What it does | On 0.8.7 | |---|---|---| | `default_host:` | A terminating host serving every name no `hosts:` entry matches; without it such a request is answered 404. It takes no `tls:` block - a client asking for an unknown name is shown `proxy.default_certificate` from `config.yaml` - and it is tried after exact names and wildcards. A bare `*` under `hosts:` is still refused. | **whole file refused** (unknown key) | | `match_expr` on a v2 route | An expression decides whether the route matches, and it can now read the query string and the headers, not only the path, method and host. A host's expressions are tried in the order of their keys. | **whole file refused** (unknown key on the route) | | `tls: { terminate: {} }` | A host terminates TLS naming no certificate of its own: it is served over TLS with whatever the listener presents for that name - a certificate found for it in the certificates directory, or the listener's default - and in plain HTTP. | **whole file refused**: `tls.terminate must specify either cert or acme` | | `paths:` on a passthrough host | TLS for the name is still passed through, and a plain-HTTP request for it - an ACME HTTP-01 challenge, say - is served by the host's `paths:`. | **whole file refused**: `passthrough hosts cannot have paths: - synapse does not see L7` | | `health_check: { type: none }` | Turns upstream probing off for one route. On a terminating route, leaving `health_check` out means probed with the global settings, not unprobed - unlike a passthrough host, where no block means no probing. | **whole file refused**: `health_check.type "none" - expected tcp or http` | :::danger[What a rollback does to such a file] Nothing above is ignored by 0.8.7. The v2 parser rejects unknown keys at the root and on a route, and the other three are explicit validation errors, so the **whole file** fails to load: the proxy keeps running and routes nothing. Two error lines say so - `Failed to parse v2 upstreams YAML (kind=…, source=…)` with the validation message, then `Failed to load upstreams configuration from: ` - and a `config_reload` error metric is recorded, which is the alertable signal. Keep the pre-upgrade file if a rollback is a possibility. One case is quieter. A `match_expr` that reads `http.request.query` or `http.request.headers` - fields 0.8.8 added - **compiles** against no such field on 0.8.7. In a v1 file that is warn-and-skip, not a refusal: the file loads and that one route is simply missing, with one warning among the others - `Skipping expression route () "": compile failed: …`. Grep for `Skipping expression route` after a rollback; nothing else looks like a failure. ::: ### A new warning on every v1 upstreams file Loading an upstreams file in the v1 schema now logs a deprecation warning on every load. The file is still read, and nothing about it stops working. Moving to the v2 schema (`version: 2`, with `hosts:` in place of `upstreams:`) is what closes the warning, and there is still no migration tool - conversion is by hand. ### Unchanged, despite what you may have read - **The main `config.yaml` surface did not move.** No key, flag or default was added, removed or changed between 0.8.7 and 0.8.8. The OTLP environment variables did change effect - the header, compression and per-signal endpoint forms alike - and every one of them is described above. - **Passthrough's schema is the same.** `protocol: http` and the per-wire `passthrough.tls` / `passthrough.http` blocks work on both releases; their parse types are identical. What 0.8.8 adds is `paths:` on a passthrough host, in the table above. - **Telemetry fan-out is not new.** 0.8.7 also pushed to every configured destination. Documentation describing a precedence order, where one destination suppressed the others, was wrong about both releases. - **`match_expr` itself is not new.** It has worked in a **v1** file since before 0.8.8, where it reads the path, method and host. What 0.8.8 adds is `match_expr` in a **v2** file, and the two new fields an expression can read. --- # Upstreams & Routing Source: https://docs.gen0sec.com/docs/client/upstreams # Upstreams & Routing Routing lives in its own file, `upstreams.yaml`, separate from `config.yaml` and reloaded independently of it. The main config only points at it: ```yaml proxy: upstream: conf: "/etc/synapse/upstreams.yaml" ``` Everything on this page is the **v2** schema, which is what `version: 2` at the top of the file selects. v1 files still load — see [Migrating from v1](#migrating-from-v1). :::info[A rejected file means no routing at all] The file is parsed on startup and on every change. The two outcomes are not the same: - **At startup**, a file that fails to parse leaves the proxy with **no routing table**. It logs `Failed to load upstreams configuration from: ` and serves nothing. The file watcher never starts either, so fixing the file afterwards needs a restart. - **On reload**, a file that fails to parse is discarded and the **previous table keeps serving**. The failure is logged and counted, not applied. So a broken edit never takes a running proxy down, but a broken file at boot leaves it with nothing to route. ::: :::danger[The sample shipped at `config/upstreams.yaml` is out of date] The commented sample that ships beside the binary still describes older behaviour. At 0.8.8 it offers suffixed timeouts (`seconds, or use suffix: 30s / 5m`) and the defaults `30/120/30/60`, calls sticky sessions a `consistent-hash by client IP`, says `key:` is `optional; default: .key`, and writes a **path** in `tls.terminate.cert`. Every one of those is wrong for the code that reads the file, and a timeout copied from it with its suffix fails the load outright. This page describes what 0.8.8 actually does; where the two disagree, the sample is the one to ignore. ::: ## The smallest file that works Only `version` is required. A file containing nothing else is valid — it just routes nothing. Write that line in every file. A file without it is not rejected, it is read as the v1 schema, where `hosts:` is not a key at all — so it loads as an empty configuration and routes nothing, saying so in an `Unused upstreams config options (…)` warning and in the [v1 deprecation warning](#migrating-from-v1). For the same reason, a block on this page that does not start with `version: 2` is a fragment to merge into a file that does, not a file. **Terminating TLS, one backend:** ```yaml version: 2 hosts: example.com: tls: terminate: cert: example.com # a certificate NAME, not a path — see below upstream: backend:8080 ``` **TLS passthrough to a database:** ```yaml version: 2 hosts: db.example.com: tls: passthrough: true upstream: 10.0.0.5:5432 ``` :::warning[Unknown keys are a hard error, in all but three places] Almost every block in a v2 file rejects fields it does not recognise, and the error fails the whole load. A typo like `upstreem:` or a v1 key such as `provider:`, `internal_paths:` or `globals:` does not get ignored — it stops the file from loading, and the error names the unknown field. The three that do **not** check are an entry inside an `upstreams:` list, a `rewrite:` block and a `redirect:` block. Misspell `weight:` in an `upstreams:` entry and the file loads with that backend quietly weighted `1`. ::: ## Document structure | Key | Type | Required | Default | |-----|------|----------|---------| | `version` | integer | **yes** | — (must be `2` for this schema) | | `hosts` | map of SNI → host | no | empty (nothing is routed) | | `default_host` | one host, without `tls:` or `listeners:` | no | absent — a name no host matches gets `404` | | `proxy.fingerprints.forward` | bool | no | `true` | | `proxy.fingerprints.header_names` | `g0s` \| `legacy` \| `both` | no | `g0s` | | `proxy.sticky_sessions.enabled` | bool | no | `false` | | `timeouts` | map | no | see [Timeouts](#timeouts) | | `headers.request`, `headers.response` | list of `"Name: value"` | no | empty | | `transforms` | map | no | empty | | `methods` | list of method names | no | all methods allowed | | `normalize_path` | bool | no | unset (the client's spelling is kept) | | `max_url_length` | integer (bytes) | no | unset (no limit) | | `max_header_count` | integer | no | unset (no limit) | | `internal` | list | no | empty — **and never applied**, see [Internal paths](#internal-paths) | `methods`, `normalize_path`, `max_url_length`, `max_header_count`, `timeouts`, `headers` and `transforms` may be set at the top level, on a host, or on a route. ## Hosts A host key is the SNI name. One leading wildcard is allowed: ```yaml hosts: # key shapes only — not a whole file example.com: # exact "*.example.com": # any number of leading labels ``` A bare `*`, a wildcard in the middle (`a.*.example.com`) and an empty key are all rejected. The error on `*` names the alternative: to serve every name nothing else matches, declare a [default host](#the-default-host). ### How a name finds its host One function matches a request's name to a host key, and every per-host table — routes, header rules, transforms — is then read with the key it returns and with nothing else, so a request is one host's in all of them. 1. **The name itself**, compared the way DNS compares names: without the root dot and without regard to case. `Example.COM.`, `example.com.` and `example.com` are one name, and a key written in any of those spellings serves all three. 2. **The wildcard with the longest suffix.** `*.example.com` serves `a.example.com` and `a.b.example.com` alike — any depth — and never the bare `example.com`, which needs its own key. 3. **The default host**, when the file declares one. A name none of the three finds is answered `404`. :::caution[A wildcard covers fewer names on the TLS side] Any depth is the HTTP table's rule. The SNI router that serves **passthrough** hosts matches a wildcard **one label deep**, so `*.example.com` passes `a.example.com` through and not `a.b.example.com`. The difference is deliberate, and it is why `paths:` is refused on a wildcard passthrough host — see [Plain HTTP on passthrough paths](#plain-http-on-passthrough-paths). ::: ### Every host picks exactly one mode | Mode | How you select it | What it does | |------|-------------------|--------------| | **Terminate** | `tls.terminate` with `cert`, with `acme`, or empty (`terminate: {}`) | Synapse ends TLS, inspects HTTP, and proxies to the backend | | **Passthrough** | `tls.passthrough: true`, or a `passthrough:` block | The TLS bytes go to the backend untouched; Synapse routes on SNI alone | Every host needs exactly one of `tls.terminate`, `tls.passthrough: true` or a `passthrough:` block. A host with none of the three fails the load — `tls block missing — pick one of tls.terminate or tls.passthrough: true` — but the `tls:` block itself is optional, which is why a `passthrough:` host below carries none. Setting both `passthrough:` and `tls.passthrough: true` is an error too: they are two spellings of one thing and the parser refuses to guess which was meant. The mode decides which other keys are legal, and the parser enforces it rather than quietly ignoring them: | Key | Terminate | Passthrough | |-----|-----------|-------------| | `paths` | yes | yes, under three conditions — [below](#plain-http-on-passthrough-paths) | | `headers`, `transforms` | yes | **rejected** | | `methods`, `normalize_path`, `max_url_length`, `max_header_count` | yes | **rejected** | | `idle_timeout`, `max_connections`, `proxy_protocol_v2` | **rejected** | yes | ### The default host `default_host:` sits beside `hosts:` and serves every name no entry under `hosts:` matches. Without one, such a request is answered `404`. ```yaml version: 2 hosts: example.com: tls: terminate: cert: example.com upstream: backend:8080 default_host: paths: /: upstream: catch-all:8080 ``` It is a terminating host, and it takes everything one takes: `upstream:`/`upstreams:` or `paths:`, timeouts, headers, transforms, `methods` and the limits. Its routes are checked against `internal:` for colliding paths like any other host's, and that error names it `default_host.paths."/…"`. Three keys are refused rather than ignored, because they describe a name it does not have: | Written on `default_host:` | Outcome | |---|---| | `passthrough:`, or `tls.passthrough: true` | Load fails — passthrough picks its backend by name before any request is read | | `tls.terminate` or `tls.passthrough`, in any form | Load fails — there is no name for a certificate to be chosen by | | `listeners:` | Load fails — it serves every listener a terminating host can be served on | A client that reaches it over TLS is shown `proxy.default_certificate` from `config.yaml`, which is what a name no host has is shown anyway. :::note[In a v1 file the key `"*"` does this] A v1 host key goes into the routing table as written, and `*` is the key the default host is kept under — so a v1 file gets a catch-all by naming a host `"*"`. In a v2 file that key is refused and `default_host:` is the only spelling. ::: ## Terminating TLS ### `cert` is a name, not a path ```yaml version: 2 hosts: example.com: tls: terminate: cert: example.com upstream: backend:8080 ``` :::danger[A path here silently serves the wrong certificate] `tls.terminate.cert` is matched against the **logical names** of the certificates Synapse found when it scanned `proxy.certificates`. A logical name is the file stem (`example.com.crt` → `example.com`), or the **parent directory name** when the stem is `fullchain`, `cert` or `chain` (certbot's `…/example.com/fullchain.pem` → `example.com`). No logical name can contain a `/`, so a value like `/etc/synapse/admin.pem` can never match. The load still succeeds — nothing checks the string at parse time — and then at handshake Synapse logs ``` Certificate '/etc/synapse/admin.pem' specified in upstreams config for hostname 'example.com' not found in cert_name_map. Available certificates: [...] ``` and serves the **default certificate** instead. Clients get a certificate for the wrong name, and the only sign is that warning. For a certificate to be available at all, the directory scan has to have found a pair: `.crt` beside `.key`, or `/fullchain.pem` beside `privkey.pem`. A `.crt` with no matching key is skipped. ::: :::note[A host can terminate without naming a certificate] `terminate: {}` is a complete TLS block. The host terminates, and names no certificate of its own: ```yaml version: 2 hosts: internal.example.com: tls: terminate: {} upstream: backend:8080 ``` Over TLS such a host is served with whatever the listener presents for its name — a certificate the directory scan found for it, or the listener's default. In plain HTTP it is served like any other host. It is the v2 spelling of a v1 host with no `certificate:`, and the only way to write a host in a v2 file that is reached over plain HTTP alone. ::: :::note[`key:` is accepted and then discarded] A sibling `key:` under `terminate` parses, and is then dropped — nothing reads it, and nothing derives a `.key` filename from `cert`. The private key actually used is always the one found next to the certificate during the scan. Where `key:` still has an effect, it is a refusal: beside `acme:` it is an error, and with no `cert` beside it, an error of its own. ::: ### ACME ```yaml version: 2 hosts: example.com: tls: terminate: acme: challenge_type: http-01 # http-01 | dns-01 email: admin@example.com # optional; falls back to proxy.acme.email wildcard: false # true requires challenge_type: dns-01 upstream: backend:8080 ``` `acme: {}` is valid and takes every default. `cert` and `acme` are mutually exclusive — both together is an error — and neither is valid too: that is [`terminate: {}`](#terminating-tls), the host with no certificate of its own. `tls-alpn-01` is accepted by the parser and is **not implemented**. Any value other than `dns-01` takes the HTTP-01 path, so writing it gets you http-01 without a word of warning. Enabling ACME, its port and its storage live in `proxy.acme` in `config.yaml`; this block only supplies the per-certificate settings. ## Routes A terminating host either names one backend for everything (`upstream`/`upstreams`), or splits traffic by path with `paths`. Both together is an error, and neither is an error. ```yaml version: 2 hosts: example.com: tls: terminate: cert: example.com paths: /: upstream: web:8080 /api: upstreams: - addr: api-1:9000 weight: 3 - addr: api-2:9000 weight: 1 timeouts: read: 120 max_body_size: 10485760 ``` Route paths must start with `/` — unless the route sets `match_expr`, where the key is only a label — and matching is longest-prefix rather than exact: with `/` and `/api` declared, `/api/v2/users` is served by `/api` and everything else by `/`. Within one host, the order is: the exact path, then the [expression routes](#routes-chosen-by-an-expression) in the order of their keys, then the longest prefix, then `/`. (Synapse's own internal paths are tried before all of them, whatever the host.) A route whose backends were all unusable is skipped rather than failing — see [Backends](#backends). | Field | Type | Default | What it does | |-------|------|---------|--------------| | `upstream` | `"host:port"` | — | One backend. Exclusive with `upstreams` | | `upstreams` | list of `"host:port"` or `{addr, weight}` | — | Several backends; `weight` defaults to `1` | | `methods` | list | inherit | Anything else gets `405` with an `Allow` header — with two exemptions, below | | `normalize_path` | bool | inherit | Send the backend the resolved path instead of the client's spelling | | `max_url_length` | bytes | inherit | Over the limit gets `414` | | `max_header_count` | integer | inherit | Over the limit gets `431` | | `max_body_size` | bytes | unlimited | Over the limit gets `413` | | `timeouts` | map | inherit | See [Timeouts](#timeouts) | | `headers`, `transforms` | map | inherit + append | See [Headers](#headers-and-transforms) | | `force_https` | bool | `false` | Redirect plaintext to HTTPS | | `ssl_enabled` | bool | `true` for port 443 or 8443, else `false` | TLS **to the backend** — until the health probe observes otherwise | | `http2_enabled` | bool | `false` | Offer HTTP/2 to the backend. Decides anything only for a cleartext backend | | `disable_access_log` | bool | `false` | Drop this route's access-log lines | | `rewrite` | `{match_path?, set_path}` | — | Rewrite the path sent upstream; `set_path` required | | `redirect` | `{status, location}` | — | Answer directly; both required | | `health_check` | `{type, path?}` | probed | See [Health checks](#health-checks) | | `match_expr` | expression string | — | Decide the route by the request, not by the key — see [below](#routes-chosen-by-an-expression) | Four of these deserve a second look, because the name does not say which side of the proxy they act on or what else overrides them: - `ssl_enabled` is TLS **to the backend**, not from the client. Its default is inferred from the backend port: 443 and 8443 imply TLS, anything else plaintext. On a probed route it is not the last word: the probe tries HTTPS before HTTP, and once a backend has answered, what the probe saw is what Synapse speaks to it — an explicit `ssl_enabled: false` to a backend that answers HTTPS is overridden. Treat the key as what gets used until the first probe lands, not as a way to force plaintext. On a route that is not probed the answer differs by which kind it is: [What decides TLS to the backend](#what-decides-tls-to-the-backend) has the three cases. - `http2_enabled` is settled by ALPN whenever the backend is reached over TLS, so the key only decides anything for a cleartext backend you mean to speak h2c to. - Synapse does **not** verify the backend's certificate when it speaks TLS upstream: neither the chain nor the hostname is checked. Upstream TLS here is encryption on the wire, not authentication of the backend. - `force_https` only makes sense when a TLS listener exists. Configure it without one and Synapse warns once and keeps serving plaintext. :::note[`methods` is not a complete gate] Two requests get past a `methods:` list: - **`HEAD` rides along with `GET`.** List `GET` and a `HEAD` is served too, and the `Allow:` header on a 405 advertises the implied `HEAD` alongside it. There is no way to list `GET` and refuse `HEAD`. - **An http-01 challenge is never method-gated.** A request for `/.well-known/acme-challenge/` skips the check on whatever route covers that path, so an external ACME client (certbot, cert-manager) cannot be locked out of a renewal by a narrow `methods:` list. Everything else outside the list gets `405`. Treat `methods:` as narrowing a route, not as an access control. ::: A `redirect.status` outside 301, 302, 303, 307 and 308 silently becomes 302. ### Routes chosen by an expression A route with `match_expr` is chosen by the request rather than by its key, and its key becomes a label — it need not start with `/`, and no other route's prefix matching sees it. ```yaml version: 2 hosts: example.com: tls: terminate: cert: example.com paths: /: upstream: web:8080 beta: match_expr: 'http.request.query matches "(^|&)beta=1(&|$)"' upstream: beta:8080 prod-writes: match_expr: 'http.request.method eq "POST" and any(http.request.headers["x-env"][*] eq "prod")' upstream: writes:9000 ``` Within a host they are tried **in the order of their keys**, after the exact path and before the longest prefix. Sort order is the decider, so name them for the order you want. Five things a request offers an expression: | Field | Type | Notes | |-------|------|-------| | `http.request.path` | string | always present | | `http.request.method` | string | always present | | `http.request.host` | string | the name the client asked for, which for a wildcard or the default host is **not** the host key | | `http.request.query` | string | the query string, without the `?`; read on demand, see below | | `http.request.headers` | map of name → list of values | names in **lower case**, each with a list of values; read on demand, see below | `any`, `all` and `lower` are available: ``` any(http.request.headers["x-canary"][*] eq "1") any(lower(http.request.headers["x-env"][*])[*] eq "prod") all(http.request.headers["x-env"][*] matches "^[A-Za-z]+$") ``` A request is only handed what the expressions of its host actually read: the query string when one asks for it, and of the headers, the ones they name — or all of them when an expression asks about headers in general. :::warning[A capitalised header name is refused at load, not quietly unmatched] Header names are held in lower case, as HTTP keeps them. `http.request.headers["X-Env"]` would match no request at all, so the load fails instead and the error gives the spelling to use: `header names are in lower case: write "x-env" for "X-Env"`. A name no header could have is refused the same way. An expression that does not compile also fails the load — the whole file, as any other v2 error does. ::: :::caution[An expression route is never health-probed] The prober goes by a route's path, and an expression route has none. Two consequences: - A `health_check:` block on it is refused unless it says `type: none`, rather than being accepted and ignored. - Its backends are reached by what `ssl_enabled` says, not by what a probe found — see [What decides TLS to the backend](#what-decides-tls-to-the-backend). With `ssl_enabled` unset and backends on a mix of TLS and plaintext ports, the load fails and asks for an explicit value. ::: ### Inheritance Global, host and route levels merge most-specific-wins — a route's `max_url_length` overrides its host's, which overrides the top level's. `headers` and `transforms` are the exception: they **append** rather than replace, in that order, so a route's header rules run after its host's. ### Backends `upstream` takes one `"host:port"`. A **host-level** `upstream:` given a list gets a dedicated error pointing at `upstreams:`; the same mistake on a route gets serde's bare `invalid type: sequence, expected a string`. `upstreams` takes several, each either a plain string or `{addr, weight}` with a `weight` of `1` by default. An empty `upstreams` list is an error, and so is a list where every weight is `0`. :::caution[IPv6 literals do not work as terminate backends] A backend address is split at the first `:` and the part after it parsed as a port, so `[2001:db8::1]:8080` does not parse — and the entry is **skipped without any log line**. An address with no `:port` at all goes the same way. What a route does when every one of its backends was skipped is worse than failing: it counts as **not declared**, so the request falls through to the next shorter route — usually `/` — or to a 404. One typo'd backend on `/api` can quietly send `/api` traffic to the `/` backend. Name IPv6 backends by hostname instead. ::: ### Load balancing `lb:` belongs on a host that names its own backends, on a route, or inside a passthrough wire block. There is no top-level `lb:` — written at the top of the file it is an unknown key and the load fails. ```yaml paths: /api: upstreams: [api-1:9000, api-2:9000] lb: method: round_robin # round_robin | random | least_request | weighted_round_robin | consistent ``` An unknown method is an error listing the five valid ones — in each of the three places `lb:` is read. A host-level `lb:` on a passthrough host that declares `tls:`/`http:` wires is **not** one of them: each wire reads its own, so the host-level block is dropped without being looked at, and even `method: nonsense` loads there. Put it inside the wire. :::warning[`lb.method` only takes effect on passthrough hosts] On a **terminating** host or route, `lb.method` is validated and then dropped. Backend selection there is always the weighted cycle, so `least_request` and `consistent` on an HTTP route do nothing at all. Weights are honoured; the method name is not. On a **passthrough** host it is honoured, and only three of the five are accepted: `round_robin`, `weighted_round_robin` and `random`. `least_request` and `consistent` are rejected outright. ::: ## Timeouts ```yaml timeouts: connect: 5 read: 60 write: 30 idle: 60 ``` | Timeout | Default | What it bounds | |---------|---------|----------------| | `connect` | `5` | Opening the connection to the backend | | `read` | `60` | Reading the backend's response | | `write` | `30` | Writing to the backend | | `idle` | `60` | How long a pooled backend connection is kept for reuse | :::caution[Plain integer seconds only] These are integers. `connect: 30s` and `read: 5m` are **type errors that fail the whole file** — including the suffixed forms some older examples show. `idle` is a connection pool keepalive, not a cap on how long a connection may live. ::: Set them at any of the three levels; they merge most-specific-wins. For the bytes a passthrough host passes through, only `connect` applies: `read`, `write` and `idle` are parsed and unused there, and idle connections are bounded by its own `idle_timeout` instead. The three are not dead on such a host, though. A passthrough host with [`paths:`](#plain-http-on-passthrough-paths) hands its merged timeouts down to those routes, which read all four the way a terminating host's routes do — so a host-level `timeouts.read` that the passed-through wire ignores still bounds its HTTP routes. :::note[`idle: 0` means the connection is not reused at all] A route with `timeouts.idle: 0` gets a connection of its own per request. Over HTTP/1 the request carries `Connection: close` upstream, so the backend closes it once the response is read instead of it going to the pool to be closed a moment later; over HTTP/2, and on an upgrade, the pool key alone keeps it from being reused. Use it for a backend that must not see a second request on a connection — not as a short keepalive. ::: ## Health checks A v2 route is health-checked **by default**. `type: none` turns probing off for one. ```yaml paths: /api: upstream: api:9000 health_check: type: http # http | tcp | none path: /healthz # http only; must start with "/"; defaults to "/" ``` `type: tcp` with a `path` is an error, and anything other than the three names is too — `expected tcp, http or none`. The **interval and thresholds are not per route.** `interval`, `consecutive_failures` and `consecutive_successes` are rejected inside a terminating route's `health_check`; the probe interval for terminating hosts comes from `proxy.upstream.healthcheck` in `config.yaml`. They are accepted only under a passthrough host's `health_check`: ```yaml version: 2 hosts: db.example.com: upstream: 10.0.0.5:5432 passthrough: health_check: type: tcp interval: 10s # a suffix is REQUIRED here: ms | s | m consecutive_failures: 3 consecutive_successes: 2 ``` A `passthrough:` block with no `tls:`/`http:` wire inside it takes the host-level `upstream:` and serves the **TLS** wire, which is why the example above needs no `tls.passthrough: true` — writing both would be rejected. :::note[`interval` is the one duration that needs a unit] Every other duration in this file is a bare integer of seconds. `health_check.interval` is the opposite: it is a string and the unit suffix is mandatory — `ms`, `s` or `m` (`h` is not supported). `interval: 10` is an error; `interval: 10s` is correct. In passthrough it is then **truncated** to whole seconds with a floor of one, so `1500ms` probes every second, not every two. ::: ### What decides TLS to the backend A route that is never probed has nothing to learn the scheme from, so which key decides it depends on why the route is unprobed. Three cases, for a terminating host's routes: | The route | Probed? | What decides TLS to the backend | |---|---|---| | probed — no `health_check:` block, or `type: http` / `type: tcp` | yes | the probe — whatever the check type, it tries HTTPS before HTTP, and what answered overrides `ssl_enabled`. When **neither** answers — a backend that accepts the connection but speaks no HTTP scheme — the **port** decides instead, so an explicit `ssl_enabled` holds only until the route's first verdict | | `health_check: {type: none}` on a route chosen by its path | no | the backend's **port**: TLS on 443 and 8443, plaintext on anything else. `ssl_enabled` is not consulted, and a value that disagrees with the port **fails the load** | | chosen by [`match_expr`](#routes-chosen-by-an-expression) | no | `ssl_enabled`, which is honoured as written. Left unset it is taken from the ports, and backends split across TLS and plaintext ports **fail the load** | A route chosen by `match_expr` that *also* carries `type: none` is the last row, not the middle one: it loads, and `ssl_enabled` is honoured as written. The port rule is applied to the table of path-chosen routes, and an expression route is not in it. The refusal in the middle row is there rather than sending TLS traffic in plaintext or the reverse: drop `type: none` so the route is probed, or use a port that says what you mean. :::note[`type: none` takes no other key] `path`, `interval`, `consecutive_failures` and `consecutive_successes` are each refused beside it, since there is no probe for them to describe. Under a passthrough host's `health_check` it is accepted too, where it means the same as leaving the block out. ::: :::caution[v2 refuses what v1 discards in silence] The port rule is not new and it is not a v2 rule. A v1 file with `healthcheck: false` discards a disagreeing `ssl_enabled` in exactly the same way — it accepts the key and the serving table overwrites it. The difference is only whether you are told: v1 says nothing, v2 refuses the file. So a v1 route that has run for years with `healthcheck: false`, `ssl_enabled: true` and a backend on `:8080` has been speaking **plaintext** to it, not TLS. Converting it to `health_check: {type: none}` changes nothing about which scheme it speaks, and the file will not load until the `ssl_enabled` goes or the port says what you mean. That is the file stopping hiding a disagreement, not a change of behaviour to plan around. ::: ## Passthrough The modern form names a wire explicitly. `tls` is routed by SNI on the TLS listener, `http` by the `Host` header on the plaintext listener: ```yaml version: 2 hosts: db.example.com: passthrough: tls: upstreams: - addr: 10.0.0.5:5432 weight: 1 lb: method: round_robin # round_robin | weighted_round_robin | random idle_timeout: 3600 # seconds; 0 = unbounded max_connections: 500 # per host, across both wires; 0 is rejected proxy_protocol_v2: false # prepend a PROXY v2 header to the backend ``` | Key | Default | Notes | |-----|---------|-------| | `idle_timeout` | `3600` | Seconds. `0` means no bound | | `max_connections` | no cap | Counted per host across both wires. `0` is an error | | `proxy_protocol_v2` | `false` | Prepends a PROXY v2 header to the backend connection, ahead of any application bytes | A passthrough host must have a backend somewhere — a wire block, or a host-level `upstream`/`upstreams`. A wire block cannot be combined with a host-level `upstream`, and `protocol:` (the older single-wire selector, `tls` or `http`; `auto` is rejected) cannot be combined with a wire block either. :::caution[An HTTP-only passthrough host still answers on 443] Declare `passthrough.http` with no `passthrough.tls` and Synapse warns that a TLS connection for that name will be **terminated with the default certificate** rather than passed through. If the name should never be terminated, give it a `tls` wire. ::: ### Plain HTTP on passthrough paths A passthrough host can serve paths of its own in plain HTTP while TLS for the same name is still passed through untouched. The case it was built for is an ACME HTTP-01 challenge for a name whose TLS belongs to the backend. ```yaml version: 2 hosts: db.example.com: tls: passthrough: true upstream: 10.0.0.5:5432 paths: /.well-known/acme-challenge: upstream: 127.0.0.1:8080 ``` Those routes are served by the same L7 pipeline that serves a terminating host's. Three conditions apply, and breaking one is refused with its reason rather than ignored: | Condition | Why | |---|---| | The host key is an exact name, not a wildcard | A wildcard matches one label deep for TLS and any depth for HTTP, so the paths would be served for names this host does not pass through | | The host passes **TLS only** — no `http:` wire, no `protocol: http` | A plain-HTTP request for the name is copied to the backend before any path could match | | `paths:` has at least one entry | An empty map says nothing; list a path or drop the key | A host-level `headers:` or `transforms:` block is still refused on a passthrough host, `paths:` or no `paths:`. Header rules for these routes go on the route. ## Listeners ```yaml version: 2 hosts: admin.example.com: listeners: [internal] tls: terminate: cert: admin.example.com upstream: 127.0.0.1:8080 ``` With no `listeners`, a host is served on every listener. Named listeners restrict it. A name that does not exist is dropped with a warning, and a host whose whole list is unknown names is **served nowhere** — Synapse says so explicitly: `host "…" is bound to no valid listener and will not be served anywhere`. Two bindings have to admit a request, not one: the binding of the name it asked for, and the binding of the host that will serve it. The two are not always the same host — `*.internal.example` serves `a.b.internal.example`, a name that matches no binding of its own. A request a listener turns away does not simply get a `404`: if the file declares a [default host](#the-default-host), that host serves it, and only without one is the request answered the way an unconfigured name is. ## Headers and transforms ```yaml headers: request: - "X-Forwarded-Proto: https" response: - "X-Frame-Options: DENY" ``` Entries are `"Name: value"` strings. An entry with no `:`, an invalid header name or value, or one naming a framing header (such as `Content-Length`) is **skipped with a warning** rather than failing the load — `Skipping unusable header entry …`. `transforms` is the richer form, with `request_headers` and `response_headers` rule lists and conditional `when` clauses. Unlike a bad `headers:` entry, a `transforms` rule that does not compile **rejects the entire file**: `Rejected upstreams configuration: …; no upstreams are served until it is fixed`. The reload loop keeps watching, so fixing the file recovers without a restart. ## Fingerprint forwarding ```yaml proxy: fingerprints: forward: true header_names: g0s # g0s | legacy | both ``` Forwarding is **on by default**, and the default names are the namespaced ones — `G0S-JA4`, `G0S-JA4H`, `G0S-JA4T`, `G0S-JA4T-Hash`, `G0S-JA4S`, `G0S-JA4TS`, `G0S-JA4TS-Hash`, `G0S-JA4L`, `G0S-JA4LS`, `G0S-JA4X` and `G0S-JA4-Raw`. `legacy` sends the pre-namespace `X-JA4*` spellings instead, and `both` sends each fingerprint under both names. A backend written against `X-JA4` therefore sees nothing until you set `legacy` or `both`. ## Sticky sessions ```yaml proxy: sticky_sessions: enabled: true ``` With this on, Synapse reads a **`backend_id` cookie** from the request and pins that request to the backend it names. It is a cookie the response set, not a hash of the client address — a client that sends no cookie is balanced normally. A `key:` field under `sticky_sessions` parses but is never read; the cookie name is not configurable. ## Internal paths :::warning[A v2 `internal:` list does nothing] A top-level `internal:` block parses, and is even checked for duplicate paths and for collisions with your routes — and is then **dropped**. Nothing serves it. Internal endpoints come from `proxy.internal_services` in `config.yaml` instead; the v1 `internal_paths:` key is the only file-level form that is actually applied. ::: ## Fields that parse but do nothing Worth knowing, because the file loads without complaint: | Field | Status | |-------|--------| | `internal` (top level) | Parsed, validated, never applied | | `tls.terminate.key` | Parsed, dropped | | `proxy.sticky_sessions.key` | Parsed, never read | | `lb.method` on terminate hosts and routes | Validated, then dropped | | `lb` on a passthrough host that declares `tls:`/`http:` wires | Parsed, **not** validated, dropped — it belongs inside the wire block | | `timeouts.read`, `.write`, `.idle` on a passthrough host with no `paths:` | Parsed, unused — only `connect` applies. With `paths:` they are inherited by those routes | ## What a bad file does | Mistake | Outcome | |---------|---------| | Unknown or misspelled key, anywhere | Load fails | | `connect: 30s` (suffix on an integer timeout) | Load fails (type error) | | `interval: 10` (no suffix on the one string duration) | Load fails | | Wildcard SNI other than one leading `*.` | Load fails | | A host with none of `tls.terminate`, `tls.passthrough: true` and `passthrough:`, or with more than one | Load fails | | `cert` and `acme` together | Load fails (neither is valid — it is `terminate: {}`) | | `key` under `terminate` without a `cert` | Load fails | | `paths` together with `upstream`, or neither, on a terminate host | Load fails | | Empty `upstreams`, or all weights `0` | Load fails | | Route path not starting with `/`, on a route without `match_expr` | Load fails | | `default_host:` with `tls.terminate`, `tls.passthrough`, a `passthrough:` block or `listeners:` | Load fails | | `paths:` on a wildcard passthrough host, on one that also passes plain HTTP, or an empty `paths:` map | Load fails | | `path`, `interval` or a `consecutive_*` key beside `health_check: {type: none}` | Load fails | | `ssl_enabled` beside `type: none` disagreeing with the backend's port | Load fails | | A `health_check` other than `type: none` on a `match_expr` route | Load fails | | A `match_expr` that does not compile, or that writes a header name with a capital in it | Load fails | | A `match_expr` route whose backends mix TLS and plaintext ports with no `ssl_enabled` | Load fails | | Passthrough-only key on a terminate host, or the reverse | Load fails | | A `transforms` rule that does not compile | Configuration rejected — the file parsed, and a fixed edit recovers without a restart | | Bad `headers:` entry | Warning, entry skipped, file loads | | `cert` naming a certificate that was never scanned | Warning per handshake, **default certificate served** | | Backend address an IPv6 literal, or no `:port` | Silently skipped, no log; a route left with none falls through | | Unknown listener name | Warning, entry dropped | Remember the asymmetry above: at startup a failed load means no routing, while on reload the previous table keeps serving. ## Migrating from v1 A file without `version: 2` is parsed as v1, and v1 is still read — with a warning on every load: ``` the upstreams file /etc/synapse/upstreams.yaml is in the v1 schema, which is deprecated. It is still read. Move it to the v2 schema (`version: 2`, with `hosts:` in place of `upstreams:`) ``` An empty document is exempt, being in no schema yet. The mapping: | v1 | v2 | |----|----| | `upstreams:` | `hosts:` | | `upstreams..certificate` | `hosts..tls.terminate.cert` | | `upstreams..acme` | `hosts..tls.terminate.acme` | | `paths.

.servers:` (string or `{address, weight}`) | `upstream:` for one, or `upstreams:` with `addr`/`weight` | | `https_proxy_enabled` (alias `force_https`) | `force_https` | | `ssl_enabled`, `http2_enabled`, `disable_access_log`, `max_body_size`, `rewrite`, `redirect` | unchanged, per route | | `request_headers` (alias `headers`), `response_headers` | `headers.request` / `headers.response` | | `healthcheck`, `healthcheck_path`, `healthcheck_tcp` | `health_check: {type, path}` | | `healthcheck: false` | `health_check: {type: none}` | | host key `"*"` | `default_host:` | | `connection_timeout`, `read_timeout`, `write_timeout`, `idle_timeout` | `timeouts.connect` / `.read` / `.write` / `.idle` | | `allowed_methods` | `methods` | | `normalize_path`, `max_url_length`, `max_header_count` | unchanged | | `sticky_sessions` | `proxy.sticky_sessions.enabled` | | `config.forward_fingerprints`, `config.fingerprint_header_names` | `proxy.fingerprints.forward` / `.header_names` | | `provider` | none — v2 is always a file, and the key is rejected outright | | `internal_paths` | none that works — see [Internal paths](#internal-paths) | Two more move rather than disappear: `config.global_request_headers` / `global_response_headers` become the top-level `headers.request` / `headers.response`, and `config.healthcheck_interval` / `healthcheck_method` leave this file altogether for `proxy.upstream.healthcheck.interval` / `.method` in `config.yaml`. :::caution[A v1 `provider` other than `file` costs you the whole routing table] `file` is the only value implemented. Anything else logs `Unknown provider ` and loads **nothing** — at startup that leaves the proxy with no routing table and no file watcher, so it cannot be fixed without a restart. v2 rejects the key instead, which is the louder failure of the two. ::: These have **no v2 form at all**: `config.https_proxy_enabled` — there is no global force-https, only the per-route `force_https` — along with `globals` and `authorization`. `match_expr` routes are not among them: see [Routes chosen by an expression](#routes-chosen-by-an-expression). :::caution[A typo in `version` falls back to v1 and loads almost nothing] Only the bare integer `version: 2` works. A quoted `version: "2"` is recognised as v2 and then fails to load, because the field is an integer: `invalid type: string "2", expected u32`. `version: 3`, or a missing `version`, makes the file parse as **v1** — where `version:` and `hosts:` are not keys at all, so they land in an `Unused upstreams config options (…)` warning and the file loads as an effectively empty v1 config. The proxy starts, routes nothing, and says so twice: that warning, and the v1 deprecation notice, which any non-empty file that is not `version: 2` earns. There is no "unsupported version" error for this. ::: There is no migration tool at 0.8.8; conversion is by hand. ## Upstream DNS caching When a backend is named by hostname, the resolved address is cached for `proxy.upstream.dns_cache_ttl_secs` (default 60) rather than resolved per connection. On Kubernetes that matters more than it sounds: without it every upstream connection makes a DNS query, which puts CoreDNS in the request path of all your traffic. ## Next steps - [Configuration Reference](./configuration.mdx) — `config.yaml`, including the `proxy.upstream.*` keys that point at this file - [Load Balancing](../features/load-balancing.mdx) — the balancing behaviour itself - [Kubernetes Ingress](./kubernetes-ingress.mdx) — generating this file from Ingress objects --- # SynapseOS Source: https://docs.gen0sec.com/docs/synapseos/ # SynapseOS *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* **The operating system your appliance arrived with.** SynapseOS is the Linux that runs on a Gen0Sec appliance, whether that is a Cerebrum Max, a Cerebrum Edge, a server or virtual machine of your own, or an NVIDIA BlueField-3 card. It exists to run [Synapse](../client/index.mdx) and nothing else. Your unit arrives with it installed and signed; these pages start at the login prompt. ## How it differs from a general-purpose Linux | | On SynapseOS | |---|---| | The system itself (`/usr`) | Read-only, and checked block by block on every boot. An edit there does not persist and makes the next boot fail verification | | Your configuration (`/etc`, `/var`, `/root`) | Writable, and kept across updates | | Updates | One signed update replaces the whole system and its kernel together. The previous system stays on disk, and the appliance returns to it by itself if the new one does not come up | | Software | There is no package manager. What is on the unit is what the image was built with; new software arrives through an update | | Network exposure | Inbound traffic is dropped by default, except SSH | | Access | SSH accepts root with a key only. A password, where one is set, works on the console only | The read-only system is the reason the rest holds: the kernel, Synapse and every library are verified against a hash that is sealed into the signed kernel image, so a changed file cannot be run, and an update either verifies completely or is not booted. ## What you see after logging in The login banner names the build the unit was delivered with (the running one, after an update, is in `synapseos-ab-update status`): ```text SynapseOS 20261009-141152-1bf85f2 (hardened ab) | kernel 6.18.54-synapseos A/B slots | /usr sealed with dm-verity ``` and lists the three commands you will use most: | Command | What it does | |---|---| | `synapseos-synapse list` | The Synapse versions on the unit, and which one runs | | `synapseos-ab-update` | Check for, install and roll back OS updates | | `systemctl status synapse-agent synapseos-firewall` | The data plane and the base firewall | ## Where to go next - [Logging in](./access.mdx): credentials, keys and the console. - [Configuring the appliance](./configure.mdx): network, firewall, Synapse and logs. - [Network requirements](./network.mdx): what the unit accepts inbound and reaches outbound, proxies, a private CA. - [Updating](./updates.mdx): checking for an update, installing one, and going back. - [Synapse versions](./synapse-versions.mdx): choosing which of the shipped releases runs. - [Running the proxy](./proxy.mdx): enabling Synapse's proxy mode beside the agent. - [High availability](./high-availability.mdx): operating two units as a pair with one floating address. - [Platforms](./platforms.mdx): what differs on a Cerebrum Max, a Cerebrum Edge, a server, a VM or a BlueField-3. - [Serial console](./serial-console.mdx): connecting a laptop to a Cerebrum Edge's console port. - [Backup, restore and decommission](./backup.mdx): what to keep, how to put it on a replacement, how to wipe a unit. - [Troubleshooting](./troubleshooting.mdx): the messages you may see, what they mean, and what to collect for support. - [Configuration reference](./reference.mdx): every file you may change, and three reference configurations. - [Security model](./security.mdx): what is verified, what cannot be changed, how to report a vulnerability. - [Known limitations](./limitations.mdx): what the unit does not do yet, and what to do instead. - [What changed](./changelog.mdx): dated changes to these pages and the builds they were checked against. :::note[The command line voids nothing by itself] The banner carries the usual warning about the CLI. Everything on these pages is a supported way to operate the appliance. What is not supported is changing the system itself, and the unit refuses that on its own. ::: --- # Logging in Source: https://docs.gen0sec.com/docs/synapseos/access # Logging in *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* **Root, with a key, over SSH. A password is for the console.** ## The first credential An image ships with the root account locked and no password at all: a credential baked into every unit would be a secret shared by every customer. Access is provisioned per unit instead, in one of two ways. - **Delivered with the unit.** A physical appliance, or a VM Gen0Sec deployed for you, arrives with an SSH key already installed for root, or with a console password. Use what came with the unit. - **Your own cloud-init seed.** A VM you start from a SynapseOS image reads a NoCloud seed on first boot: a FAT-formatted volume labelled `CIDATA` holding `meta-data` and `user-data`. The key it installs is what you log in with. ```yaml title="user-data" #cloud-config disable_root: false ssh_authorized_keys: - ssh-ed25519 AAAA... you@example.com ``` ```text title="meta-data" instance-id: appliance-01 local-hostname: appliance-01 ``` The seed delivers credentials and, through `local-hostname`, the unit's name. Networking on the appliance is not managed by cloud-init; see [Configuring the appliance](./configure.mdx#network). ## SSH ```bash ssh root@

``` As shipped, root logs in with a key only (`PermitRootLogin prohibit-password`) and password authentication is off for every account. A unit can be delivered with password SSH turned on instead, by a drop-in under `/etc/ssh/sshd_config.d/`; `sshd -T | grep -i passwordauthentication` shows which applies, and removing the drop-in restores key-only. SSH on port 22 is the only service the base firewall accepts from outside by default. To add a key of your own, append it to `/root/.ssh/authorized_keys`. It lives on the writable root filesystem and survives updates. ## The console A password works on the serial console and nowhere else. If no password came with the unit, set one once you are in over SSH: ```bash passwd root ``` That also unlocks the account for console login; SSH stays key-only. The console is the only way in when the network is misconfigured, so it is worth having a password set before you change addressing. Where the console is depends on the unit: a Cerebrum Max's BMC, a Cerebrum Edge's serial port over USB, a server's BMC, or a VM's serial port. See [Platforms](./platforms.mdx) and, for the cable and the terminal settings, [Serial console](./serial-console.mdx). ## Naming the unit ```bash hostnamectl set-hostname appliance-01 ``` The name is written to `/etc/hostname` and kept across updates. ## Knowing what you are running `/etc/os-release` and the login banner name the build the unit was **delivered** with: | Field | Meaning | |---|---| | `VERSION` | The delivered image's build stamp, as shown in the login banner | | `SYNAPSEOS_RECIPE` | Which platform build this is | | `SYNAPSEOS_BUILD_OPTIONS` | The image options it was built with, such as `hardened` and `ab` | Updates replace the system, not `/etc`, so neither changes afterwards. The build that is **running** is in `synapseos-ab-update status`, beside the slot marked `<- running`, and in `/usr/lib/os-release`, which belongs to the running system. Use those when reporting a unit. See [Updating](./updates.mdx). --- # Backup, restore and decommission Source: https://docs.gen0sec.com/docs/synapseos/backup # Backup, restore and decommission *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* **Back up `/etc` before every update. Wipe the writable partition before a unit leaves.** ## What to back up Everything you have changed on a unit is on the writable partition, and all of it is small: | Path | What it holds | |---|---| | `/etc/synapse/` | Synapse's configuration, the version pin, the upstreams file, and `certs/` with your private keys | | `/etc/synapseos/update.conf` | The update channel | | `/etc/systemd/network/` | Static addressing | | `/etc/nftables/synapseos-base.nft` | The base firewall, with the ports you opened | | `/etc/axosyslog/conf.d/` | Log forwarding | | `/etc/systemd/system/` | Unit drop-ins, such as the proxy's | | `/etc/keepalived/` | The pair configuration, on a pair | | `/etc/hostname`, `/root/.ssh/authorized_keys`, `/etc/ssl/certs/` (your CA, if any) | Identity and trust | Not worth backing up: `/usr` (it is the signed system, replaced by updates), the journal (forward it instead, see [Logs](./configure.mdx#logs)), and `/var/lib/synapse` (databases and managed rules the agent downloads again). ## Taking a backup From a machine that can reach the unit over SSH: ```bash ssh root@
'tar -C / --exclude=etc/machine-id --exclude=etc/os-release -czf - etc root/.ssh' \ > synapseos-$(date -u +%F).tar.gz ``` That is the whole of a unit's configuration. Keep it with the care you keep the private keys in it, since `etc/synapse/certs/` and the SSH host keys are in there. Take one before every update and after every change you mean to keep. The two exclusions are the unit's identity, which must not travel to another unit: `machine-id` (two units sharing one collide on DHCP leases and in logs) and `os-release` (it names the build the unit was delivered with). The SSH host keys are carried on purpose, so clients that knew the old unit accept the replacement; leave them out if you would rather the replacement have its own. ## Restoring onto a replacement unit A replacement arrives like the first unit did: installed, with its own first credential. Restore in this order, over SSH, on a unit of the same platform. 1. Check it runs the same or a newer SynapseOS build as the backup came from (`/etc/os-release`, `VERSION`), and update it first if not; see [Updating](./updates.mdx). 2. Copy the archive to the unit and unpack it over `/`: ```bash scp synapseos-2026-10-10.tar.gz root@
:/var/tmp/ ssh root@
'tar -C / -xzf /var/tmp/synapseos-2026-10-10.tar.gz && rm /var/tmp/synapseos-*.tar.gz' ``` 3. Apply what needs a reload, then reboot so everything else is picked up together: ```bash systemctl daemon-reload networkctl reload systemctl restart synapseos-firewall systemctl reboot ``` 4. After the reboot: `synapseos-synapse status` (the pin from the backup is applied, and a version the new image no longer ships falls back to the newest, so check), `systemctl status synapse-agent`, and on a pair, `ip -br addr` on both units. The unit's machine identity (`/etc/machine-id`) is not in the backup, so the replacement keeps its own; it registers with the platform under the hostname from the backup. ## Decommissioning a unit The writable partition is not encrypted, and it holds the private keys and credentials listed above. Before a unit is returned, resold or scrapped, wipe it from the console. 1. Take a final backup if anything on it matters. 2. From the console, identify the writable partition: it is the last one on the disk, mounted at `/`. ```bash findmnt -no SOURCE / # for example /dev/sda6 or /dev/nvme0n1p6 ``` 3. Reboot into the other slot's system is not enough: both slots share the same writable partition. Discard it instead. On flash and NVMe, `blkdiscard` tells the device to erase it; on disks that do not support discard, overwrite it: ```bash blkdiscard -f /dev/sda6 # or shred -n 1 -v /dev/sda6 ``` Both destroy the running system's root filesystem; the command runs from memory and the unit will not boot again until re-imaged. 4. Power off. The system partitions that remain hold nothing of yours: they are the signed image every unit of that platform ships with. For a unit under a support contract, a wipe followed by a return is what Gen0Sec expects; a unit sent back without one still holds your keys. --- # What changed in these pages Source: https://docs.gen0sec.com/docs/synapseos/changelog # What changed in these pages *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later).* **Newest first. Each entry names the build the pages were checked against.** Every page carries an "applies to" line under its title naming the oldest SynapseOS build it describes. A command or a file that differs on an older build is called out where it differs. Report a page that does not match what your unit does to [support@gen0sec.com](mailto:support@gen0sec.com), with the build from the login banner. ## 2026-10-12 The sections below were checked against SynapseOS build `20261012151622-6.18.54`, Synapse 0.8.8. The `status` example under [Checking](./updates.mdx#checking) still shows the output of earlier builds. - [Updating](./updates.mdx): a new section, Setting up network updates. It covers the update key, the URL form, `check` results and what a `401` or `404` means, keeping the key out of shared output, channels, and the bootloader fix for units on builds before 2026-10-12. - [Updating](./updates.mdx): offline installation is now a full walkthrough. It covers what you receive, checking the checksum, preparing ext4 media (FAT32 cannot hold the system image), installing, and the same bootloader fix for older builds. - [High availability](./high-availability.mdx): moving the address before updating the active unit. ## 2026-10-10 Checked against SynapseOS builds of 2026-10-09 (`20261009-141152` and later), Synapse 0.8.6. - New: [Network requirements](./network.mdx), with the outbound destinations and ports, the HTTP proxy for updates, and a private CA. - New: [Troubleshooting](./troubleshooting.mdx), [High availability](./high-availability.mdx), [Backup, restore and decommission](./backup.mdx), [Configuration reference](./reference.mdx). - [Updating](./updates.mdx): the commands come first, `status` is shown with its real output, and the checks after a reboot are listed. - [Security model](./security.mdx): what leaves the unit, and to where. - [Known limitations](./limitations.mdx): no HTTP proxy for the agent; no self-service evaluation image; the clock is not synchronised as delivered; the variant check cannot be overridden although the updater's message says it can. - Corrections from a second review: the login banner and `/etc/os-release` name the build the unit was delivered with, not the running one; a backup leaves `machine-id` and `os-release` out; the audit log and the local syslog files are outside the journal; a pair tracks its external leg only when the address lives there; the macOS device name with the Prolific driver; the proxy configuration file is created with its final mode. ## 2026-10-09 Checked against SynapseOS builds of 2026-10-09, Synapse 0.8.6. - The section's first version: overview, logging in, configuring, updating, Synapse versions, the proxy, platforms, the serial console, the security model and known limitations. - Corrections from review before publication: the offline update is an archive to unpack on the media; `(verified)` in `status` is dm-verity, and the trial counter in a boot entry's name is what says a slot was confirmed; `reset` restarts nothing; BlueField-3 units run with the card's Secure Boot off; a same-architecture build for another platform can install and boot. --- # Configuring the appliance Source: https://docs.gen0sec.com/docs/synapseos/configure # Configuring the appliance *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* **Everything you change lives in `/etc`. Everything else is sealed.** ## What is writable | Path | Status | |---|---| | `/etc`, `/var`, `/root` | Writable. Configuration, state, journals and your SSH keys live here, and an update leaves them alone | | `/usr` | Read-only and verified on every boot. An edit there is refused, and one forced through by other means makes the next boot fail verification | There is no package manager and nothing to install. If a tool you need is not on the unit, it is not available until an update brings it. Every file this page touches is listed with its keys on the [Configuration reference](./reference.mdx); what the unit needs to reach is on [Network requirements](./network.mdx). ## Network ### The default With no addressing configured, every interface named `en*` or `eth*` runs DHCP for IPv4 and IPv6 and takes its DNS servers from the lease. Interfaces are managed by `systemd-networkd`; `networkctl status` shows what each one got. ### A static address systemd-networkd applies the first `.network` file whose `[Match]` fits an interface, in name order. The DHCP fallback is `90-fallback-dhcp.network`, so a file that sorts before it wins. Match on the MAC address rather than the name, so the file survives a rename: ```ini title="/etc/systemd/network/30-external.network" [Match] MACAddress=52:54:00:12:34:56 [Network] Address=203.0.113.10/24 Gateway=203.0.113.1 DNS=203.0.113.53 IPv6AcceptRA=yes ``` ```bash networkctl reload networkctl status ``` Change addressing from the console, or with a second way in, and keep the old address until the new one answers. ### DNS Name resolution goes through `systemd-resolved`, with the servers that DHCP or your `.network` file supplied. `resolvectl status` shows them. Link-local discovery (LLMNR and mDNS) is turned off; the appliance resolves through configured servers only. ## The base firewall Inbound traffic is dropped unless a rule accepts it. The rules are in `/etc/nftables/synapseos-base.nft`, loaded by `synapseos-firewall.service` before the network comes up, and the only service open by default is SSH: ```text chain services { tcp dport 22 accept comment "ssh" } ``` To open another port, add a line to that chain, check the file, and reload it: ```bash nft -c -f /etc/nftables/synapseos-base.nft # syntax check, changes nothing systemctl restart synapseos-firewall nft list chain inet synapseos_base services # the new line is in the live ruleset ``` Established connections, loopback and ICMP are always accepted; forwarding is dropped; outbound is open. :::note[Two firewalls] This is the appliance's own perimeter. Synapse's packet filter, the one that enforces access rules and bans in the kernel, is separate and is configured through Synapse. See [Firewall Rules](../features/firewall-rules.mdx). ::: ## Synapse Synapse reads `/etc/synapse/config.yaml`. The one value every unit needs before it does anything useful is the key that connects it to the platform: ```yaml title="/etc/synapse/config.yaml" platform: api_key: "" ``` ```bash systemctl restart synapse-agent systemctl is-active synapse-agent # active journalctl -u synapse-agent -b -n 20 # "Started Synapse", then nothing that says failed ``` The full key reference is on the [Configuration](../client/configuration.mdx) page. Two things are specific to the appliance: - The file is seeded once, from the reference configuration of the newest Synapse version on the unit, and then left alone. Each version's reference copy stays readable at `/usr/share/synapse//config.yaml` for comparison. - `/etc/synapse/version` records which Synapse version runs. Do not edit it by hand; use `synapseos-synapse`, which checks the version before switching to it. See [Synapse versions on the appliance](./synapse-versions.mdx). The proxy mode is on the unit as well, disabled, and needs a configuration of its own before it is started. See [Running the proxy](./proxy.mdx). ## Logs ### Retention The journal is the local log store for every service. As shipped it is persistent, capped at 4 GB and two weeks, and keeps 2 GB of the root filesystem free. The rate limit is raised so a burst of detections is kept rather than dropped. The forwarder also writes the classic files `/var/log/messages`, `auth.log`, `kern.log` and `daemon.log` from the same journal, and the audit log is separate; see below. ```bash journalctl -u synapse-agent -f journalctl --disk-usage ``` ### Forwarding An `axosyslog` forwarder reads the journal and ships it. Its destination is a drop-in under `/etc/axosyslog/conf.d/`; a unit delivered with forwarding configured has one there already. To add or change a collector, write the file and reload: ```text title="/etc/axosyslog/conf.d/10-remote.conf" destination d_remote { syslog("logs.example.com" transport("tls") port(6514) tls(ca-dir("/etc/ssl/certs") peer-verify(required-trusted)) disk-buffer(disk-buf-size(268435456) reliable(no) dir("/var/lib/axosyslog"))); }; log { source(s_journal); destination(d_remote); }; ``` ```bash systemctl reload axosyslog ``` TLS verifies the collector against the system trust store. `tcp` and `udp` transports work too, on port 514 by convention, but neither authenticates the collector. The disk buffer holds up to 256 MB while a collector is unreachable and replays it afterwards. ### Audit `auditd` runs with a rule set that records changes to the configuration above, to credentials and to the system's own files. Its records go to `/var/log/audit/audit.log`, which `auditd` rotates itself (8 MB, five files), not to the journal; read them with `ausearch -ts boot` or `ausearch -k `. --- # High availability Source: https://docs.gen0sec.com/docs/synapseos/high-availability # High availability *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* **Two units, one address. The one that holds it does the work.** A pair is configured at delivery. This page is what you need to know to operate one; the configuration itself is set once and listed on the [Configuration reference](./reference.mdx#a-pair). ## How a pair works | | | |---|---| | The floating address | One address, the one traffic is sent to. It lives on whichever unit is active. | | Where it lives | On the external leg, when both units share that network segment. On the internal leg when they do not: a floating address cannot cross a router, so a pair whose external addresses are in different routed networks floats the address on the shared private segment and takes public traffic through a provider floating IP or an upstream balancer instead. | | The election | VRRP, between the two units, over the internal leg, unicast to the peer's address. The unit with the higher priority wins while it can advertise. When the address lives on the external leg, that leg is tracked too: if it goes down, the unit withdraws and the other takes the address. When the address lives on the internal leg, the election and the address share it, so a failure of that leg ends the advertisements by itself, and a failure of the external leg withdraws nothing. | | After a failover | The unit that failed does not take the address back when it recovers (`nopreempt`). The pair stays where it is until the active unit fails in turn. This avoids a second cutover for a unit that was flapping. | | The MAC | Optionally a virtual MAC (`00:00:5e:00:01:`) on a macvlan interface named `vrrp.`, so the address keeps the same MAC through a failover and upstream devices have nothing to relearn. Without it, failover relies on every upstream device honouring a gratuitous ARP. | Both units run the full data plane all the time; only the address moves. Synapse's own state (bans, verdicts, rules) comes from the platform, so the standby is as current as the active unit. ## What to check ```bash systemctl status keepalived # active on both units ip -br addr # the floating address appears on exactly one unit journalctl -u keepalived -b # elections, transitions and the reason for each ``` On a pair with a virtual MAC, the floating address is on an interface named `vrrp.`, not on the physical interface: ```text eno1 UP /24 ... vrrp.51@eno1 UP /24 ``` A unit that holds the address logs `Entering MASTER STATE`; the other logs `Entering BACKUP STATE`. ## Testing a failover From the active unit: ```bash systemctl stop keepalived ``` Within a few seconds the other unit logs `Entering MASTER STATE` and `ip -br addr` on it shows the floating address. Traffic to the address continues from there. Then: ```bash systemctl start keepalived ``` The address stays on the new active unit: `nopreempt` means the recovered unit becomes the standby. Do this in a maintenance window the first time; existing connections through the address are cut at the cutover. :::note[If the address moves but traffic does not follow] An upstream router that cached the address against the old unit's physical MAC, and ignores gratuitous ARP, keeps sending to the dead unit until the entry ages out. On a pair with a virtual MAC, one packet sent *from* the floating address refreshes it: `ping -c3 -I ` on the new active unit. After that the MAC never changes again, which is the point of the virtual MAC. ::: ## Updating a pair Update the standby first, confirm it, then the active unit. The address moves once. 1. On the standby: `synapseos-ab-update update`, `systemctl reboot`, then `synapseos-ab-update status` and `synapseos-synapse status` to confirm the new slot is running and the data plane is up. See [Updating](./updates.mdx). 2. On the active unit: the same. When it reboots, the election hands the address to the already-updated standby, and `nopreempt` keeps it there afterwards. 3. Confirm with `ip -br addr` on both. To move the address at a time of your choosing rather than at the active unit's reboot, do it before step 2. On the active unit, run `systemctl stop keepalived`, confirm with `ip -br addr` that the standby now holds the address, then run `systemctl start keepalived` straight away. `nopreempt` leaves the address on the standby, and the pair can still fail over while the active unit downloads and writes its update. As with [Testing a failover](#testing-a-failover), connections in flight are cut when the address moves. Watch the service from a client for a few minutes before you update. Do not update both at once: a pair with both units in their trial boots has no unit to fall back to. ## What is the same on both Configuration is written identically on both units at delivery, and should be kept so: the same `/etc/synapse/config.yaml` apart from the unit's own addresses, the same firewall rules, the same log forwarding. A change you make on one unit by hand has to be made on the other, or the pair behaves differently after a failover. The [backup](./backup.mdx) of one unit is a good template for the other. ## Capacity Each unit carries the full load on its own after a failover, so size a pair for one unit's capacity, not two. The throughput each product delivers is on its product page; the pair adds availability, not capacity. --- # Known limitations Source: https://docs.gen0sec.com/docs/synapseos/limitations # Known limitations *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* If you meet something that is not on this page, we would like to hear about it at [support@gen0sec.com](mailto:support@gen0sec.com). ## An update for another platform is accepted **What it means.** The update channel URL you were given is specific to your platform build. An update published for a different platform carries a valid signature, and the unit does not check which platform it was built for, so a unit pointed at the wrong channel installs it. **What you see.** It depends on how different the platform is. A build for another CPU architecture does not boot at all: after three attempts the bootloader returns to the previous slot by itself. A build for another platform of the same architecture can boot, bring Synapse up and be confirmed good, leaving the unit on a kernel and drivers that were not meant for it, with no automatic return. **What to do.** Before the first update, check `UPDATE_BASE_URL` in `/etc/synapseos/update.conf` against what Gen0Sec gave you for this platform. If a wrong build was installed, `synapseos-ab-update rollback` returns to the previous slot while it is still there, then fix the URL. See [Updating](./updates.mdx). ## A service introduced by an update starts disabled **What it means.** An update replaces the system and the kernel, and leaves `/etc` alone. For most services, whether they start at boot is recorded in `/etc`, so a service that an update brings for the first time can arrive installed but not enabled. Services the unit already had keep their state. **What you see.** The release notes name a new service; after the update `systemctl status ` reports `disabled`. **What to do.** Check its state first, since some services are enabled by the system itself and need nothing. For one that reports `disabled`, once per unit: ```bash systemctl enable --now ``` ## Synapse versions are fixed per image **What it means.** The system is read-only, so the Synapse releases on a unit are the ones its image was built with. `synapseos-synapse use` can only choose among them. **What to do.** A newer release arrives with the next OS update. See [Synapse versions](./synapse-versions.mdx). ## Data at rest is not encrypted **What it means.** The system itself is integrity-checked, but the writable partition that holds configuration, keys and logs is not encrypted. Anyone with the disk can read it. **What to do.** Treat the unit's storage as sensitive when a unit is transported, returned or disposed of. See [Security model](./security.mdx#what-is-not-covered). ## Log retention is bounded **What it means.** The journal is the unit's local log store, capped at 4 GB and two weeks as shipped. Older entries are gone from the unit. **What to do.** Forward logs to a collector of yours; the forwarder buffers while the collector is unreachable. See [Logs](./configure.mdx#logs). ## Updates are all-or-nothing per channel **What it means.** A channel names one update, and every unit following it takes that update when it next checks. There is no staged rollout within a channel beyond the `canary` and `stable` convention. **What to do.** Point a few units at `canary` and the rest at `stable`, and ask Gen0Sec to move `stable` once the canary units have run the release. The channel is `UPDATE_CHANNEL` in `/etc/synapseos/update.conf`. ## No HTTP proxy for the agent **What it means.** Synapse connects to `api.gen0sec.com` and `download.gen0sec.com` directly; it has no proxy setting. Only the updater honours one. **What you see.** On a network that allows outbound traffic through a proxy only, the agent logs connection failures and runs without the platform's rules and verdicts. **What to do.** Allow the two hosts directly for the unit, on 443. See [Network requirements](./network.mdx#behind-an-http-proxy). ## The clock is not synchronised as delivered **What it means.** `systemd-timesyncd` ships disabled, so the unit's clock runs free from the moment it is powered on. TLS verification and certificate expiry depend on the clock. **What you see.** After a long power-off, connections to the platform or a collector fail certificate checks, or a certificate appears expired early. **What to do.** Enable time synchronisation once, and allow 123/UDP to the servers you name. See [Network requirements](./network.mdx#time-synchronisation). ## The variant check cannot be overridden, although its message says it can **What it means.** An update of another variant (hardened versus baseline) is refused with `bundle is variant 'X' but this appliance runs 'Y' ... override deliberately with --allow-downgrade`. The flag does not override this check; only the version check honours it. **What you see.** The same refusal with the flag. **What to do.** Check the channel: this almost always means the wrong one. A unit that is meant to change variant is re-imaged by Gen0Sec. See [Troubleshooting](./troubleshooting.mdx#bundle-is-variant-x-but-this-appliance-runs-y). ## No self-service evaluation image **What it means.** A unit arrives installed, or Gen0Sec deploys a VM for you. These pages describe no way to download an image and try the appliance on your own hardware. **What to do.** Ask Gen0Sec for an evaluation unit or VM; the first credential comes with it. See [Logging in](./access.mdx#the-first-credential). --- # Network requirements Source: https://docs.gen0sec.com/docs/synapseos/network # Network requirements *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* **Inbound: SSH. Outbound: the platform, the update channel, and your log collector.** Open these before the unit is expected to do anything. Every outbound row is a connection the unit opens itself; nothing on the platform connects back in. ## Inbound | Port | Protocol | When | Where it is set | |---|---|---|---| | 22 | TCP | Always: SSH, key-only | The base firewall's `services` chain, see [Configuring the appliance](./configure.mdx#the-base-firewall) | | 80, 443 | TCP | Only when the proxy mode runs | Opened by you, see [Running the proxy](./proxy.mdx#open-the-ports) | | VRRP (IP protocol 112) | On the internal leg | Only in a two-unit pair | Set at delivery, see [High availability](./high-availability.mdx) | | ICMP, ICMPv6 | | Always: ping and neighbour discovery are accepted | Base firewall | Everything else inbound is dropped. The unit answers nothing on the external leg that is not in this table. ## Outbound | Destination | Port | Used by | When | |---|---|---|---| | `api.gen0sec.com` | 443/TCP | Synapse | Continuously: registration, detections and telemetry up; access rules, fleet bans and verdicts down | | `download.gen0sec.com` | 443/TCP | Synapse | Periodically: indicator and threat databases, and their version check | | `github.com`, `raw.githubusercontent.com` | 443/TCP | Synapse | Periodically: the GeoIP database the shipped configuration points at | | The host in your `UPDATE_BASE_URL` | 443/TCP | `synapseos-ab-update` | Only when you run `check` or `update`; never on its own | | Your log collector | 6514/TCP (TLS), or 514 TCP or UDP | `axosyslog` | Continuously, if forwarding is configured | | Your DNS servers | 53 UDP and TCP | `systemd-resolved` | Always | All three Gen0Sec hosts and the update host are reached over HTTPS and verified against the system trust store. The update channel and the update archive carry their own signatures on top of that, so the update host needs no trust at all; see [Updating](./updates.mdx#what-an-update-is). The GeoIP row comes from the database URL in the `geoip` section of the shipped Synapse configuration (`/etc/synapse/config.yaml`). Point it at a mirror of your own and that row changes with it; the keys are on the [Configuration](../client/configuration.mdx) page. ## Time synchronisation The unit does not synchronise its clock as delivered: `systemd-timesyncd` is present but disabled. TLS verification and certificate expiry both depend on the clock, so either enable it and allow 123/UDP outbound to the servers you name in `/etc/systemd/timesyncd.conf`, or set the time from the console after any long power-off: ```bash systemctl enable --now systemd-timesyncd # uses NTP= in /etc/systemd/timesyncd.conf timedatectl status # "System clock synchronized: yes" once it has a server ``` ## Behind an HTTP proxy `synapseos-ab-update` fetches with `curl`, and it sources `/etc/synapseos/update.conf` before it does, so a proxy for the update channel goes in that file: ```bash title="/etc/synapseos/update.conf" UPDATE_BASE_URL= UPDATE_CHANNEL=stable export https_proxy=http://proxy.example.com:3128 ``` `export` matters: without it the variable is set for the script and not for `curl`. Then `synapseos-ab-update check` proves the path. Synapse itself has no proxy setting: the agent connects to `api.gen0sec.com` and `download.gen0sec.com` directly. On a network that only allows outbound traffic through a proxy, allow those two hosts directly for the unit, or the agent runs blind; see [Known limitations](./limitations.mdx#no-http-proxy-for-the-agent). ## A private certificate authority If your log collector, or a TLS-inspecting proxy, presents certificates from your own CA, the unit has to trust it. The trust store is `/etc/ssl/certs`, on the writable partition, and `openssl` is on the unit: ```bash cp my-ca.pem /etc/ssl/certs/my-ca.pem openssl rehash /etc/ssl/certs systemctl reload axosyslog ``` The file survives updates, which leave `/etc` alone. Do not put a private CA in front of `api.gen0sec.com` or the update host: the agent and the updater pin nothing, but a proxy that re-signs their traffic is a machine in the middle of your security sensor, and the update signatures will still verify only against Gen0Sec's keys. ## Checking from the unit ```bash curl -sS -o /dev/null -w '%{http_code}\n' https://api.gen0sec.com/ # expect a 2xx or 4xx, not a timeout synapseos-ab-update check # exercises the update host journalctl -u axosyslog -n 20 # a collector that is unreachable says so here ``` A `000` from `curl`, or `cannot reach ` from the updater, is the network, not the unit. See [Troubleshooting](./troubleshooting.mdx). --- # Platforms Source: https://docs.gen0sec.com/docs/synapseos/platforms # Platforms *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* **The same system everywhere; the way in differs.** Every platform runs the same SynapseOS: the same commands, the same configuration, the same [update](./updates.mdx) mechanism. What differs is how you reach the console, which network interface answers first, and what the firmware beneath it does. | Unit | Console | |---|---| | Cerebrum Max | The BMC: serial over LAN, or the BMC's web console | | Cerebrum Edge | A serial port on the unit, over USB; see [Serial console](./serial-console.mdx) | | A server of your own | Its BMC | | A VM | Its serial port | ## Cerebrum Max The 2U sensor, on an NVIDIA Grace host with a BlueField-3 card. SynapseOS runs on the host; the card is managed from it. **How to reach it.** The unit's BMC carries the console: connect to the BMC's address over the management network and open serial-over-LAN (IPMI `sol activate`, or the BMC's web console). The console shows the whole boot and works whatever state the system is in. The BMC's address and credentials are delivered with the unit. **First network path.** Every interface named `en*` or `eth*` runs DHCP until it is given a static address; see [Configuring the appliance](./configure.mdx#network). ## Cerebrum Edge The 1U sensor, on one or two NXP LX2160A systems. There is no BMC: the console is a serial port on the unit, reached over USB from a laptop. | | Value | |---|---| | Serial console | `ttyAMA0`, 115200 baud, 8N1; see [Serial console](./serial-console.mdx) | | Firmware | UEFI (EDK2), in the board's SPI flash; the OS does not update it | | System disk | The on-board eMMC | | Kernel | NXP's vendor kernel, which drives the board's SD and eMMC controllers | The firmware is selected by the board's boot switches, which are set at delivery: the board boots from SPI flash and leaves the eMMC to the OS. ## Servers and virtual machines x86-64 and arm64 servers boot SynapseOS through UEFI. A VM runs the same image as a disk file. | | Where | |---|---| | Console | The server's BMC or the VM's serial port | | First network path | Every interface named `en*` or `eth*` runs DHCP until it is given a static address | | First credential | Delivered with the unit, or from your cloud-init seed on a VM | See [Logging in](./access.mdx) for the seed and [Configuring the appliance](./configure.mdx#network) for addressing. ## NVIDIA BlueField-3 On a BlueField-3, SynapseOS runs on the card's own Arm cores, as a separate system from the host the card is installed in. Two things differ from a server. **How to reach it.** The card's out-of-band 1 GbE port is the network path: it runs DHCP like any other interface, so cable it. Do not rely on the virtual link between the host and the card that NVIDIA's own system uses; whether it is available depends on the build the unit runs. The card's BMC also exposes the card's serial console, which shows the whole boot and works whatever state the system is in. **Secure Boot.** On this platform the card's firmware does not currently enforce the signature on the SynapseOS kernel image, so units are delivered with the card's Secure Boot off. The integrity check on the system itself does not depend on it and holds as on every other platform; see [Security model](./security.mdx#the-boot-chain). A unit that manages a BlueField-3 from an arm64 host runs SynapseOS on the host instead, with the tools for the card included. ## What is the same - `synapseos-ab-update`, channels and rollback: see [Updating](./updates.mdx). - `synapseos-synapse`: see [Synapse versions](./synapse-versions.mdx). - The base firewall, logs and Synapse's configuration: see [Configuring the appliance](./configure.mdx). One thing to know on any platform: a service that a future update introduces arrives installed but not enabled. See [Known limitations](./limitations.mdx). --- # Running the proxy Source: https://docs.gen0sec.com/docs/synapseos/proxy # Running the proxy *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* **The proxy is already on the unit. Give it its own configuration before you start it.** ## What ships Synapse is one binary with two modes. The agent (`synapse-agent.service`) runs on every appliance; the proxy (`synapse-proxy.service`) is installed beside it and left disabled. Nothing needs installing. :::warning[Do not start the shipped unit as it is] As shipped, `synapse-proxy.service` reads the same `/etc/synapse/config.yaml` as the agent. One file holds one `mode`, one `firewall` block and one `capture` block, so a proxy started that way re-attaches the kernel packet filter on the external interface and takes enforcement away from the agent. Both services keep reporting healthy while that happens. The steps below give the proxy a configuration of its own first. ::: ## Derive a configuration for the proxy Start from the agent's file and flip only what has to differ. The agent keeps packet capture, detection and kernel enforcement; the proxy keeps everything else, so a later Synapse release that adds settings reaches both files. ```bash python3 - <<'EOF' import os, yaml c = yaml.safe_load(open("/etc/synapse/config.yaml")) def section(name): if not isinstance(c.get(name), dict): # absent, or present but empty c[name] = {} return c[name] c["mode"] = "proxy" section("capture")["enabled"] = False # the agent owns packet capture section("ids")["enabled"] = False # the agent owns detection section("firewall")["mode"] = "none" # the agent owns the kernel filter # Created with its final mode: the file carries platform.api_key. fd = os.open("/etc/synapse/config-proxy.yaml", os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o640) with os.fdopen(fd, "w") as f: yaml.safe_dump(c, f, default_flow_style=False, sort_keys=False) EOF ``` Rerun this after changing the agent's file, so the two do not drift apart. ## Point the service at it ```ini title="/etc/systemd/system/synapse-proxy.service.d/10-separate-config.conf" [Service] ExecStart= ExecStart=/usr/bin/synapse --daemon --daemon-pid-file /run/synapse-proxy.pid --config /etc/synapse/config-proxy.yaml ``` The empty `ExecStart=` line clears the shipped command before the new one is set; systemd appends otherwise. ## Say where traffic goes The proxy routes by hostname, from `/etc/synapse/upstreams.yaml`. A minimal file with one host, TLS terminated on the appliance: ```yaml title="/etc/synapse/upstreams.yaml" version: 2 hosts: "app.example.com": tls: terminate: cert: "/etc/synapse/certs/default.crt" key: "/etc/synapse/certs/default.key" upstream: "10.0.0.5:443" ``` Every terminating host needs the `tls.terminate` block, even one only ever served over plain HTTP. A file that does not parse leaves the proxy with no routes at all, so every host answers 404, not only the broken one. Passthrough hosts, weighted backends, health checks and the rest of the format are on the [Upstreams & Routing](../client/upstreams.mdx) page. ## Provide a certificate If `/etc/synapse/certs/` holds your certificate already, skip this. Otherwise a self-signed placeholder lets the TLS listener start; clients will not trust it, so replace it with a real one. ```bash mkdir -p /etc/synapse/certs openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \ -keyout /etc/synapse/certs/default.key -out /etc/synapse/certs/default.crt \ -subj "/CN=default.$(hostname)" chmod 0640 /etc/synapse/certs/default.key ``` ## Open the ports The proxy listens on ports 80 and 443 by default (`proxy.address_http` and `proxy.address_tls`), and the base firewall drops inbound traffic to both until told otherwise. Add them to the `services` chain and reload it; see [The base firewall](./configure.mdx#the-base-firewall). ```text tcp dport { 80, 443 } accept comment "synapse proxy" ``` ## Start it ```bash systemctl daemon-reload systemctl enable --now synapse-proxy systemctl status synapse-proxy synapse-agent ``` Both services should be active. The agent still owns the packet filter; the proxy serves the hosts in `upstreams.yaml`, and reloads that file whenever it changes. :::note[Written out from the installer] These are the steps Gen0Sec's own configuration tooling performs, written out for a unit you manage by hand. The upstreams format shown is the one Synapse 0.8.5 and later read. ::: --- # Configuration reference Source: https://docs.gen0sec.com/docs/synapseos/reference # Configuration reference *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* **Everything you may change is in `/etc`. This is the list.** Files not on this page are the system's, and the system is read-only. The pages that explain each file are linked from its row; this page is the lookup. ## Files and keys ### `/etc/synapseos/update.conf` Read by `synapseos-ab-update`, which sources it as shell. See [Updating](./updates.mdx). | Key | Default | Effect | |---|---|---| | `UPDATE_BASE_URL` | unset | The HTTPS base the unit's channels are published under, specific to your platform. Unset: no channel, offline updates only. | | `UPDATE_CHANNEL` | `stable` | Which channel the unit follows. `canary` by convention for a few units that take each release first. | | `export https_proxy` | unset | An HTTP proxy for the update host; `export` is required. See [Network requirements](./network.mdx#behind-an-http-proxy). | ### `/etc/systemd/network/*.network` Read by `systemd-networkd`; the first file whose `[Match]` fits an interface applies, in name order. `90-fallback-dhcp.network` ships and runs DHCP on `en*` and `eth*`; a file numbered below 90 wins. See [Configuring the appliance](./configure.mdx#network). | Key | Section | Effect | |---|---|---| | `MACAddress=` | `[Match]` | Which interface, by MAC, so a rename does not matter | | `Address=` | `[Network]` | Address with prefix length, repeatable | | `Gateway=` | `[Network]` | Default route | | `DNS=` | `[Network]` | Resolver, repeatable | | `IPv6AcceptRA=` | `[Network]` | `yes` to take router advertisements, `no` to ignore them | ### `/etc/nftables/synapseos-base.nft` Loaded by `synapseos-firewall.service` before the network comes up. Inbound is dropped except what the `services` chain accepts; loopback, established traffic and ICMP are accepted above it. See [The base firewall](./configure.mdx#the-base-firewall). | Line | Ships as | Effect | |---|---|---| | `tcp dport 22 accept comment "ssh"` | present | SSH | | `tcp dport { 80, 443 } accept comment "synapse proxy"` | absent | Add when the proxy runs | ### `/etc/axosyslog/conf.d/*.conf` Drop-ins for the log forwarder. One `destination` block and one `log` line per collector. See [Logs](./configure.mdx#logs). | Element | Values | Effect | |---|---|---| | `transport()` | `"tls"`, `"tcp"`, `"udp"` | TLS verifies the collector against `/etc/ssl/certs` | | `port()` | `6514` for TLS by convention, `514` otherwise | | | `disk-buffer(disk-buf-size(N) reliable(no) dir("/var/lib/axosyslog"))` | 268435456 as shipped | How much is held while the collector is unreachable | ### `/etc/systemd/journald.conf.d/10-synapseos.conf` Ships with `Storage=persistent`, `SystemMaxUse=4G`, `SystemKeepFree=2G`, `MaxRetentionSec=2week`, `RateLimitBurst=20000`. Change retention here; forward logs rather than growing it. ### `/etc/synapse/` | File | Effect | |---|---| | `config.yaml` | Synapse's configuration; keys on the [Configuration](../client/configuration.mdx) page. `platform.api_key` is the one every unit needs. | | `version` | The Synapse version that runs, one line. Written by `synapseos-synapse use`; absent means the newest. See [Synapse versions](./synapse-versions.mdx). | | `config-proxy.yaml` | The proxy's configuration, derived from `config.yaml`. Exists only where the proxy runs. See [Running the proxy](./proxy.mdx). | | `upstreams.yaml` | The proxy's routes, hot-reloaded. Format on the [Upstreams & Routing](../client/upstreams.mdx) page. | | `certs/` | TLS certificates and private keys the proxy serves | ### `/etc/systemd/system/synapse-proxy.service.d/*.conf` Drop-ins that point the proxy at its own configuration. `10-separate-config.conf` as on [Running the proxy](./proxy.mdx#point-the-service-at-it). ### `/etc/keepalived/keepalived.conf` The pair configuration, written at delivery. See [High availability](./high-availability.mdx). | Setting | Effect | |---|---| | `interface` | The leg the election runs on | | `virtual_router_id` | The pair's id; also the last byte of the virtual MAC | | `priority` | Higher wins while healthy | | `nopreempt` | A recovered unit does not take the address back | | `use_vmac` | The address lives on `vrrp.` with a fixed MAC | | `unicast_peer { }` | The other unit's address on the election leg | | `virtual_ipaddress { }` | The floating address | | `track_interface { }` | The leg whose failure withdraws this unit. Present only when the address lives on a different leg from the election; when they share a leg, that leg's failure ends the advertisements by itself | ### Identity and trust | File | Effect | |---|---| | `/etc/hostname` | The unit's name, set with `hostnamectl set-hostname` | | `/root/.ssh/authorized_keys` | Who may log in | | `/etc/ssl/certs/` | The trust store; your CA goes here, then `openssl rehash` | | `/etc/systemd/timesyncd.conf` | `NTP=` servers, once `systemd-timesyncd` is enabled | ## Reference configurations ### A single unit, as delivered Nothing beyond the first credential and the API key. ```yaml title="/etc/synapse/config.yaml" platform: api_key: "" ``` ```bash title="/etc/synapseos/update.conf" UPDATE_BASE_URL= UPDATE_CHANNEL=stable ``` Addressing by DHCP on every `en*` or `eth*` interface; inbound SSH only; logs kept locally for two weeks. ### A single unit with static addressing and log forwarding The delivered files above, plus: ```ini title="/etc/systemd/network/30-external.network" [Match] MACAddress=52:54:00:12:34:56 [Network] Address=203.0.113.10/24 Gateway=203.0.113.1 DNS=203.0.113.53 ``` ```text title="/etc/axosyslog/conf.d/10-remote.conf" destination d_remote { syslog("logs.example.com" transport("tls") port(6514) tls(ca-dir("/etc/ssl/certs") peer-verify(required-trusted)) disk-buffer(disk-buf-size(268435456) reliable(no) dir("/var/lib/axosyslog"))); }; log { source(s_journal); destination(d_remote); }; ``` Outbound: the [Network requirements](./network.mdx#outbound) table plus the collector. ### A pair Each unit has the single-unit files, with its own addresses on both legs, and the pair configuration written at delivery with the same `virtual_router_id`, `virtual_ipaddress` and `nopreempt` on both, a different `priority`, and each other's internal address as the `unicast_peer`. Inbound VRRP on the internal leg. Everything else identical on both units. See [High availability](./high-availability.mdx). --- # Security model Source: https://docs.gen0sec.com/docs/synapseos/security # Security model *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* **Nothing runs on the unit that Gen0Sec did not sign; whether firmware enforces that is the platform's setting.** ## The boot chain Each link checks the next, and the chain starts in firmware. | Link | What it checks | |---|---| | The firmware | The kernel image's signature, against the Gen0Sec certificate enrolled in its Secure Boot key store | | The kernel image | Kernel, initrd and command line are one signed file. The command line carries the hash of the system (`/usr`) this kernel belongs to | | dm-verity | Every block of `/usr` is checked against that hash as it is read. A changed block is a read error, not a silently different file | | Kernel lockdown | In integrity mode: no unsigned kernel modules, no `kexec`, no access to kernel memory from userspace | So a modified binary, library or module cannot be run, a different kernel cannot be booted, and the kernel command line cannot be edited at the boot menu. If an update does not verify completely, it is never the system that boots. :::note[Firmware Secure Boot state] Whether the first link is enforced depends on the platform's firmware setting: a server or VM can have Secure Boot off, and BlueField-3 units are delivered with it off (see [Platforms](./platforms.mdx#nvidia-bluefield-3)). With it off, the signature on the kernel image is not checked by firmware. The dm-verity check on `/usr` and the kernel's own restrictions do not depend on that setting and hold either way. ::: ## On the running system - **Hardened units run a jailer**: a kernel-level policy that confines Synapse and the system services to what they need. The policy is part of the sealed system; there are no local overrides, and the directory where overrides would go is empty, read-only and audited. - **Inbound traffic is dropped by default**, except SSH. See [The base firewall](./configure.mdx#the-base-firewall). - **SSH accepts root with a key only.** Passwords work on the console only, and only where one has been set. See [Logging in](./access.mdx). - **Changes are audited.** `auditd` records changes to configuration, credentials and the system's own files; the journal is persistent and bounded. See [Logs](./configure.mdx#logs). - **There is no package manager** and no compiler. The unit cannot install software, and no tool on it can modify `/usr`. ## What leaves the unit | Flow | To | What it carries | |---|---|---| | Platform traffic | `api.gen0sec.com`, 443 | Registration, detections and the telemetry Synapse is configured to send; access rules, fleet bans and verdicts coming back. `platform.telemetry_sending_enabled` and `platform.include_response_body` on the [Configuration](../client/configuration.mdx) page say how much of a request leaves | | Database downloads | `download.gen0sec.com` and the GeoIP host, 443 | Nothing outbound beyond the request; indicator, threat and GeoIP databases inbound | | Updates | The host in `UPDATE_BASE_URL`, 443 | Nothing outbound beyond the request; the signed archive inbound, and only when you run `check` or `update` | | Logs | Your collector | Everything in the journal, which includes Synapse's events; only if you configured forwarding | Nothing else opens a connection, and nothing connects in except SSH and, where you run it, the proxy. The full host and port list is on [Network requirements](./network.mdx). The audit rule set records changes to the files on the [Configuration reference](./reference.mdx), to credentials, to the system's own files, and to the jailer's override directory; it does not record traffic. ## Before an update is installed An update is accepted only if its manifest and the channel pointer both carry a valid signature from the Gen0Sec release key the unit already trusts, every file's digest matches the manifest, the version is newer than the one running, and the update is of the same variant as the unit. The new kernel image is then checked by firmware at boot like any other, and the new `/usr` by dm-verity. Details and commands are on [Updating](./updates.mdx). ## What is not covered The appliance protects the integrity of what runs on it. It does not encrypt data at rest: configuration, keys and logs on the writable partition are readable from a disk removed from the unit. Treat disposal and transport of a unit accordingly. See [Known limitations](./limitations.mdx). ## Reporting a vulnerability If you believe you have found a security vulnerability in SynapseOS or any Gen0Sec software, report it privately to [support@gen0sec.com](mailto:support@gen0sec.com). Do not open a public issue. Include the affected component and version, the configuration, and steps or a proof of concept that trigger the issue. Sensitive details can be encrypted to the Gen0Sec package-signing key, fingerprint `C1631499081A7146332D2038798FF49A6E9DA146`. We will acknowledge the report, investigate, and keep you informed of the resolution. Please allow a reasonable opportunity to release a fix before any public disclosure. --- # Serial console Source: https://docs.gen0sec.com/docs/synapseos/serial-console # Serial console *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* **115200 baud, 8 data bits, no parity, 1 stop bit. Everything else is finding the port.** A Cerebrum Edge has no BMC; its console is a serial port behind a USB connector on the unit. Connect it to a laptop with the USB cable supplied and the console appears as a serial device on the laptop. It shows the whole boot, from firmware to login prompt, and works whatever state the system is in, so it is the way in when the network is not. The unit's password, where one is set, works here; see [Logging in](./access.mdx#the-console). The USB-to-serial converter in the cable is a Prolific PL2303. Linux and current Windows have its driver built in; macOS needs it installed. ## Windows With the cable connected and the unit powered on, open **Device Manager** and expand **Ports (COM & LPT)**: the unit is the new **USB Serial Port**, with a COM number you will need. In this example it is COM9. ![Device Manager showing a USB Serial Port on COM9](/img/synapseos/serial-windows-1.png) The screenshot shows a port whose manufacturer reads FTDI; a Cerebrum Edge's cable identifies as a Prolific PL2303 and appears in the same place, under the same heading. Then open [PuTTY](https://www.chiark.greenend.org.uk/~sgtatham/putty/), choose **Serial** as the connection type, enter that COM port as the serial line and `115200` as the speed, and open the session. ![PuTTY configured for a serial connection to COM9 at 115200](/img/synapseos/serial-windows-2.png) If Windows does not recognise the cable, install the PL2303 driver from Prolific's site, then unplug and reconnect it. ## Linux The driver is in the kernel. After connecting the cable, find the device the converter was given: ```bash dmesg | grep -i tty ``` It appears as `/dev/ttyUSB0` when it is the only USB serial device; the examples below use that name, so check yours. Then, with either program: ```bash sudo tio -b 115200 /dev/ttyUSB0 # leave with Ctrl+t, then q sudo minicom -b 115200 -D /dev/ttyUSB0 # leave with Ctrl+a, then x ``` `sudo` is needed unless your user is in the group that owns the device (`dialout` on most distributions). ## macOS Install the [PL2303 Serial driver](https://apps.apple.com/app/pl2303-serial/id1624835354) from the App Store first; without it the cable does not appear at all. Then find the device; with that driver it is named after the chip, not `usbserial`: ```bash ls -l /dev/tty.* # /dev/tty.PL2303G-USBtoUARTnnn, or tty.PL2303-… on older cables ``` and connect with `screen`, which macOS ships, or with `tio` from Homebrew: ```bash screen /dev/tty.PL2303G-USBtoUARTnnn 115200 # leave with Ctrl+a, then k, then y tio -b 115200 /dev/tty.PL2303G-USBtoUARTnnn # leave with Ctrl+t, then q ``` Use the name `ls` showed. A graphical terminal such as ZTerm works as well: set the connection to 115200 8 N 1 with local echo off. ## If nothing appears - Press Enter once: the console only prints when something happens, and an idle unit at a login prompt is silent. - Check the speed. Any setting other than 115200 shows nothing, or unreadable characters. - Check the port. A laptop with another serial device attached gets a different number or name for this one. - On a unit that is powering on, the firmware's output comes first, before SynapseOS prints anything. If that never appears, the problem is the cable, the driver or the port, not the unit. --- # Synapse versions on the appliance Source: https://docs.gen0sec.com/docs/synapseos/synapse-versions # Synapse versions on the appliance *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* **Several releases on the unit; one of them runs.** ## How it works The system is read-only, so the set of Synapse versions on a unit is fixed by the image it runs. What is not fixed is which one is used: that choice lives in `/etc`, which is writable. ```text /usr/lib/synapse/0.8.5/synapse one directory per version, read-only /usr/lib/synapse/0.8.6/synapse /usr/bin/synapse reads the selection and runs that version /etc/synapse/version the selection: one line, writable ``` With no selection made, the newest version on the unit runs. A selection is called a **pin**, and it stays in place across OS updates. ## Commands ```bash synapseos-synapse list # versions on the unit, and which one runs synapseos-synapse use 0.8.6 # select a version and restart the active services synapseos-synapse status # the selection, against what each process is running synapseos-synapse reset # drop the pin; the newest version is selected ``` `use` refuses a version that is not on the unit, and runs the chosen binary once before selecting it, so a release that cannot start on this kernel fails there rather than after the running service has been stopped. If the restart fails anyway, the selection is put back automatically. `reset` only removes the pin. Unlike `use`, it restarts nothing: run `systemctl restart synapse-agent` (and `synapse-proxy`, if it runs) afterwards, or `use` the newest version instead, which restarts for you. `status` matters after any change: a running process keeps the binary it started with, so a pin changed without a restart is otherwise invisible. ## What switching does not do - **It does not change your configuration.** `/etc/synapse/config.yaml` is seeded once and left alone. Each version's reference configuration stays readable at `/usr/share/synapse//config.yaml`, so a release that changes a setting can be compared against the file you have. See [Configuring the appliance](./configure.mdx#synapse). - **It does not touch the OS.** Going back to a previous Synapse version is a restart and leaves the other OS slot free for [rolling back the OS itself](./updates.mdx#going-back). ## Getting a version that is not on the unit A release that is not in the image cannot be selected, only shipped. It arrives with the next OS update, beside the versions already there, and the pin you have is kept. See [Updating](./updates.mdx#synapse-versions-and-os-updates). --- # Troubleshooting Source: https://docs.gen0sec.com/docs/synapseos/troubleshooting # Troubleshooting *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* **Find the message you see; each entry says what caused it and what to do.** ## Where the logs are The journal holds every service's output. Two things are written elsewhere: the audit records, in `/var/log/audit/audit.log` (`auditd` rotates it itself), and the syslog files the forwarder also writes locally, `/var/log/messages`, `auth.log`, `kern.log` and `daemon.log`. ```bash journalctl -b # this boot, everything journalctl -b -1 # the previous boot: what happened before a rollback or reboot journalctl -u synapse-agent -f # the data plane, live journalctl -u synapseos-firewall -u systemd-networkd -b ausearch -ts boot # audit records since boot, from /var/log/audit/audit.log ``` `synapseos-ab-update status` and `synapseos-synapse status` are the two commands to run before anything else when a unit misbehaves after an update. ## Boot ### `device-mapper: verity: ... data block N is corrupted` then `Timed out waiting for device /dev/mapper/usr` **Cause.** The system partition's content does not match its hash tree. On a unit you imaged yourself this is almost always a writer that skipped blocks it thought were empty (balenaEtcher's "trim unallocated space" does this); on a delivered unit it is a failing disk. **Fix.** A unit you imaged: write the image again with a tool that writes every block (`dd` from the decompressed image). A delivered unit: `synapseos-ab-update rollback` from the console returns to the other slot if it still verifies; if both fail, the disk is the problem, contact support. ### `A start job is running for Wait for Network to be Online` for two minutes **Cause.** A managed interface has no link or no address, and `systemd-networkd-wait-online` waits for every managed interface. The unit boots fine afterwards; only the wait is lost. **Fix.** Cable or configure every interface that `networkctl` lists as `configuring`, or leave it: the delay is bounded and nothing depends on it. ### The unit boots but has no address **Cause.** No `.network` file matched the interface. The default runs DHCP on interfaces named `en*` and `eth*` only; an interface with another name, or one you gave a `.network` file with a wrong `MACAddress=`, stays unconfigured. **Fix.** On the console, `networkctl status` shows which file applied to which interface (`unmanaged` means none). Match on the MAC the console shows, then `networkctl reload`. See [Configuring the appliance](./configure.mdx#network). ### `synapseos-ab-update status` shows the previous slot running after an update and a reboot **Cause.** The unit rolled back: the new slot did not confirm within its trial boots. The login banner and `/etc/os-release` do not tell you this; they name the build the unit was delivered with and are not rewritten by updates. `status` and `/usr/lib/os-release` name the running one. **Fix.** `journalctl -b -1 -u synapse-agent` shows why the data plane did not come up in the previous boot. See "A new slot keeps rolling back" below. ## Access ### `Permission denied (publickey)` **Cause.** SSH accepts root with a key only, and the key offered is not in `/root/.ssh/authorized_keys`. As shipped, a password is not accepted over SSH; a unit delivered with password SSH turned on carries a drop-in under `/etc/ssh/sshd_config.d/` that says so, and `sshd -T | grep -i passwordauthentication` shows which applies. **Fix.** Use the key delivered with the unit (`ssh -i`), or add yours from the console. See [Logging in](./access.mdx). ### The console asks for a password and refuses every one **Cause.** No password is set. Root is locked until one is. **Fix.** Over SSH, `passwd root`. Set it before changing addressing, since the console is the way back in. See [Logging in](./access.mdx#the-console). ### The serial console shows nothing See [Serial console](./serial-console.mdx#if-nothing-appears): the speed, the port and the driver, in that order. ## Updates ### `no UPDATE_BASE_URL in /etc/synapseos/update.conf -- offline appliance; use: install ` **Cause.** The unit has no channel configured, so `check` and `update` have nowhere to look. **Fix.** Put the URL Gen0Sec gave you for this platform in the file, or update from external media. See [Updating](./updates.mdx#where-updates-come-from). ### `cannot reach ` **Cause.** The update host is not reachable from the unit: no route, a firewall, DNS, or a proxy the updater was not told about. **Fix.** `curl -sS -o /dev/null -w '%{http_code}\n' ` from the unit tells which. A proxy goes in `update.conf` with `export`. See [Network requirements](./network.mdx). ### `bundle serial N is not newer than the running M. Refusing a downgrade` **Cause.** The channel, or the media, holds a version equal to or older than the one running. Equal counts: reinstalling the running version is refused too. **Fix.** If that is what you want, `synapseos-ab-update update --allow-downgrade` (or `install --allow-downgrade`). See [Updating](./updates.mdx#installing-over-the-network). ### `bundle is variant 'X' but this appliance runs 'Y'` **Cause.** The update was built for a different variant (hardened versus baseline). Installing it would change the unit's security posture, so the updater refuses, and the `--allow-downgrade` its message mentions does not override this check. **Fix.** Check `UPDATE_BASE_URL` against what you were given for this platform; this usually means the wrong channel. A unit that is meant to change variant is re-imaged. ### `SIGNATURE VERIFICATION FAILED` or `the UKI is not signed by any certificate this device trusts` **Cause.** Either the archive was damaged in transit, or it was signed with a key this unit does not trust: a build for another platform, or a unit whose trusted keys predate a key rotation. **Fix.** Compare the archive's checksum with the one published beside it. If it matches, the keys are the problem; contact support with the output of `synapseos-ab-update status`, since a unit in this state cannot take an update on its own. ### `slot B FAILED verity verification -- not switching to it` **Cause.** The slot was written but its content does not match the hash tree it came with: a damaged archive, or faulty media. **Fix.** Nothing changed on the unit; the bootloader still points at the running slot. Unpack the archive again from a fresh copy and run `install` again. ### A new slot keeps rolling back **Cause.** The new system booted but Synapse (and the jailer on hardened units) did not reach a running state within 120 seconds, so the slot was never confirmed and the bootloader returned to the previous one after three attempts. **Fix.** `journalctl -b -1 -u synapse-agent` shows what the agent did in the trial boot. The usual causes are a configuration key the new version rejects (compare `/etc/synapse/config.yaml` with `/usr/share/synapse//config.yaml`) and a pinned Synapse version the new image no longer ships. Fix the configuration, then `synapseos-ab-update update` again. ### A service the release notes mention reports `disabled` after the update See [Known limitations](./limitations.mdx#a-service-introduced-by-an-update-starts-disabled): once per unit, `systemctl enable --now `. ## Synapse ### `synapseos-synapse use` refuses the version **Cause.** Either the version is not on the unit (`list` shows what is), or its binary failed the test run `use` performs before switching: a release that cannot start on this image. **Fix.** `list` first. If the version is there and still refused, the message from the test run says why; the running version is untouched. See [Synapse versions](./synapse-versions.mdx). ### `synapse-agent.service: Failed to create reference to PID from file '/run/synapse-agent.pid': Invalid argument` **Cause.** A message systemd prints while the agent writes its PID file; the next line is `Started Synapse`. It does not indicate a failure. **Fix.** None. If the unit is actually not filtering, look at the lines after it. ### The proxy was enabled and the agent stopped filtering **Cause.** `synapse-proxy.service` was started with the shipped unit file, so it read the agent's configuration and took over the kernel packet filter. **Fix.** `systemctl stop synapse-proxy`, then follow [Running the proxy](./proxy.mdx) from the first step: the proxy needs its own configuration before it starts. ### The proxy answers 404 for every host **Cause.** `/etc/synapse/upstreams.yaml` did not parse, so the proxy has no routes at all. One bad host takes every host down. **Fix.** `journalctl -u synapse-proxy -n 50` names the line. Every terminating host needs its `tls.terminate` block. See [Running the proxy](./proxy.mdx#say-where-traffic-goes). ## Collecting diagnostics for support One command gathers what support asks for first, into a file you can send: ```bash D=/var/tmp/diag-$(hostname)-$(date -u +%Y%m%dT%H%M%SZ); mkdir -p "$D" synapseos-ab-update status > "$D/ab-update-status.txt" 2>&1 synapseos-synapse status > "$D/synapse-status.txt" 2>&1 cat /etc/os-release > "$D/os-release.txt" networkctl status > "$D/networkctl.txt" 2>&1 nft list ruleset > "$D/nft.txt" 2>&1 journalctl -b --no-pager > "$D/journal-this-boot.txt" journalctl -b -1 --no-pager > "$D/journal-previous-boot.txt" 2>/dev/null cp /var/log/audit/audit.log /var/log/messages "$D/" 2>/dev/null tar -C /var/tmp -czf "$D.tar.gz" "$(basename "$D")" && rm -r "$D" && echo "$D.tar.gz" ``` Copy it off with `scp`, then delete it; the journal in it may contain addresses from your traffic. Send it to [support@gen0sec.com](mailto:support@gen0sec.com) with the message you saw and when. --- # Updating Source: https://docs.gen0sec.com/docs/synapseos/updates # Updating *Applies to SynapseOS builds from 2026-10-09 (`20261009-141152` and later). See [What changed](./changelog.mdx).* **One update, the whole system, and the old one stays on disk.** ## Where updates come from A unit follows a **channel**, named in `/etc/synapseos/update.conf`: ```bash title="/etc/synapseos/update.conf" UPDATE_BASE_URL= UPDATE_CHANNEL=stable ``` `stable` is the default. A unit delivered with network updates has `UPDATE_BASE_URL` set already; with it unset the unit has no channel and takes updates from external media only. Each platform build has its own channels, so the URL you were given applies to units of that platform and no other. ## Setting up network updates A unit takes updates over the network once `UPDATE_BASE_URL` points at its platform's channels and carries an update key. Updates never run on their own: the unit contacts the update host only when you run `check` or `update`, and you choose when to reboot. **What you need from Gen0Sec:** - the platform name of your units, which is part of the URL; - an **update key**. A key issued for the package repositories does not open the update channels; ask for one for SynapseOS updates. **Write the URL, with the key, into the unit's configuration:** ```bash title="/etc/synapseos/update.conf" UPDATE_BASE_URL=https://:@releases.gen0sec.com/synapseos/ UPDATE_CHANNEL=stable ``` The key travels as the password of the URL, over HTTPS. The user name before the colon is not checked; your organization ID keeps it recognisable. The file holds a credential, so make it readable by root only: ```bash chmod 600 /etc/synapseos/update.conf ``` The file is on the unit's persistent configuration, not on the system an update replaces, so it survives every update and rollback. Behind an HTTP proxy, add the proxy line from [Network requirements](./network.mdx#behind-an-http-proxy). **Prove the path:** ```bash synapseos-ab-update check ``` | Result | Meaning | |---|---| | `=> up to date` or `=> an update is available` | The URL, the key and the channel signature are all good | | `401` in the error | No key in the URL, or a key that does not open the update channels | | `404` in the error | The key is accepted, but the platform or the channel in the URL does not exist | | A signature error | The answer did not come from Gen0Sec unaltered; do not override it, report it | :::note[Keep the key out of what you share] `synapseos-ab-update status` prints the channel's URL in full, key included, and an error from a failed download can too. Before pasting either into a ticket, remove the key: ```bash synapseos-ab-update status 2>&1 | sed -E 's#//[^@/]*@#//***@#' ``` ::: **Choosing a channel.** `stable` is what a production unit follows. Where Gen0Sec has published a `canary` channel for your platform, it receives a release first; point a test unit at it, or take one update from it without changing the file: ```bash synapseos-ab-update update --channel=canary ``` **Units on builds before 2026-10-12** need one correction after `update` and before the reboot; see [Units on builds before 2026-10-12](#units-on-builds-before-2026-10-12). ## Checking ```bash synapseos-ab-update check ``` It prints the version you are running and the version the channel names, then either `=> up to date` or `=> an update is available`. It changes nothing. In a script, note that it exits non-zero when the unit is up to date. ```bash synapseos-ab-update status ``` prints, on a unit that has taken one update and confirmed it: ```text disk : /dev/sda active slot : B (verified) slot A : 20260917155226-6.18.48 serial 20260917155226 kernel 6.18.48-synapseos slot B : 20260918191637-6.18.48 serial 20260918191637 kernel 6.18.48-synapseos <- running channel : stable @ boot entries : synapseos_a.efi synapseos_b.efi loader defaul: synapseos_b.efi ``` | Line | Meaning | |---|---| | `active slot` | Which of the two slots is running. `(verified)` means its system passed the integrity check for this boot; it does not say whether the slot has been confirmed good | | `slot A`, `slot B` | The version, serial and kernel each slot holds; `<- running` marks the active one. A slot never written says `empty (never installed)` | | `channel` | The channel followed and the base it is read from; `unset` means offline updates only | | `boot entries` | The files the bootloader can boot. A slot still on trial carries a counter in its name, `synapseos_b+3.efi`; a confirmed one does not. That counter, not `(verified)`, is what says a slot was confirmed | | `loader defaul` | Which entry boots next | ## Installing over the network ```bash synapseos-ab-update update systemctl reboot synapseos-ab-update status ``` `update` downloads, verifies and writes the update into the inactive slot, then points the bootloader at it. The reboot is yours to schedule: nothing changes until then. Take a [backup of `/etc`](./backup.mdx) before it, as a rule; the update does not touch `/etc`, but a configuration you cannot restore is the one thing a rollback does not give back. After the reboot, confirm the unit is doing its job, not only that it booted: ```bash synapseos-ab-update status # the new version is the running slot, the previous one beside it synapseos-synapse status # every service runs the Synapse version you expect systemctl is-active synapse-agent # active journalctl -u synapse-agent -b -n 20 # "Started Synapse", and nothing after it that says failed ``` On a pair, update the standby first; see [High availability](./high-availability.mdx#updating-a-pair). The update refuses, before writing anything: - an update that is not newer than the running system, the same version included; - an update of a different variant, which would silently change the unit's security posture; - anything whose signature does not verify. `--allow-downgrade` overrides the first check only: reinstalling an older or equal version is a decision, and the flag records it. The variant check cannot be overridden, although the refusal message suggests the same flag; a unit changes variant by being re-imaged. The signature check is never overridden. ## Installing offline, from external media A unit with no network path to updates installs the same update from external media. It needs no `UPDATE_BASE_URL` and no update key; the update carries its own signature, and the unit checks it exactly as it does for a network update. **What you receive from Gen0Sec:** one archive per release and platform, `synapseos--.bundle.tar.zst`, and its checksum file, `synapseos--.bundle.tar.zst.sha256`. The variant (`hardened`, for example) must match the unit's, which `grep -o '"variant": *"[^"]*"' /var/lib/synapseos/slots.json` prints; `install` refuses any other. `synapseos-ab-update status` shows the running version. ### Preparing the media On a Linux machine, with a USB drive of at least 8 GB. Formatting erases the drive, so check the device name with `lsblk` first: ```bash f=synapseos--.bundle.tar.zst echo "$(cut -d' ' -f1 "$f.sha256") $f" | sha256sum -c # ": OK" mkfs.ext4 -L SYNAPSEOS-UPD /dev/sdX1 # the drive's partition, not the machine's disk mount /dev/sdX1 /mnt/usb mkdir /mnt/usb/update tar --zstd -xf synapseos--.bundle.tar.zst -C /mnt/usb/update ls /mnt/usb/update # manifest.json manifest.json.asc roothash uki.efi usr.hash.img usr.img umount /mnt/usb ``` Use ext4. The system image, `usr.img`, is a single 5 GiB file, which FAT32 cannot hold. Unpack **on the media, or on the machine preparing it**, never on the unit: the appliance's root filesystem is small, and the installer streams the update from the media straight into the inactive slot. The archive unpacks its files into the directory you name, so give it one of its own. ### Installing ```bash synapseos-ab-update status # note the running version mkdir -p /mnt/media mount -o ro /dev/disk/by-label/SYNAPSEOS-UPD /mnt/media synapseos-ab-update install /mnt/media/update umount /mnt/media systemctl reboot ``` `install` takes the directory that holds `manifest.json`. Read-only media is fine; nothing is written to it. Before writing anything it checks the signature on the manifest, that every file matches the manifest, and that the kernel image is signed. It refuses an update that is not newer, or of a different variant, just as `update` does ([Installing over the network](#installing-over-the-network)). On a unit running a build from before 2026-10-12, correct the bootloader entry after `install` and before the reboot; see [Units on builds before 2026-10-12](#units-on-builds-before-2026-10-12). After the reboot, run the same checks as after a network update. The new slot boots on trial and falls back by itself if it does not come up healthy; see below. Remove the media once the unit is confirmed. ## Units on builds before 2026-10-12 The updater on these builds marks the new slot as the bootloader's `default`. The bootloader does not count trial boots of a `default` entry, so if the new slot fails, the unit does not fall back to the old one by itself. After `update` or `install`, and before the reboot, mark the entry `preferred` instead. Do it only on a bootloader that understands `preferred`, which is systemd-boot 260 or later: ```bash bootctl status | grep 'Product: systemd-boot' # needs 260 or later grep -E '^(default|preferred) ' /efi/loader/loader.conf sed -i -E 's/^default (synapseos_[ab])(\.efi)?$/preferred \1.efi/' /efi/loader/loader.conf grep -E '^(default|preferred) ' /efi/loader/loader.conf # preferred synapseos_.efi ``` If the version is below 260, leave `default` in place and contact [support@gen0sec.com](mailto:support@gen0sec.com): on an older bootloader, an entry marked `preferred` is ignored, and the unit would boot whichever slot sorts first. After the change, `synapseos-ab-update status` on these builds shows an empty `loader defaul:` line until the reboot. That is expected: the old updater reads only `default` lines, so trust the `grep` above instead. The 2026-10-12 build you update to writes `preferred` itself from then on. ## What happens on the first boot after an update The new slot boots as a **counted** entry. If the kernel does not come up, or the unit boots but Synapse (and the jailer, on hardened units) does not reach a running state, the bootloader stops trying after three attempts and returns to the previous slot by itself. Only once the appliance's services are up is the new slot confirmed good: the counter disappears from its boot entry, and from then on it boots like any other slot. A boot that fails for any other reason later, once a slot is confirmed, is not caught by this: the safety net exists for the trial boots after an update. ## Going back If the new version runs but behaves worse than the old one: ```bash synapseos-ab-update rollback systemctl reboot ``` The previous slot is still on disk, exactly as it was. Rolling back the OS does not change anything in `/etc`. ## What an update is A SynapseOS update is not a set of packages. It is a complete new system (`/usr`, with every binary, library and kernel module) together with the kernel that matches it, signed as one. The unit holds two copies of the system, called slots A and B. An update is written into the slot that is not running, verified in place, and only then made the one to boot next. The running slot is never touched, which is what makes going back possible. Two signatures are checked before anything is written, and a third on every boot: | Signature | Covers | Checked by | |---|---|---| | The update's manifest | Its version and the digest of every file in it | The updater, before writing | | The channel pointer | Which update the channel currently names | The updater, on every check | | The kernel image | Kernel, initrd and command line, which carries the hash of the new system | The firmware, at every boot | Hosting adds nothing to this: an update served from a mirror you do not trust can be withheld, but not forged. ## Synapse versions and OS updates An update brings new Synapse versions with it, but does not change which one runs. A unit with a version pinned keeps it, as long as the new image still carries that version; one with no pin runs the newest. If a pinned version is no longer shipped, the newest runs instead, and `synapseos-synapse status` is where that shows. After the update: ```bash synapseos-synapse list synapseos-synapse use ``` Switching Synapse versions is a service restart, not an OS update, and it does not use the other slot, which stays free for rolling back the OS itself. See [Synapse versions on the appliance](./synapse-versions.mdx). --- # Overview Source: https://docs.gen0sec.com/docs/cerebellum/ # Overview ## What is Cerebellum? Cerebellum is the enterprise version of the Gen0Sec platform that you can host in your own Kubernetes cluster. ## Charts | Installation order | Chart | Namespace | What it installs | |---|---|---|---| | 1. | `gen0sec-infra` | `gen0sec-system` | Three operators (Postgres, Kafka, ingress), the object store, and two caches: one shared by the application, one for the ingress data plane | | 2. | `gen0sec-data` | `gen0sec-system` | The Postgres cluster, the Kafka cluster, topics and database roles | | 3. | `gen0sec-platform` | `gen0sec` | Schema migrations, then the application services and the dashboard | The installation order is mandatory, as each chart creates resources the next one depends on. ## Installation methods ### Installation from registry (Recommended) Choose it when, - your cluster can access our registry, - you don't mind depending on an external registry ### Install via creating an offline bundle Choose it when, - your cluster don't have access to our registry, - you'd rather host mirrored images yourself, - you're facing issues pulling images from our registry :::note[Same content] Worry not, the offline bundle builds the same release and installs the same charts. ::: If you are unsure, go with **Install from registry**. You can always switch later. :::info[One thing that can decide this for you] Our registry challenges pulls that arrive with no `User-Agent`, and on some Kubernetes distributions the kubelet's pull path sends none. That is deliberate policy on our side, and the result is a 403 the kubelet cannot recover from. If you cannot change container runtime configuration on your nodes, use **Create an offline bundle**. Your own registry does not inspect user agents. See [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: You will need to decide the installation method in the [Prepare the artifacts](./04-prepare-artifacts.mdx) step. ## Connectivity :::warning[What 'offline' means] The 'offline' part in the offline bundler is used to describe the installation method, not the resulting product. ::: It's important to note that even by creating an offline bundle, doesn't mean you can operate the platform in an airgapped cluster that is entirely isolated from reaching the public internet. Cerebellum needs exactly one outbound destination from your cluster: - `api.gen0sec.com` on port `443`. That single connection carries **threat intelligence, GeoIP databases, ML models and IDS rules**. All four are licensed by their upstream providers and are not included in any of the artifacts we ship. Every agent in your fleet reaches it through one in-cluster service, so it is one destination to allow and one to audit. Nothing else in either namespace needs a route out. The dashboard can optionally check for new versions and send email; see [Network and connectivity](./03-network.mdx). Read [Network and connectivity](./03-network.mdx) before you start. It is the page to send to whoever runs your firewall. ## How long it takes Set aside **half a day for a first install**, and about an hour once you have done one. The install is 26 numbered steps across five pages, and most of the elapsed time is spent waiting on image pulls and on two clusters becoming ready, not on typing. The waits are gated explicitly, so the ceiling is knowable. These are the timeouts the commands themselves carry — the upper bound before a step is treated as failed, not a prediction: | Stage | You are waiting on | Gate | |---|---|---| | [Prepare the artifacts](./04-prepare-artifacts.mdx) | Downloads and signature verification. From the registry this is minutes; building an offline bundle copies every image and is the longest single stage | None | | [Install](./05-install.mdx) infra | Three operators and the object store | None | | [Install](./05-install.mdx) data | Kafka ready, then Postgres `Running` | `15m` each | | [Install](./05-install.mdx) platform | Schema migrations, then every pod ready | `20m` + `10m` | | [Publish](./06-publish.mdx) | The ingress rollout | `20m` | | [First sign-in](./07-first-signin.mdx) | You: MFA enrolment, a forced password change, the onboarding wizard | None | | [Verify](./08-verify.mdx) | Four checks you run | None | What actually decides your number is how fast your nodes pull images. Everything else is bounded by the table above. :::note[The Quickstart is not a shortcut] It is the same procedure with one extra values file per chart, on one node. It is quicker because a single-node cluster pulls and schedules less — not because it skips steps. ::: ## Before you install 1. [Requirements](./02-requirements.mdx). Cluster, tools, and your access token. 2. [Network and connectivity](./03-network.mdx). What to allow, and why. 3. [Prepare the artifacts](./04-prepare-artifacts.mdx). Prepare for the installation. If you want a working instance to look at before you plan a real install, use the [Quickstart](./quickstart.mdx). It builds a throwaway single-node install and tears it down again. --- # Overview Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/ # Overview ## What is Cerebellum? Cerebellum is the enterprise version of the Gen0Sec platform that you can host in your own Kubernetes cluster. ## Charts | Installation order | Chart | Namespace | What it installs | |---|---|---|---| | 1. | `gen0sec-infra` | `gen0sec-system` | Three operators (Postgres, Kafka, ingress), the object store, and two caches: one shared by the application, one for the ingress data plane | | 2. | `gen0sec-data` | `gen0sec-system` | The Postgres cluster, the Kafka cluster, topics and database roles | | 3. | `gen0sec-platform` | `gen0sec` | Schema migrations, then the application services and the dashboard | The installation order is mandatory, as each chart creates resources the next one depends on. ## Installation methods ### Installation from registry (Recommended) Choose it when, - your cluster can access our registry, - you don't mind depending on an external registry ### Install via creating an offline bundle Choose it when, - your cluster don't have access to our registry, - you'd rather host mirrored images yourself, - you're facing issues pulling images from our registry :::note[Same content] Worry not, the offline bundle builds the same release and installs the same charts. ::: If you are unsure, go with **Install from registry**. You can always switch later. :::info[One thing that can decide this for you] Our registry challenges pulls that arrive with no `User-Agent`, and on some Kubernetes distributions the kubelet's pull path sends none. That is deliberate policy on our side, and the result is a 403 the kubelet cannot recover from. If you cannot change container runtime configuration on your nodes, use **Create an offline bundle**. Your own registry does not inspect user agents. See [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: You will need to decide the installation method in the [Prepare the artifacts](./04-prepare-artifacts.mdx) step. ## Connectivity :::warning[What 'offline' means] The 'offline' part in the offline bundler is used to describe the installation method, not the resulting product. ::: It's important to note that even by creating an offline bundle, doesn't mean you can operate the platform in an airgapped cluster that is entirely isolated from reaching the public internet. Cerebellum needs exactly one outbound destination from your cluster: - `api.gen0sec.com` on port `443`. That single connection carries **threat intelligence, GeoIP databases, ML models and IDS rules**. All four are licensed by their upstream providers and are not included in any of the artifacts we ship. Every agent in your fleet reaches it through one in-cluster service, so it is one destination to allow and one to audit. Nothing else in either namespace has a route out. Read [Network and connectivity](./03-network.mdx) before you start. It is the page to send to whoever runs your firewall. ## How long it takes Set aside **half a day for a first install**, and about an hour once you have done one. The install is 26 numbered steps across five pages, and most of the elapsed time is spent waiting on image pulls and on two clusters becoming ready, not on typing. The waits are gated explicitly, so the ceiling is knowable. These are the timeouts the commands themselves carry — the upper bound before a step is treated as failed, not a prediction: | Stage | You are waiting on | Gate | |---|---|---| | [Prepare the artifacts](./04-prepare-artifacts.mdx) | Downloads and signature verification. From the registry this is minutes; building an offline bundle copies every image and is the longest single stage | None | | [Install](./05-install.mdx) infra | Three operators and the object store | None | | [Install](./05-install.mdx) data | Kafka ready, then Postgres `Running` | `15m` each | | [Install](./05-install.mdx) platform | Schema migrations, then every pod ready | `20m` + `10m` | | [Publish](./06-publish.mdx) | The ingress rollout | `20m` | | [First sign-in](./07-first-signin.mdx) | You: MFA enrolment, a forced password change, the onboarding wizard | None | | [Verify](./08-verify.mdx) | Four checks you run | None | What actually decides your number is how fast your nodes pull images. Everything else is bounded by the table above. :::note[The Quickstart is not a shortcut] It is the same procedure with one extra values file per chart, on one node. It is quicker because a single-node cluster pulls and schedules less — not because it skips steps. ::: ## Before you install 1. [Requirements](./02-requirements.mdx). Cluster, tools, and your access token. 2. [Network and connectivity](./03-network.mdx). What to allow, and why. 3. [Prepare the artifacts](./04-prepare-artifacts.mdx). Prepare for the installation. If you want a working instance to look at before you plan a real install, use the [Quickstart](./quickstart.mdx). It builds a throwaway single-node install and tears it down again. --- # Component versions Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/configure/components # Component versions :::note[Read this from your release, not from this page] Every release pins its own versions, and the authoritative list is in the install kit you downloaded. ```bash yq '.spec.images[] | [.image, .imageTag, .digest] | @tsv' $KIT/release-manifest.yaml yq '.spec.charts[] | [.chart, .version, .digest] | @tsv' $KIT/release-manifest.yaml ``` This page describes what those versions mean and which of them have surprising requirements. ::: {/* This page should be generated from release/components.yaml and release-manifest.yaml in CI, per the structure proposal section 7. Hand-maintained, it drifts within one release and is then worse than nothing, because operators trust it. */} ## Third-party components | Component | Version | Notes | |---|---|---| | Postgres operator | 2.0.1 | Zalando. **1.15.x cannot be used**: its schema rejects PostgreSQL 18 | | PostgreSQL | 18, with extensions | Includes AGE, TimescaleDB, `pg_cron`, `pg_net`, `pglogical`, `pgsodium` | | Connection pooler | 2.0.1 | Operator-launched. Services connect through it, never directly | | Logical backup | 2.0.1 | Only used when logical backups are enabled | | Kafka operator | 1.0.0 | Strimzi. v1 CRD API only | | Apache Kafka | 4.1.1 | **KRaft, no ZooKeeper.** Operator-launched | | Object store | Chart 0.10.0 | S3-compatible. Its Service has a `-svc` suffix | | Shared cache | 1.38.1 | Redis protocol. Application services degrade to in-process caching without it | | Ingress cache | Valkey | Deployed by the ingress chart and used only by it. Kept separate on purpose, so a shared-cache outage cannot take the ingress data plane with it. One replica: a restart costs a cold start, not data | | Ingress operator | Commit-tagged | `ingressClassName: synapse`. Built in a sibling repository, so the tag is a commit hash rather than a version | | Platform services | The release version | Resolved from the registry host, namespace and image tag | ## Mirror paths, for the offline bundle method Every mirrored image lands under `gen0sec/`. Two paths look wrong and are not negotiable. | Image | Must be at | |---|---| | Kafka broker | `/gen0sec/strimzi/kafka:1.0.0-kafka-4.1.1` | | Kafka entity operator | `/gen0sec/strimzi/operator:1.0.0` | **Why Kafka keeps an extra path segment.** The Kafka operator's image-registry setting replaces only the registry *host* and preserves the repository path. So the chart points it at `/gen0sec` rather than ``, and the upstream `strimzi/` segment survives inside our namespace. :::note[This exact thing shipped wrong once] With the namespace omitted, the chart asked for `/strimzi/operator` while the release mirrored `/gen0sec/strimzi/operator`. One path segment apart, and the entire Kafka stack failed to pull on a cluster with no fallback. Every chart is now rendered in CI and the build fails on any image reference not backed by the release manifest, so the two cannot drift again. That gate is also why the table above is trustworthy: it catches a chart asking for a tag nobody publishes, which is how the ingress operator once requested a version that was never released. ::: `bundler.sh mirror` reproduces these paths exactly. You do not compute them. ## Not everything a chart renders is mirrored The vendored Kafka chart hardcodes an image map covering four Kafka versions plus Connect and Bridge build tooling. This edition ships one Kafka version and no Connect clusters, so the rest are deliberately absent from the bundle. That is roughly 1.5 GB of images nothing ever pulls. **If you enable Kafka Exporter, Cruise Control, Kafka Bridge or Connect builds, mirror their images first.** The release does not carry them. ## Kubernetes | | | |---|---| | Supported | 1.23 or later | See [Requirements](./../02-requirements.mdx#kubernetes-versions). ## Support policy | | | |---|---| | Supported releases | The latest two minor trains | | Upgrade path | Sequential minors. **Skipping a minor is not supported.** Patches in place | | Upgrade order | Always infra, then data, then platform | | CRDs | Helm never upgrades them. Apply them before the chart. See [Upgrade](./../operate/upgrade.mdx) | | Downgrade | Not supported across a minor that ran a schema migration. Within a train, use rollback. See [Roll back](./../operate/rollback.mdx) | | Kubernetes upgrades | Independent of ours. Stay within the supported range | ## Verifying what you are actually running The manifest says what a release pins. This says what your cluster pulled: ```bash kubectl -n gen0sec-system get pods -o yaml | grep -E '^\s+image:' | sort -u kubectl -n gen0sec get pods -o yaml | grep -E '^\s+image:' | sort -u ``` Reading the YAML rather than a container template also catches init containers, which is where the object store's init image is. On the offline bundle method, every reference should be under your own registry. Anything else means a values file was not passed. --- # Sign-in and identity Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/configure/identity # Sign-in and identity ## What is available | Method | Status | Where credentials live | |---|---|---| | Email and password | **Available** | Your Postgres | | Multi-factor authentication | **Available, and required during onboarding** | Your Postgres | | Microsoft Entra ID | **In development, not supported at the moment ** | | **Nothing outside your cluster is involved in sign-in.** Accounts, password hashes and sessions are all in your own database. :::note[Single sign-on is not available yet] Microsoft Entra ID is in development. No generic OIDC provider, no SAML and no LDAP integration exists either. If single sign-on is a requirement for you, tell us which provider, because that shapes what ships first. ::: ## The first account Registration is invite-only and invites need an existing member, so a fresh install seeds one account with a **published password**. Signing in, completing onboarding and changing that password is a required install step, not an optional one. See [First sign-in](./../07-first-signin.mdx). ## The published address decides whether sign-in works The dashboard derives its sign-in base URL from the host you publish it on. If that does not match the address people actually type in the browser, every sign-in is rejected as an invalid origin. Two rules: 1. **Use the address people type.** Not an internal service name, not a load balancer's own hostname. 2. **If TLS terminates upstream of the ingress**, set the scheme explicitly, because the chart cannot infer it and guesses `http` while the browser is on `https`. ```bash --set externalScheme=https ``` Leave that unset when TLS terminates at the ingress. See [Publish the endpoints](./../06-publish.mdx#if-tls-terminates-upstream). ## Sessions The signing secret lives in `g0s-db-ui`, created with all the other database secrets in [Install step 6](./../05-install.mdx#6-create-the-application-secrets). | Action | Effect on sessions | |---|---| | Re-running `make-db-secrets.sh` | None. The existing signing secret is preserved deliberately | | Rotating the signing secret | Every session ends and every outstanding invite link stops working | | Upgrading | None | ## Turning the dashboard off entirely If your operators use the API only: ```bash --set services.ui.enabled=false ``` The agent-facing APIs are unaffected. Note that agent API keys are issued through the dashboard, so turn it off only once you have the keys you need. ## If it fails | Symptom | Cause | |---|---| | Sign-in rejected as an invalid origin | The published host does not match the address typed, or the scheme is wrong. See above | | Sessions drop after a re-run of the secret script | The signing secret changed. It should not; check whether the secret was deleted first | | Every sign-in fails after an upgrade | Check that `externalScheme` and the host survived the upgrade. `helm get values g0s -n gen0sec` | --- # Reference configurations Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/configure/reference-configs # Reference configurations Two named configurations rather than a blank values file. Each ships in the install kit, and each is reproduced in full on this page, so you can read it before you download anything. | Configuration | For | Redundancy | Storage | |---|---|---|---| | [Single node](#single-node) | Evaluation, training, upgrade rehearsal | **None** | Smallest | | [Production](#production) | Anything you depend on | Full | Smallest production grade | ## Single node **Not for production.** Everything at one replica on one node. Losing the node loses the data. Ships as three overlays in the kit: ``` $KIT/examples/single-node/infra-values.yaml $KIT/examples/single-node/data-values.yaml $KIT/examples/single-node/platform-values.yaml ``` Layer each one **after** the production values file, so it wins. Full procedure: [Quickstart](./../quickstart.mdx). | | Production | Single node | |---|---|---| | Object store | 3 replicas, erasure coded, StatefulSet | 1 replica, standalone, **Deployment** | | Postgres | 3 instances, both poolers on | 1 instance, replica pooler off | | Kafka | 3 brokers, replication factor 3 | 1 broker, every factor 1 | | Services | 2 or 3 replicas | 1 each | | StorageClass | Yours, set explicitly | The cluster's default (empty in the overlay) | ### What the overlays set Verbatim from the v0.1.0 kit, so you can read them without unpacking it first.
infra-values.yaml — object store and cache ```yaml title="$KIT/examples/single-node/infra-values.yaml" # ============================================================================ # gen0sec-infra — SINGLE-NODE EVALUATION overlay # ============================================================================ # Layered ON TOP of values-onprem.yaml (pass both, in this order): # helm install g0s-infra charts/gen0sec-infra -n gen0sec-system \ # -f values/gen0sec-infra-values-onprem.yaml \ # -f examples/single-node/infra-values.yaml # # Shrinks the footprint to fit one worker node. NOT for production: the object # store loses erasure coding and every component becomes a single point of # failure. Use it for evaluation, training, and dry-running an upgrade. # ============================================================================ rustfs: # Distributed mode needs >= 2 nodes for erasure coding; on one node the pods # would schedule but the cluster is not fault-tolerant anyway, so run the # single-process server and be explicit about it. mode: standalone: enabled: true distributed: enabled: false replicaCount: 1 storageclass: # Empty = the cluster's default StorageClass, matching values-onprem.yaml. On # k3s that resolves to local-path, which this overlay used to hardcode — but a # literal `local-path` exists on no other distribution, so the PVCs bound # nowhere and only surfaced as pods Pending past the rollout timeout. name: "" dataStorageSize: 20Gi logStorageSize: 5Gi dragonfly: # Cache only — shrink it so the node's RAM goes to Postgres and Kafka. Keep # maxMemory under resources.limits.memory so it evicts instead of being killed. maxMemory: 256mb proactorThreads: 1 resources: requests: { cpu: 100m, memory: 128Mi } limits: { cpu: 500m, memory: 512Mi } ```
data-values.yaml — Postgres and Kafka ```yaml title="$KIT/examples/single-node/data-values.yaml" # ============================================================================ # gen0sec-data — SINGLE-NODE EVALUATION overlay # ============================================================================ # helm install g0s-data charts/gen0sec-data -n gen0sec-system \ # -f values/gen0sec-data-values-onprem.yaml \ # -f examples/single-node/data-values.yaml # # One Postgres instance, one Kafka node, replication factor 1. Any node loss is # data loss — evaluation only. # ============================================================================ postgres: numberOfInstances: 1 # With a single instance there is no replica to pool. enableReplicaConnectionPooler: false volume: size: 20Gi # Empty = the cluster's default StorageClass (see infra-values.yaml). storageClass: "" resources: requests: { cpu: 500m, memory: 1Gi } limits: { cpu: "4", memory: 6Gi } parameters: shared_buffers: 512MB effective_cache_size: 1536MB max_connections: "400" max_worker_processes: "16" max_parallel_workers: "8" timescaledb.max_background_workers: "8" kafka: replicas: 1 storage: size: 20Gi # Empty = the cluster's default StorageClass (see infra-values.yaml). class: "" # min.insync.replicas must stay < replicas, so every factor drops to 1 # together with the node count. config: offsets.topic.replication.factor: 1 transaction.state.log.replication.factor: 1 transaction.state.log.min.isr: 1 default.replication.factor: 1 min.insync.replicas: 1 log.compression.type: zstd topicDefaults: partitions: 1 replicas: 1 ```
platform-values.yaml — one replica per service ```yaml title="$KIT/examples/single-node/platform-values.yaml" # ============================================================================ # gen0sec-platform — SINGLE-NODE EVALUATION overlay # ============================================================================ # helm install g0s charts/gen0sec-platform -n gen0sec \ # -f values/gen0sec-platform-values-onprem.yaml \ # -f examples/single-node/platform-values.yaml \ # --set global.imageTag=$VERSION --timeout 20m # # One replica of everything: no HA, and a rolling update is a short outage per # service. Resource requests are unchanged — they are already small. # ============================================================================ services: auth-api: { replicaCount: 1 } ban-ledger-api: { replicaCount: 1 } config-api: { replicaCount: 1 } data-api: { replicaCount: 1 } download-api: { replicaCount: 1 } feeds-api: { replicaCount: 1 } ids-rules-api: { replicaCount: 1 } rules-validator: { replicaCount: 1 } service-graph-api: { replicaCount: 1 } signal-api: { replicaCount: 1 } telemetry-api: { replicaCount: 1 } threat-api: { replicaCount: 1 } agent-status: { replicaCount: 1 } ui: { replicaCount: 1 } ```
## Production The default. `values-onprem.yaml` in the bundle **is** the production configuration, so there is no overlay to apply: what you install in [Install](./../05-install.mdx) is this. What you set yourself: ```yaml # gen0sec-infra rustfs: storageclass: name: your-block-storage-class # gen0sec-data postgres: volume: storageClass: your-block-storage-class kafka: storage: class: your-block-storage-class ``` For requirements see [Sizing](./../02-requirements.mdx#sizing-a-production-install). ## Extending retention The shipped sizes are deliberately modest, so growing is the normal direction. Which key you grow depends on what you are retaining. See [Sizing storage for your retention](./../02-requirements.mdx#sizing-storage-for-your-retention). eg. ```yaml # gen0sec-data postgres: volume: size: 200Gi storageClass: your-block-storage-class kafka: storage: size: 200Gi class: your-block-storage-class ``` ```yaml # gen0sec-infra rustfs: storageclass: name: your-block-storage-class dataStorageSize: 100Gi ``` :::note[You can also grow later] Volume expansion grows a volume if your StorageClass has `allowVolumeExpansion: true`. Starting at the defaults and expanding when you need to is a reasonable plan, and it is the direction that works: nothing shrinks a volume. ::: ## Publishing endpoints Independent of the three above, and needed by all of them: ``` $KIT/examples/single-entrypoint/ingress-values.yaml ``` One address for the dashboard and every agent-facing API, routed by path prefix. See [Publish the endpoints](./../06-publish.mdx) for the route table and the two values you set.
The header of that file, with both customer inputs and why they exist The rest of the file is the per-service route list. It is reproduced as a table on [Publish the endpoints](./../06-publish.mdx#the-route-table) rather than twice here. ```yaml title="$KIT/examples/single-entrypoint/ingress-values.yaml (head)" # ============================================================================ # gen0sec-platform — SINGLE-ENTRYPOINT INGRESS overlay # ============================================================================ # Publishes the dashboard and the agent-facing APIs through ONE address, served # by the Synapse proxy that gen0sec-infra deploys for the `synapse` IngressClass. # # helm upgrade --install g0s charts/gen0sec-platform -n gen0sec \ # -f values/gen0sec-platform-values-onprem.yaml \ # -f examples/single-entrypoint/ingress-values.yaml # # Layered ON TOP of values-onprem.yaml, which leaves every ingress disabled. # That default is deliberate — turning this on makes the APIs reachable from # wherever the proxy Service is reachable — so this is opt-in, not the default. # # A HOST IS REQUIRED — set `entrypointHost` below, it is aliased into every # rule. Host-less rules do NOT work with this controller even though the chart # renders them: synapse's upstreams.yaml is keyed by host, and the operator # skips any rule with an empty host (controllers/ingress_controller.go, "if # host == \"\" || rule.HTTP == nil { continue }"). Such a rule produces an # Ingress object that its own controller silently ignores, with no event and no # log line. # # The dashboard needs this host to be the address people actually type: # BETTER_AUTH_URL is derived from it (services.ui.urlFromIngress), and sign-in # breaks otherwise. # # ROUTING IS BY PATH PREFIX, because one address serves every backend. The # prefixes below are the real route roots, read off each service's router -- # they are not a naming convention and do not survive being guessed at. # # One overlap to keep in mind, resolved by longest-prefix matching: # /v1/agents/events config-api serves this; data-api owns /v1/agents # If a request lands on the wrong backend, that is the first thing to check. # # /v1/ids-rules goes to download-api, NOT the in-cluster ids-rules-api: in proxy # mode IDS rules are relayed from the platform, and ids-rules-api serves a local # ET ingest that this edition does not run. # ============================================================================ # CUSTOMER-INPUT: the single address every rule below is published under. entrypointHost: &entrypointHost gen0sec.internal # CUSTOMER-INPUT: the scheme users actually reach that address on. # # Only needed when TLS terminates UPSTREAM of the Ingress -- a cloud load # balancer, a CDN, a tunnel. The chart otherwise infers the scheme from the # Ingress's own `tls:` block, which is absent in that setup, so it guesses http # while the browser is on https. The dashboard hands this to Better Auth as its # base URL and every sign-in is then rejected with "Invalid origin". # # Leave it unset when TLS terminates at the Ingress (i.e. you set ingress.tls). externalScheme: &externalScheme https ```
## Keeping your overrides Put them in a file under version control, not in `--set` arguments. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f my-site/platform.yaml \ --set global.imageTag=$VERSION \ --timeout 20m ``` `helm upgrade` uses only what you pass it. Anything set with `--set` on a previous install and omitted now reverts to the chart default, silently. A file makes that impossible to forget. Two things that are awkward to keep in a file, because they are per-release or per-secret: | Keep as `--set` | Why | |---|---| | `global.imageTag` | Changes every release | | `downloadProxy.apiKeySecret.name` | Or put it in your file. Just never omit it. See [Data relay](./relay.mdx) | ## What the kit contains ```bash ls $KIT/examples/*/ ``` Everything under `examples/` is a starting point you are meant to read and edit, not a black box. Each file carries comments explaining which values are yours to set. --- # Threat intelligence and the data relay Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/configure/relay # Threat intelligence and the data relay The relay is one in-cluster service, `download-api`. It is the only component with a route out, and it is how your fleet gets threat intelligence. This is not a choice you make. It is a requirement. See [Known limitations](./../troubleshoot/known-limitations.mdx#no-zero-egress-deployment). ## What it relays Read-only, upstream-produced content only. | Class | Routes | Why it is not shipped | |---|---|---| | CTI indicators | `/v1/indicators/version`, `/v1/indicators/download` | Built by the upstream intelligence pipeline | | ML models | `/v1/models/*` and the per-model aliases | Trained and published by Gen0Sec | | GeoIP | `/v1/geoip/manifest`, `/v1/geoip/{file}/download` | Licensed by MaxMind and IPInfo. The IPInfo Lite bundle is the default source | | IDS rules | `/v1/ids-rules/*` | Licensed as part of the Emerging Threats ruleset | **It is a relay, not a cache.** Nothing is stored on the way through. There is no local artifact store to size, no synchronization job to supervise, and no staleness window. An agent that polls gets whatever upstream has at that moment. **The relayed URLs are identical to the direct ones**, so your agents need no configuration change. ## What it never relays | Class | Routes | Why | |---|---|---| | Workload identity | `/v1/identity/*` | Produced in your cluster, describes your workloads | | Policy edges | `/v1/policy-edges/*` | Pod IPs, namespaces, NetworkPolicy edges | | Fleet bans | `/v1/fleet-bans/*` | Your own agents' ban decisions | These three are tenant-scoped: they describe your infrastructure or your decisions. Relaying them would push that to a service you do not run. They keep the local object store path behind the local authorization check, in relay mode exactly as without it. Uploads are never relayed. The relay is read-only, so `PUT` and `POST` on the download surface are served locally or not at all. ## How authorization works Two hops, two credentials, checked in two places. | Hop | Credential | Checked by | |---|---|---| | Agent to relay | The agent's own key, issued by your dashboard | Your cluster. Relayed routes sit behind the local auth middleware and each carries the same permission it has in direct mode | | Relay to Gen0Sec | Your Gen0Sec API key | Gen0Sec | **A caller's key is never forwarded upstream on a self-hosted deployment.** A locally-issued key cannot authenticate to gen0sec. Forwarding it would leak a local credential to a third party without making the request succeed. With no Gen0Sec API key configured, the relay sends no credential at all. ## The setting that must be right ```yaml downloadProxy: enabled: true upstream: "https://api.gen0sec.com" apiKeySecret: name: gen0sec-download-proxy # named by default key: DOWNLOAD_PROXY_API_KEY ``` ### Why the key is required Your agents' keys are issued by **your** dashboard and exist only in **your** database, so they cannot authenticate to gen0sec. The relay presents your own Gen0Sec API key instead, and that is the only credential the upstream will accept from your installation. Without it, the relay sends no credential, Gen0Sec answers 401, and no relayed artifact reaches your fleet. :::note[The chart names the secret by default, so this fails loudly now] `apiKeySecret.name` defaults to `gen0sec-download-proxy`. If that secret does not exist, the relay pod never starts: `CreateContainerConfigError`, visible immediately. That is deliberate. An empty name renders no environment variable at all, the relay starts, and the only symptom is every relayed class answering 401 while agents report empty rulesets rather than errors. A failed rollout at install time is the cheaper failure. Blank the name only if you relay through a gateway that supplies its own credential. ::: ### Getting the key right ```bash printf '%s' '' > ~/.gen0sec/relay-key kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-file=DOWNLOAD_PROXY_API_KEY=$HOME/.gen0sec/relay-key ``` The value renders as a `secretKeyRef`, so it never appears in a manifest or in a values file. ## All settings | Key | Default | Meaning | |---|---|---| | `downloadProxy.enabled` | `true` | Turning it off is not supported | | `downloadProxy.upstream` | `https://api.gen0sec.com` | The upstream base URL. A malformed value **fails startup** rather than degrading | | `downloadProxy.external` | `geoip,ids-rules` | Which external classes to relay. An unknown name fails startup | | `downloadProxy.timeout` | `60s` | Upstream request timeout, matching the router's own | | `downloadProxy.apiKeySecret.name` | `""` | **Set this.** See above | | `downloadProxy.apiKeySecret.key` | `DOWNLOAD_PROXY_API_KEY` | The key within the secret | :::note[Failing startup is deliberate] A malformed upstream is not treated as "fall back to reading from the local object store". That would leave every agent poll erroring against a store this deployment does not populate. Failing fast is the safer wrong answer. ::: ## Verifying ```bash kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' ``` The output is similar to this: ```json { "status": "ready", "mode": "proxy", "upstream": "https://api.gen0sec.com", "external_relays": ["geoip", "ids-rules"] } ``` **That tells you the relay started, not that it is authorized.** For that, read the log: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means your Gen0Sec API key is missing or wrong. ## Response codes worth knowing | Code | Means | What an agent does | |---|---|---| | 200 | Relayed successfully | Uses the artifact | | **401** | The upstream rejected the credential | **Reports an empty ruleset, not an error.** This is the silent failure | | **502** | The upstream is unreachable | Reads it as "ask again later" and keeps what it holds. Correct degradation | | 404 | The artifact does not exist upstream | Treats it as withdrawn | The 401 and 502 rows are the reason this page exists. ## Preview a change before applying it ```bash helm template g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ --set downloadProxy.apiKeySecret.name=gen0sec-download-proxy \ | grep -A3 DOWNLOAD_PROXY ``` --- # Secrets Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/configure/secrets # Secrets **No chart contains a credential.** Every one is referenced by name. This page is explains what must exist, with which keys, before the platform starts. ## The secrets you create | Secret | Namespace | Keys | Created in | |---|---|---|---| | `gen0sec-registry` | **Both** | `.dockerconfigjson` | [Install step 2](./../05-install.mdx#the-image-pull-secret) | | `gen0sec-object-store` | `gen0sec-system` | `RUSTFS_ACCESS_KEY`, `RUSTFS_SECRET_KEY` | [Install step 2](./../05-install.mdx#the-object-store-credentials) | | `gen0sec-download-proxy` | `gen0sec` | `DOWNLOAD_PROXY_API_KEY` | [Install step 2](./../05-install.mdx#your-gen0sec-api-key) | The pull secret is needed in **both** namespaces. The application runs in `gen0sec`; the operators, object store and cache run in `gen0sec-system`. ## The twenty the script creates `make-db-secrets.sh` composes one secret per service from the passwords the Postgres operator generated. | Secret | Contains | Role it uses | Endpoint | |---|---|---|---| | `g0s-db-` | `DATABASE_URL`, plus object store keys for services that use it | Its own per-service role, read-only where it only reads | The connection pooler | | `gen0sec-db-migrate` | `DATABASE_URL` | `postgres`, superuser | The primary, unpooled | | `g0s-db-ui` | `DATABASE_URL`, `BETTER_AUTH_SECRET`, object store keys | `frontend` | The connection pooler | Every service connects as its own role. The migration job needs a superuser on the primary because it runs `CREATE EXTENSION`, which an owner role cannot do. :::note[The session signing key is preserved] `g0s-db-ui` carries the dashboard's session signing secret. Re-running the script keeps the existing value, so sessions and invite links stay valid across a re-run. Nothing else regenerates it. ::: Four services need no database connection and get no `DATABASE_URL`: the GeoIP updater, the GeoIP database updater, the model publisher and the rules validator. ## Where the database passwords come from The Postgres operator creates a credentials secret per role it provisions, named after the role: ``` .core.credentials.postgresql.acid.zalan.do ``` Underscores in the role become dashes in the secret name. Compose each connection string from that role's password: ``` postgres://:@core-pooler.gen0sec-system.svc.cluster.local:5432/ax_prod?sslmode=require ``` | Consumer | Role | Endpoint | |---|---|---| | Each service | Its own per-service role | `core-pooler` | | Migration job | `postgres` | `core`, the primary, with the on-prem marker appended | | Dashboard | `frontend` | `core-pooler` | :::note[One deliberate exception] The dashboard's connection string uses `sslmode=no-verify`. Its client library verifies the certificate chain for `sslmode=require` and rejects the pooler's self-signed certificate; the Go library every other service uses reads the same value as encrypt-without-verify. **Do not copy `no-verify` onto the other services.** Their driver rejects it outright. ::: ## For production: use your secret manager `make-db-secrets.sh` is fine for evaluation. In production, treat its output as a **specification** rather than a tool, and have your secret manager produce secrets of the same shape. Two reasons: 1. The script writes plain Kubernetes secrets from operator-generated passwords. Your manager should be the source of truth, with the cluster holding copies. 2. Rotation is then a manager operation rather than a re-run plus a rollout. An external secret can template the connection string from the operator-generated password, so nothing is copied by hand: ```yaml # shape, not a working manifest: adapt to your operator data: - secretKey: DATABASE_URL remoteRef: key: g0s-auth-api-ro.core.credentials.postgresql.acid.zalan.do property: password # then template: postgres://:{{ .password }}@core-pooler...:5432/ax_prod?sslmode=require ``` To see the exact list your install needs, read it out of the values file rather than from this page: ```bash grep -oE 'secretRef:\s*\{?\s*name:\s*[a-z0-9-]+' \ $B/values/gen0sec-platform-values-onprem.yaml | sort -u ``` Or inspect your live installation inside your cluster. ## Rotation | Credential | How | |---|---| | Database role passwords | Rotate through the Postgres operator, then restart the consumers | | Object store keys | Update **both** `gen0sec-object-store` and every `g0s-db-` that carries them, then restart the consumers | | Registry pull credentials | Recreate `gen0sec-registry` in both namespaces. Existing pods keep running; new pulls use the new value | | Your Gen0Sec API key | Recreate `gen0sec-download-proxy`, then restart the relay | | Dashboard session key | Rotating invalidates every session and every outstanding invite link | :::note[One credential pair, surfaced two ways] The object store keys appear as `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` to the object store itself, and as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to the services that read it. Same values, two naming conventions. Rotating means updating both places. ::: ## Verifying Nothing should be missing before you install the platform: ```bash $KIT/scripts/preflight.sh ``` After install, confirm the count: ```bash kubectl -n gen0sec get secret | wc -l # ~20 service secrets plus the ones you created kubectl -n gen0sec-system get secret | grep credentials | wc -l # expect 6 ``` ## What is never in a secret | | Where it is instead | |---|---| | The registry host and namespace | The values file | | The relay upstream URL | The values file | | The cache connection string | The values file, deliberately with no credential in it | If you enable authentication on the cache, remove that line from the values file and put the connection string in each service's secret instead. --- # Values reference Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/configure/values # Values reference :::note[This page is the reference] It covers the keys you actually set, which is a much shorter list than every key a chart accepts, and states which of them are required. For anything not listed here, read the chart's own `values.yaml` in the bundle. There is no generated key-by-key reference. The charts carry no JSON schema yet, so nothing can be generated from one; when that changes this page will link to it. ::: ## How values are layered Three files, applied in order, last one wins. ``` chart values.yaml the chart's own defaults, registry deliberately blank values-onprem.yaml shipped in the bundle, points everything at a registry your overrides what you set ``` **Always pass `-f values-onprem.yaml`.** The default `values.yaml` leaves the registry blank on purpose, so installing without it cannot resolve images. Put your overrides in a file rather than in `--set` arguments, because `helm upgrade` uses only what you pass it. Anything set with `--set` last time and omitted this time reverts to the chart default. ```bash helm get values g0s -n gen0sec # what the running release actually has ``` ## gen0sec-infra Top-level keys: `postgres-operator`, `strimzi-kafka-operator`, `synapse-operator`, `synapse`, `rustfs`, `dragonfly`. | Key | Default | Set it when | |---|---|---| | `rustfs.storageclass.name` | `""`, meaning the cluster default | You want a specific StorageClass | | `rustfs.storageclass.dataStorageSize` | `50Gi` | Sizing the object store. **Four drives per replica** | | `rustfs.storageclass.logStorageSize` | `10Gi` | Rarely | | `rustfs.replicaCount` | `3` | Fewer nodes. Erasure coding assumes they spread | | `rustfs.clusterDomain` | `cluster.local` | Your cluster uses a custom DNS domain | | `dragonfly.extraArgs` | `[]` | Adding `--force_epoll` when `io_uring` is blocked | | `dragonfly.maxMemory` | Chart default | Keep it comfortably under the memory limit, or the kubelet kills the pod before it can evict | See [Sizing storage for your retention](./../02-requirements.mdx#sizing-storage-for-your-retention). ## gen0sec-data Top-level keys: `postgres`, `kafka`, `kafka2pg`. | Key | Default | Set it when | |---|---|---| | `postgres.numberOfInstances` | `3` | Never below 3 in production | | `postgres.volume.size` | `50Gi` | Sizing the database. This is the system of record, so it is the one to grow first | | `postgres.volume.storageClass` | `""` | You want a specific StorageClass. **Fast block storage** | | `postgres.resources` | 1 vCPU / 2 GiB requested, 8 vCPU / 12 GiB limit | Tuning. The limit sets the per-node minimum | | `kafka.replicas` | `3` | Fewer nodes. **Drop the replication factors together**, or brokers refuse writes | | `kafka.storage.size` | `50Gi` | Sizing the message log and the ingest buffer | | `kafka.storage.class` | `""` | You want a specific StorageClass | :::danger[Kafka replicas and replication factor move together] `min.insync.replicas` must stay below `replicas`. Reducing brokers without reducing the factors gives you a cluster that schedules and then refuses every write. ::: ## gen0sec-platform Top-level keys: `global`, `migration`, `downloadProxy`, `services`. ### global | Key | Default | Notes | |---|---|---| | `global.registry.host` | `registry.gen0sec.com` | Rewritten by the bundler on the offline path | | `global.registry.namespace` | `gen0sec` | Every image lives under this | | `global.imageTag` | Empty | **Required.** Set it to the release version | | `global.imagePullSecrets` | `gen0sec-registry` | Must exist in both namespaces | | `global.commonEnv.S3_ENDPOINT` | The object store service URL | Must match the object store Service, which has a `-svc` suffix | | `global.commonEnv.ARXIGNIS_DATA_URL` | The same URL plus `/platform-data` | **The bucket name must match the bucket you created** | | `global.commonEnv.REDIS_URL` | The cache service URL | No credential in it, deliberately | | `global.podSecurityContext` | Non-root, uid 65532 | Do not weaken. See [Hardening](./../security/hardening.mdx) | | `global.securityContext` | Read-only root, no capabilities | Do not weaken | | `global.ingress.className` | `synapse` | The ingress controller the infra chart installs | ### Publishing Set by the [single-entrypoint overlay](./../06-publish.mdx), not by `values-onprem.yaml`, which leaves every ingress disabled. Both are top level, and both are aliased into every rule the overlay writes — which is why they are listed here rather than under `services`. | Key | Default | Notes | |---|---|---| | `entrypointHost` | `gen0sec.internal` | **Required to publish anything.** The one address that serves the dashboard and every agent-facing API. A rule with an empty host renders, and the controller then skips it with no event and no log line. It must also be the address people actually type: the dashboard derives its sign-in URL from it | | `externalScheme` | Unset | Set to `https` **only when TLS terminates upstream** of the Ingress — a cloud load balancer, a CDN, a tunnel. With no `tls:` block to read, the chart otherwise infers `http` while the browser is on `https`, and every sign-in is rejected with `Invalid origin`. Leave it unset when TLS terminates at the Ingress | ### downloadProxy | Key | Default | Notes | |---|---|---| | `downloadProxy.enabled` | `true` | Turning it off is not supported | | `downloadProxy.apiKeySecret.name` | `""` | **Required. Empty is a silently broken deployment** | Full detail on [Data relay](./relay.mdx). This is the single most consequential key in the chart. ### services Twenty-one entries. Each one accepts the same shape, and you rarely need to touch any of it. | Key pattern | Notes | |---|---| | `services..enabled` | `threat-api` is not shipped at this release | | `services..replicaCount` | 2 or 3 in production, 1 in the quickstart overlay | | `services..resources` | Requests and limits. See [Sizing](./../02-requirements.mdx#sizing-a-production-install) | | `services..env` | Per-service environment, including `PORT` | | `services..ingress` | Off by default. Set by the entrypoint overlay | | `services..volumes`, `.volumeMounts` | **Do not remove these.** They exist because the read-only root filesystem needs a writable path where each service expects one | | `services.ui.enabled` | `true`. The dashboard | | `services.ui.entraId.enabled` | `false`. Entra ID is in development and not supported at this release. See [Sign-in and identity](./identity.mdx) | :::danger[Removing a service's volume mount breaks that service] Several services default to writing into a path relative to their working directory, which fails on a read-only root filesystem. Each one has an explicit writable volume mounted where it writes. Those mounts are load-bearing, not leftovers. ::: ## Checking what a change will do Always render before you apply. ```bash helm template g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f my-overrides.yaml \ | less ``` To diff against what is running: ```bash helm get manifest g0s -n gen0sec > running.yaml helm template g0s ... > proposed.yaml diff running.yaml proposed.yaml ``` ## Named starting points Two tested configurations rather than a blank page: [Reference configurations](./reference-configs.mdx). --- # First sign-in Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/first-signin ## Why this page exists Registration is invite-only, and invites can only be created by an existing member. A fresh install can be logged in by a pre-seeded admin account. ## The seeded account | | | |---|---| | Email | `admin@gen0sec.local` | | Password | `Gen0Sec!ChangeMe` | :::danger[This password is published, and it is the same on every install] It is a constant in our migration file, so anyone who has read our documentation knows it. The dashboard will not let you past it: the account is created with a must-change flag, so setting a new password is a step you cannot skip or postpone. What that does not do is protect the account between the moment you publish the address and the moment you first sign in. **Whoever signs in first is who sets the new password.** Sign in before you give the address to anyone else. If you have already shared it, sign in now and treat the interval as exposed. ::: ## 1. Sign in Open your published address and sign in with the details above. ```bash echo https://$ENTRYPOINT_HOST ``` If sign-in is rejected as an invalid origin, the published host or scheme does not match what you typed. See [Sign-in and identity](./configure/identity.mdx#the-published-address-decides-whether-sign-in-works). ## 2. Complete first-run setup The dashboard holds you at each of these in turn, and it will not let you skip ahead. A deep link to any other page returns you to the step you are on. 1. **Enroll multi-factor authentication.** Nothing else opens until you confirm a six-digit code. 2. **Set a new password.** Step 3 below. 3. **Create your organization.** 4. **Finish the remaining onboarding steps.** The wizard creates your first API key, then walks the WAF, access rule and monitoring surfaces. Multi-factor comes first, so the published password stops being enough to sign in before you have even reached the password screen. ## 3. Mandatory password change Because the password is preseeded, after setting up MFA, you'll be asked to change your password. :::note[The change signs out this account everywhere else] Every other session on the seeded account ends when the password changes. On a first install there are none, which is the point: if the address was reachable before you signed in, this is what closes that off. ::: The seed is also idempotent on the email address. Re-running the migration or the secret script neither duplicates the account nor re-flags it, and it does not reset a password you have already changed. ## 4. Invite the people who need access Invite your operators rather than sharing the seeded account. ## 5. Mint an agent API key Create an agent API key in the dashboard, then save it where the verification step reads it from: ```bash mkdir -p ~/.gen0sec printf '%s' 'yor-agent-key' > ~/.gen0sec/agent-key ``` ## If it fails | Symptom | Cause | |---|---| | The seeded account does not exist | The migration ran without the on-prem marker. Secrets were provisioned without `make-db-secrets.sh` | | Sign-in rejected as an invalid origin | The published host or scheme does not match the address you typed | | The dashboard loads but onboarding will not advance | Check `kubectl -n gen0sec logs deploy/g0s-ui` | | The password is rejected | It is case-sensitive and contains a `!`. Watch your shell if you pasted it | | The new password is rejected as too weak | 12 characters, one symbol, and three of the four character classes. The server applies the same rules as the form | | The password screen appears again on a later sign-in | The change did not complete. The flag clears only on success | --- # Install Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/install # Install ## Before you start - [ ] [Prepare the artifacts](./04-prepare-artifacts.mdx) complete - [ ] `$KIT`, `$B`, `$REGISTRY`, `$REGISTRY_USER`, `$REGISTRY_TOKEN` are set in your shell - [ ] `$G0S_DOWNLOAD_API_KEY` is set. Step 2 writes it into a secret, and an unset variable produces an empty one that fails silently - [ ] You are logged in to the registry ## 1. Check the cluster ```bash $KIT/scripts/preflight.sh ``` This checks the prerequisites the charts assume but do not create: a reachable cluster, a StorageClass, and the namespaces and secrets created in step 2. On a fresh cluster it reports the namespaces and secrets as missing, which is expected at this point. Pass a StorageClass name to pin it instead of accepting the default: ```bash $KIT/scripts/preflight.sh gen0sec-system gen0sec my-block-storage ``` ## 2. Create the namespaces and the two bootstrap secrets ```bash for ns in gen0sec-system gen0sec; do kubectl create namespace $ns --dry-run=client -o yaml | kubectl apply -f - done ``` Written to converge, so re-running step 2 after a failure further down is safe rather than a wall of `AlreadyExists`. ### The image pull secret Both namespaces need it. The application runs in `gen0sec`; the operators, object store and cache run in `gen0sec-system`. ```bash for ns in gen0sec-system gen0sec; do kubectl -n $ns create secret docker-registry gen0sec-registry \ --docker-server=$REGISTRY \ --docker-username=$REGISTRY_USER \ --docker-password=$REGISTRY_TOKEN \ --dry-run=client -o yaml | kubectl apply -f - done ``` ### The object store credentials :::danger[Run this once, and only once] Unlike the rest of step 2 this does **not** converge. It mints new random keys every time it runs, and the object store's existing data was written with the old ones — recreating the secret on a second pass strands that data behind credentials nothing holds any more. The guard below stops before that happens. ```bash kubectl -n gen0sec-system get secret gen0sec-object-store >/dev/null 2>&1 \ && echo "already exists — skip to step 3" \ || echo "not present — run the block below" ``` ::: ```bash umask 077 S=$(mktemp -d) openssl rand -hex 16 | tr -d '\n' > "$S/RUSTFS_ACCESS_KEY" openssl rand -hex 32 | tr -d '\n' > "$S/RUSTFS_SECRET_KEY" kubectl -n gen0sec-system create secret generic gen0sec-object-store --from-file="$S" ``` **Save both files somewhere safe before you delete them.** Nothing regenerates these keys, and the data in the object store is written with them. ```bash mkdir -p ~/gen0sec-object-store-keys cp "$S"/RUSTFS_* ~/gen0sec-object-store-keys/ # or load them into your secret manager rm -rf "$S" ``` Step 5 reads them back from that directory. ### Your Gen0Sec API key The relay presents this key to Gen0Sec when it fetches threat intelligence, GeoIP, models and IDS rules. The chart expects this exact secret name, so the relay pod will not start without it. See [Credentials from gen0sec](./02-requirements.mdx#your-gen0sec-api-key). ```bash # Refuses to run rather than creating an empty secret. An empty key is accepted # by Kubernetes and by the chart, and only surfaces later as every relay fetch # returning 401 across the whole fleet — with nothing in the install log. : "${G0S_DOWNLOAD_API_KEY:?not set — see Prepare the artifacts, step 0}" kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-literal=DOWNLOAD_PROXY_API_KEY=$G0S_DOWNLOAD_API_KEY ``` ## 3. Install infra ```bash helm install g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml ``` This installs three operators (Postgres, Kafka, ingress), the object store and the shared cache. ### Wait for all five The next step creates resources these operators have to reconcile. Do not continue until every one is available. ```bash kubectl -n gen0sec-system rollout status deploy/g0s-infra-postgres-operator kubectl -n gen0sec-system rollout status deploy/strimzi-cluster-operator kubectl -n gen0sec-system rollout status deploy/g0s-infra-synapse-operator kubectl -n gen0sec-system rollout status deploy/dragonfly kubectl -n gen0sec-system rollout status \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" ``` The output for each is similar to this: ``` deployment "g0s-infra-postgres-operator" successfully rolled out ``` :::note[Why the last one resolves its own kind] The object store is a StatefulSet in the production configuration and a Deployment in the single-node configuration. Resolving the kind means the same command works in both. ::: ## 4. Install data ```bash helm install g0s-data $B/charts/gen0sec-data-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml ``` This creates the Postgres cluster, the Kafka cluster, its topics, and the database roles. ### Wait for both clusters Both take several minutes. Nothing after this works until Postgres reports `Running`. ```bash kubectl -n gen0sec-system wait --for=condition=Ready kafka/core --timeout=15m kubectl -n gen0sec-system wait \ --for=jsonpath='{.status.PostgresClusterStatus}'=Running postgresql/core --timeout=15m ``` ## 5. Create the object store bucket The object store starts empty and nothing creates the bucket for you. Create it now, before the platform install. Services that read or write the object store fail until it exists. This step is required on every install. The two datasets your cluster produces about itself, `identity` and `policy-edges`, live here and are never sent anywhere. See [Network and connectivity](./03-network.mdx#what-never-leaves-your-cluster). Run it as a throwaway pod in the cluster. The pod reads the keys straight from the `gen0sec-object-store` secret, so they never reach your shell and never touch your local AWS configuration. ```bash kubectl -n gen0sec-system run s3-init --rm -i --restart=Never \ --image=amazon/aws-cli:2.15.0 \ --overrides='{"spec":{"containers":[{"name":"s3-init","image":"amazon/aws-cli:2.15.0", "env":[{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"name":"gen0sec-object-store","key":"RUSTFS_ACCESS_KEY"}}}, {"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"name":"gen0sec-object-store","key":"RUSTFS_SECRET_KEY"}}}, {"name":"AWS_EC2_METADATA_DISABLED","value":"true"}, {"name":"AWS_REGION","value":"us-east-1"}], "args":["--endpoint-url","http://rustfs-svc.gen0sec-system.svc.cluster.local:9000","s3","mb","s3://platform-data"]}]}}' ``` Expect `make_bucket: platform-data`. `BucketAlreadyOwnedByYou` is also a pass, so this step converges: running it twice is harmless. :::note[Offline installs: mirror this one image first] `amazon/aws-cli` is **not** in the offline bundle. The bundle carries only what `release-manifest.yaml` pins, and its `SHA256SUMS` covers the whole tree — the bundler refuses a package containing anything the checksums do not list — so it cannot be added after the fact. Mirror it alongside the bundle, then point the pod at your copy: ```bash skopeo copy docker://amazon/aws-cli:2.15.0 docker://$REGISTRY/amazon/aws-cli:2.15.0 ``` Substitute `$REGISTRY/amazon/aws-cli:2.15.0` for `amazon/aws-cli:2.15.0` in both the `--image` flag and the `image` field of the overrides above. They must match: the `--image` flag sets the pod's image, and the override object sets the container's, and Kubernetes uses the latter. ::: Verify by running the same command with `"s3","ls"` in place of `"s3","mb","s3://platform-data"`. The bucket should be listed: ``` 2026-09-01 14:02:11 platform-data ``` An empty result means the bucket was not created, and the services in step 7 will fail with `500 NoSuchBucket`. Do not continue. :::note[If you changed the bucket name] `platform-data` is the value of `global.commonEnv.S3_BUCKET`. If you overrode it, use your name instead. ::: :::caution[Do not export the object-store keys into your shell] The in-cluster method above is recommended because the credentials never enter your shell. That matters most on managed Kubernetes. **On EKS, exporting them breaks `kubectl`.** The kubeconfig authenticates through an `aws eks get-token` exec plugin that reads `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` — the same two variables. Setting them to the object-store keys makes every later command in that shell fail with: ``` error: You must be logged in to the server (Unauthorized) ``` The error names kubectl, not the variables, which makes it hard to attribute. The in-cluster method avoids it entirely. If you use the workstation method below, scope the credentials to the one command with `env ...` rather than exporting them. **If the object store returns `401`,** the keys are not the ones this install generated. Keys left over from a previous install of the same cluster do this, and so does a stale copy in `~/gen0sec-object-store-keys/`. Credentials are genuinely validated: wrong keys return `InvalidAccessKeyId`, and none at all returns `403`. :::
Workstation method, for reference This needs no S3 client image in the cluster, which is why it was preferred: on the offline bundle method an in-cluster image has to be mirrored first, so the bucket comes to depend on a step that exists only to create the bucket. Use it if that trade-off suits you — but read the caution above first, because exporting the object-store keys will break `kubectl` on EKS. ### 5a. Open a port forward In a second terminal: ```bash kubectl -n gen0sec-system port-forward svc/rustfs-svc 9000:9000 ``` Leave it running. The output is similar to this: ``` Forwarding from 127.0.0.1:9000 -> 9000 ``` ### 5b. Create the bucket Back in your first terminal, use the two keys you generated in step 2. Scope them to the one command with `env` rather than exporting them, so they cannot clobber the variables your kubeconfig's exec plugin reads: ```bash env AWS_ACCESS_KEY_ID="$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ AWS_SECRET_ACCESS_KEY="$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" \ AWS_REGION=us-east-1 AWS_EC2_METADATA_DISABLED=true \ aws --endpoint-url http://localhost:9000 s3 mb s3://platform-data ``` The output is similar to this: ``` make_bucket: platform-data ``` `BucketAlreadyOwnedByYou` is also a pass. With the MinIO client instead: ```bash mc alias set g0s http://localhost:9000 \ "$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ "$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" mc mb g0s/platform-data mc ls g0s ``` ### 5c. Verify it exists ```bash env AWS_ACCESS_KEY_ID="$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ AWS_SECRET_ACCESS_KEY="$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" \ AWS_REGION=us-east-1 AWS_EC2_METADATA_DISABLED=true \ aws --endpoint-url http://localhost:9000 s3 ls ``` The output is similar to this: ``` 2026-09-01 14:02:11 platform-data ``` An empty result means the bucket was not created, and the services in step 7 will fail with `500 NoSuchBucket`. Do not continue. Then stop the port forward with `Ctrl-C` in the second terminal.
## 6. Create the application secrets ```bash $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml ``` This composes around twenty secrets from the passwords the Postgres operator generated in step 4. That is why it cannot run earlier. Each service gets its own database role through the connection pooler. The migration job gets a superuser connection direct to the primary, because it runs `CREATE EXTENSION`. :::note[In production, treat this script's output as a specification] `make-db-secrets.sh` is fine for evaluation. For production, have your secret manager produce secrets of the same shape. Every name and key is listed in [Secrets](./configure/secrets.mdx). ::: ## 7. Install the platform ```bash helm install g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ --set global.imageTag=$VERSION \ --timeout 20m ``` :::note[Nothing to set for the relay] The chart already points at `gen0sec-download-proxy` by default. You created that secret in step 2, and if it is missing the relay pod fails to start with `CreateContainerConfigError` rather than coming up half-configured. ::: Schema migrations run first, as a pre-install hook. If they fail, Helm aborts and nothing is deployed. That is deliberate: a partially migrated database is worse than no install. ### Check the migration ```bash kubectl -n gen0sec get job g0s-db-migrate ``` The output is similar to this: ``` Error from server (NotFound): jobs.batch "g0s-db-migrate" not found ``` **`NotFound` is the success case.** The job is a Helm hook with `hook-delete-policy: hook-succeeded`, so Helm deletes it the moment it passes. If the job is still there, it may have failed: ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` ### Wait for the pods ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m ``` :::note[Check readiness, not phase] A pod in `CrashLoopBackOff` still reports `status.phase: Running`. A `--field-selector` on phase reports a broken install as clean. ::: :::note[Two values that must match your cluster] Both are set in `values-onprem.yaml` and both point at the in-cluster object store. Check them if anything cannot read its data. | Key | Must be | |---|---| | `global.commonEnv.S3_ENDPOINT` | `http://rustfs-svc.gen0sec-system.svc.cluster.local:9000` | | `global.commonEnv.ARXIGNIS_DATA_URL` | The same endpoint, plus the bucket: `.../platform-data` | `ARXIGNIS_DATA_URL` is where in-cluster services read their datasets from. It points at your object store, not at Gen0Sec. **The bucket name at the end of it must match the bucket you created in step 5.** If you used a different name there, change it here too. The Service is `rustfs-svc`, with the `-svc` suffix. That suffix is the single most common cause of services that come up and then cannot read anything. ::: ## If it fails | Symptom | Start here | |---|---| | `ImagePullBackOff` | [ImagePullBackOff](./troubleshoot/symptoms.mdx#imagepullbackoff) | | Install aborts at `g0s-db-migrate` | [Migration failures](./troubleshoot/symptoms.mdx#migration-failures) | | `postgresql/core` never reaches `Running` | [Postgres will not start](./troubleshoot/symptoms.mdx#postgres-will-not-start) | | Kafka broker crash loops | [Kafka will not start](./troubleshoot/symptoms.mdx#kafka-will-not-start) | --- # Network and connectivity Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/network # Network and connectivity ## Why the platform must reach Gen0Sec No platform is useful without data, neither is Gen0Sec. Because the datasets are licensed per customer, they are not in the install kit, not in the offline bundle, and not in any container image. The platform fetches them at runtime from `api.gen0sec.com`. | Dataset | Licensed by | |---|---| | CTI indicators | Upstream threat intelligence providers | | GeoIP databases | MaxMind, IPInfo | | IDS rules | Emerging Threats ruleset | | ML models | Trained and published by Gen0Sec | One in-cluster service, `download-api`, does the fetching. It relays each request upstream and streams the response straight back. It is a relay, not a cache, so there is no artifact bucket to size, no sync job to supervise and no staleness window. An agent that polls gets whatever upstream has at that moment. This also means you get the most recent data we can provide. ## One hop, not many Your agents never talk to the internet. They talk to `download-api`, and `download-api` is the only service in either namespace with a route out. ``` agents -> download-api -> api.gen0sec.com:443 ^ | the only egress point in the deployment ``` That means one destination to allow, one to audit. It also means the relayed URLs are identical to the direct ones, so agents need no configuration change. ## What never leaves your cluster Two datasets describe your own infrastructure, and they are never relayed: | Dataset | What it contains | Where it stays | |---|---|---| | `identity` | Workload identity produced in your cluster | Your object store | | `policy-edges` | Pod IPs, namespaces, NetworkPolicy edges | Your object store | No hidden switches, no configuration options, it is not implemented. Your data is yours and yours only. Uploads are not relayed either. The relay is read-only, so `PUT` and `POST` on the download surface are served locally or not at all. ## How authorization works **Your agents' API keys never leave your cluster.** Agents authenticate to your installation with keys your own dashboard issued, held in your own database. On a correct configuration the relay does not forward those keys upstream. It presents **your own Gen0Sec API key**, the one issued with your subscription, on the second hop only. ``` agent --[your agent's key]--> relay --[your Gen0Sec API key]--> api.gen0sec.com agent key stops here ``` | | | |---|---| | What Gen0Sec sees | One credential, belonging to your installation | | What Gen0Sec never sees | Any individual agent key, or how many you have issued | | Revoking an agent | Entirely local, in your dashboard | | Revoking your installation's access | Revoke or rotate your Gen0Sec API key | :::danger[This key is required, and omitting it fails silently] With no Gen0Sec API key configured, the relay falls back to forwarding the caller's own key. That is correct for the hosted product, where the same platform issued it. Here it is always wrong: your keys exist only in your cluster, so Gen0Sec rejects every one with a 401, and **the agent reports an empty ruleset rather than an error**. The install may look healthy, but the fleet quietly stops receiving threat intelligence, GeoIP, models and IDS rules. Set up in [Install](./05-install.mdx) step 2, enabled in step 7, and checked by [Verify the install](./08-verify.mdx). ::: Routes that are not relayed, including `identity` and `policy-edges`, keep the local authorization check against your own database. ## Egress list All connections are outbound, TCP, TLS. Nothing needs an inbound rule from the internet. | Destination | Port | Needed by | Purpose | Required | |---|---|---|---|---| | `api.gen0sec.com` | 443 | Cluster pods (`download-api` only) | Threat intelligence, GeoIP, ML models, IDS rules | **Always** | | `releases.gen0sec.com` | 443 | Your workstation | Download the install kit and its signature | **Always** | | `registry.gen0sec.com` | 443 | Your workstation | `helm pull` for charts, `skopeo copy` for images | **Always** | | `registry.gen0sec.com` | 443 | Cluster nodes (kubelet) | Image pulls at install time | Registry install only | | `fulcio.sigstore.dev` | 443 | Your workstation | `cosign` certificate verification | **Always** | | `rekor.sigstore.dev` | 443 | Your workstation | `cosign` transparency log verification | **Always** | | `tuf-repo-cdn.sigstore.dev` | 443 | Your workstation | `cosign` trust root refresh | **Always** | | Your own registry | 443 or 80 | Cluster nodes, and the host that runs `bundler.sh apply` | Image pulls at install time | Offline bundle only | {/* ASSUMPTION: this list is derived from the release workflow, the bundler and the chart values. Confirm it is complete, and confirm the Sigstore host set, which changes with cosign versions. */} :::note[The Sigstore hosts are the ones people forget] `cosign verify-blob` checks a signature against a public transparency log. Without those three destinations it fails, and it fails on the first command of the install. Allow them on the workstation, not on the cluster. ::: ### Inside the cluster No NetworkPolicy is required for the install. If you enforce a default-deny policy, the two namespaces need to reach each other: the application in `gen0sec` talks to Postgres, Kafka, the object store and the cache in `gen0sec-system`. ## What this rules out `api.gen0sec.com` on port 443 is required for the platform to serve data. An agent fleet with the platform hosted on a cluster that is permitted no outbound connections at all cannot function correctly. The offline bundle method removes every dependency on gen0sec-hosted registries: your cluster pulls images only from your own. It does not remove the runtime connection, because the mentioned datasets above are not ours to ship. If your policy allows no cluster egress, talk to us. We would rather hear it now than during your evaluation. > **Air-gapped.** Often used to mean "no connection of any kind". Under that definition Cerebellum > is not deployable. If it means "no dependency on vendor-hosted registries and one audited > outbound destination", the offline bundle method fits, and this page is the list your firewall team > needs. --- # Back up and restore Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/operate/backup # Back up and restore ## What to back up Not everything in the deployment is equally valuable. Backing up the wrong things is how backup jobs become expensive and untested. | Component | Back up | If you lose it | |---|---|---| | **Postgres** | **Yes, always** | Everything. Configuration, agents, policy, history. This is the system of record | | **Object store** | **Yes** | The datasets and the artifacts your cluster produced about itself. Datasets are re-fetchable, `identity` and `policy-edges` are not | | Kafka | No | In-flight messages only. It is a transport, not a store. Producers re-send | | Shared cache | No | Nothing. It repopulates | | Secrets | **Yes** | Access to your own volumes. See the warning below | :::danger[Back up the volumes along with the secrets, or the volumes are useless] The database passwords are generated by the Postgres operator and stored as Kubernetes secrets. Data is written with those credentials. A volume backup without the matching secret backup restores data you cannot open. ::: ## Postgres The Postgres operator ships a logical backup image, already mirrored as part of the release. Enable it in the infra values rather than scripting `pg_dump` yourself. ```yaml # gen0sec-infra values postgres-operator: configLogicalBackup: logical_backup_schedule: "0 2 * * *" logical_backup_provider: s3 ``` Verify a backup ran: ```bash kubectl -n gen0sec-system get cronjob | grep logical-backup kubectl -n gen0sec-system get job | grep logical-backup ``` :::note[A backup you have not restored is not a backup] Restore into a scratch namespace once, before you need to. The [Quickstart](./../quickstart.mdx) cluster is a reasonable place to rehearse it, and rehearsing a restore is also a rehearsal of an upgrade rollback. ::: ## Object store Two prefixes matter, and they matter differently. | Prefix | Recoverable without a backup? | |---|---| | `identity/`, `policy-edges/` | **No.** Produced by your cluster about your cluster | | Dataset prefixes | Yes. They are fetched from Gen0Sec | Back up the whole bucket with any S3-compatible tool, with an in-cluster job or manually, over a port forward as in [Install step 5](./../05-install.mdx#5-create-the-object-store-bucket): ```bash kubectl -n gen0sec-system port-forward svc/rustfs-svc 9000:9000 ``` ```bash aws --endpoint-url http://localhost:9000 s3 sync s3://platform-data ./platform-data-backup ``` Restore is the same command with the arguments reversed. ## Secrets ```bash kubectl -n gen0sec-system get secret -o yaml > secrets-gen0sec-system.yaml kubectl -n gen0sec get secret -o yaml > secrets-gen0sec.yaml ``` **These files contain live credentials in base64.** Treat them exactly as you treat the database itself: encrypted at rest, access controlled, and out of any shared drive. If you run an external secret manager, you do not need this: your manager is already the source of truth and the cluster holds copies. See [Secrets](./../configure/secrets.mdx). ## Recovery point and recovery time | | Value | Determined by | |---|---|---| | **Recovery point** | Your backup interval | The logical backup schedule. Nightly gives up to 24 hours of loss | | **Recovery time** | Measure it | Database size and your storage throughput. Rehearse it and record your own figure | We do not publish an RTO as it depends entirely on your configuration specifics and the data size. Time your own rehearsal and use that number in your own runbook. ## Restoring Order matters, and it is the install order. 1. Install infra and data as on a [first install](./../05-install.mdx), so the operators and empty clusters exist. 2. Restore the secrets, before anything tries to use the volumes. 3. Restore Postgres from the logical backup. 4. Restore the object store bucket. 5. Install the platform. **The migration hook runs against restored data**, so the restored schema must match the release you are installing. Restore into the version it came from, then upgrade. 6. Run the smoke test: `$KIT/scripts/verify-deployment.sh`. :::note[Do not restore an old database into a new release] The migration chain expects to run forward from the schema version in the dump. Restoring a v0.1 backup and installing v0.3 skips the intermediate migrations. Restore v0.1, then upgrade one minor at a time. See [Upgrade](./upgrade.mdx). ::: --- # Monitor Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/operate/monitor # Monitor ## What the platform exposes today | Signal | Where | Notes | |---|---|---| | Health and readiness | `/status/healthz` and `/status/readyz` on each service | Three services serve only `healthz`. See [Known limitations](./../troubleshoot/known-limitations.mdx#three-services-report-readiness-from-a-weaker-probe) | | Relay mode and upstream | `/status/healthz` on the relay | Returns the mode, the upstream and the active relay classes | | Prometheus metrics | Worker port 8080 | Exposed, but no Service is created for workers, so scraping needs a direct pod target | | Kubernetes state | Pods, Deployments, custom resources | The primary signal | **No ServiceMonitor, alert rules or dashboards ship with the charts at the moment.** If you run Prometheus, you build the scrape configuration. ## The four checks worth alerting on If you set up nothing else, set up these. ### 1. Pods not ready ```bash kubectl -n gen0sec get pods --no-headers \ | awk '{split($2,r,"/"); if (r[1] != r[2]) print}' ``` Any output is a problem. **Read the ready column, never a field selector on phase:** a pod in `CrashLoopBackOff` still reports `status.phase: Running`. ### 2. The database cluster is not Running ```bash kubectl -n gen0sec-system get postgresql core \ -o jsonpath='{.status.PostgresClusterStatus}' ``` Expect `Running`. Anything else and nothing downstream works. ### 3. The relay lost its upstream ```bash kubectl -n gen0sec exec deploy/g0s-download-api -- \ curl -fsS localhost:9999/status/healthz ``` Expect `"status": "ready"` and `"mode": "proxy"`. An unreachable upstream answers **502** on relayed routes rather than 404 or 500. That is deliberate: a polling agent reads 502 as "ask again later" and keeps the artifact it already holds, instead of treating it as withdrawn. So a broken upstream degrades your fleet slowly rather than immediately, which is safer and also easier to miss. Alert on it. ### 4. The platform stopped serving agents ```bash $KIT/scripts/verify-deployment.sh ``` Read-only, safe against production, exits non-zero and names the failing check. Running it on a schedule is the single most useful synthetic check available, because it is the only one that proves data actually moves. ## Signals that are noisy on purpose Do not page on these. | Signal | Why it is not an incident | |---|---| | `falling back to L1-only cache` | The shared cache is never fatal. Services run on their in-process cache. Reads get slower, nothing breaks | | A 502 from a relayed route, once | Upstream hiccup. Agents retry and keep what they have | | The migration job absent | That is success. Helm deletes it on completion | ## Logs ```bash kubectl -n gen0sec logs deploy/ --tail=200 kubectl -n gen0sec-system logs deploy/g0s-infra-postgres-operator --tail=200 ``` All services log to standard output as structured JSON, so any collector that reads container stdout works without configuration. Log level is set per service through the values file. See [Values reference](./../configure/values.mdx). ## What a healthy install looks like Five statements, the same five as [Verify the install](./../08-verify.mdx#definition-of-installed-successfully). If you build a dashboard, build it to answer these: 1. Every pod in both namespaces is `Running` and fully ready. 2. `postgresql/core` reports `Running`. 3. `kafka/core` reports `Ready`. 4. The relay reports `"mode": "proxy"` with your upstream. 5. The smoke test exits zero. --- # Roll back Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/operate/rollback # Roll back Read the boundary section first. It decides whether a rollback is safe or whether you need a restore. ## The boundary, stated plainly **`helm rollback` reverts manifests. It does not revert your database.** That single sentence is the whole risk. Helm restores the previous Deployments, images and configuration. Any schema change the upgrade's migration hook applied is still applied. If the old application cannot read the new schema, rolling back the manifests gives you old code against a new database. | Situation | Rollback is | |---|---| | Patch upgrade within a train, no migration ran | Safe. Just do it | | Minor upgrade whose migrations only added things | Safe. Old code ignores what it does not know | | Minor upgrade whose migrations removed or renamed something the old code reads | **Not safe.** Restore from backup instead | | You are not sure which | Check the release notes. If they do not say, ask us before rolling back | :::note[Why we prefer expand and contract] Migrations that add in one train and remove in a later one mean a one-release rollback never needs a schema change. That is the design intent, and it is what makes most rollbacks safe. It is not a guarantee for every release, which is why this page exists rather than a one-line reassurance. ::: ## Before you start - [ ] You know which revision you are going back to - [ ] You have read the release notes for the version you are leaving - [ ] The previous release's images are still in your registry **Offline bundle method:** mirroring is additive, so the old tags are still there unless you deleted them. Do not prune your registry between an upgrade and the point at which you are confident. ## 1. Find the revision ```bash helm history g0s -n gen0sec ``` The output is similar to this: ``` REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Mon Sep 1 10:14:02 2026 superseded gen0sec-platform-0.1.0 0.1.0 Install complete 2 Mon Sep 1 15:41:18 2026 deployed gen0sec-platform-0.2.0 0.2.0 Upgrade complete ``` ## 2. Roll back the platform Start here. In most cases this is the only step you need, because the application is what broke. ```bash helm rollback g0s 1 -n gen0sec ``` ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` If the smoke test passes, stop. Do not roll back infra or data unless they are the problem. ## 3. Roll back the schema, only if you must Needed when the upgrade applied a change the old code cannot work with. Check the release notes first. ```bash kubectl -n gen0sec run db-migrate-down --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/db-migrate: \ --env DATABASE_URL="$OWNER_DSN" -- down ``` `$OWNER_DSN` is the superuser connection string from the `gen0sec-db-migrate` secret. See [Secrets](./../configure/secrets.mdx). :::danger[A down migration can lose data] Reversing a migration that dropped a column recreates the column, not its contents. If the upgrade removed data, only a restore brings it back. See [Back up and restore](./backup.mdx). ::: ## 4. Roll back infra and data, only if they are the problem Same order as the upgrade, in reverse: platform, then data, then infra. ```bash helm rollback g0s-data 1 -n gen0sec-system helm rollback g0s-infra 1 -n gen0sec-system ``` :::note[CRDs do not roll back] Helm never manages CRDs after the first install, in either direction. A rolled-back operator runs against the newer CRD schema. That works when the change was additive, which is the usual case, and it is another reason to roll back the platform alone where possible. ::: ## When a restore is the right answer instead Rollback is for "the new version misbehaves". Restore is for "the data is wrong". | You want to | Use | |---|---| | Undo a bad release | Rollback | | Undo a destructive migration | Restore | | Recover from data corruption | Restore | | Return to a point in time | Restore | [Back up and restore](./backup.mdx). ## If it fails | Symptom | Cause | |---|---| | `ImagePullBackOff` after rollback | The previous version's images were pruned from your registry | | Old pods start, then error on database queries | Schema mismatch. Go to step 3, or restore | | `helm rollback` reports no such revision | `helm history` was truncated. `helm get manifest g0s --revision N` to confirm what exists | ## Next [Uninstall](./uninstall.mdx), or [Back up and restore](./backup.mdx). --- # Rotate credentials Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/operate/rotate # Rotate credentials Credentials rotates differently, and some of them have a consequence worth knowing before you start. | Credential | Rotating it interrupts | Reversible | |---|---|---| | [Database role passwords](#database-role-passwords) | The services that use them, briefly | Yes | | [Object store keys](#object-store-keys) | Everything that reads the object store | Yes, **if you kept the old values** | | [Registry pull credentials](#registry-pull-credentials) | Nothing running. New pulls only | Yes | | [Your Gen0Sec API key](#your-gen0sec-api-key) | Artifact delivery, briefly | Yes | | [Dashboard session signing key](#dashboard-session-signing-key) | **Every session and every invite link** | No | If you run an external secret manager, rotation happens there and the cluster picks up the new value. This page is for installs holding plain Kubernetes secrets. See [Secrets](./../configure/secrets.mdx). ## Database role passwords Rotate through the Postgres operator, not by editing the secret. The operator owns these values, and an edited secret is overwritten on its next reconciliation. After the operator has regenerated the credentials, recompose the application secrets and restart the consumers: ```bash $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy ``` The script preserves the dashboard's session signing key, so re-running it does not sign people out. ## Object store keys These are the ones you generated during install, and **nothing regenerates them**. Rotating means updating two places. ```bash umask 077 S=$(mktemp -d) openssl rand -hex 16 | tr -d '\n' > "$S/RUSTFS_ACCESS_KEY" openssl rand -hex 32 | tr -d '\n' > "$S/RUSTFS_SECRET_KEY" kubectl -n gen0sec-system delete secret gen0sec-object-store kubectl -n gen0sec-system create secret generic gen0sec-object-store --from-file="$S" ``` Then restart the object store, and recompose the application secrets, because the services receive the same values under different names: ```bash kubectl -n gen0sec-system rollout restart \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy ``` Save the new files where you kept the old ones, then remove the temporary directory. :::danger[Do not lose these] Data in the object store was written with them. A retained volume without the matching keys is data you cannot open. See [Back up and restore](./backup.mdx). ::: ## Registry pull credentials Recreate the secret in **both** namespaces. Running pods keep running; the new value applies to the next pull. ```bash umask 077 AUTH=$(printf '%s:%s' "$REG_USER" "$(cat $REG_PASS_FILE)" | base64 | tr -d '\n') printf '{"auths":{"%s":{"auth":"%s"}}}' "$REGISTRY" "$AUTH" > dockerconfig.json unset AUTH for ns in gen0sec-system gen0sec; do kubectl -n $ns delete secret gen0sec-registry kubectl -n $ns create secret generic gen0sec-registry \ --type=kubernetes.io/dockerconfigjson \ --from-file=.dockerconfigjson=dockerconfig.json done rm -f dockerconfig.json ``` Rotate before the old credential expires, not after. An expired pull secret is invisible until something reschedules. ## Your Gen0Sec API key ```bash printf '%s' '' > $RELAY_KEY_FILE kubectl -n gen0sec delete secret gen0sec-download-proxy kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-file=DOWNLOAD_PROXY_API_KEY=$RELAY_KEY_FILE kubectl -n gen0sec rollout restart deploy/g0s-download-api ``` Confirm it worked, because a wrong key fails silently: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means the new key is wrong, or has a trailing newline. Use `printf`, not `echo`. See [Data relay](./../configure/relay.mdx). ## Dashboard session signing key :::danger[This signs everyone out] Rotating it invalidates every active session and every outstanding invite link. There is no partial rotation and no grace period. ::: Only do this if you believe the key is compromised. Delete it and let the secret script regenerate it: ```bash kubectl -n gen0sec delete secret g0s-db-ui $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy/g0s-ui ``` The script preserves an existing key, which is why deleting the secret first is the step that actually rotates it. ## Certificates | Certificate | Owned by | |---|---| | Ingress TLS | You. Rotate in your ingress or through cert-manager | | Postgres pooler | The Postgres operator, self-signed, rotated by it | | Relay to Gen0Sec | Public certificate on our side. Nothing to do | The platform presents no certificate of its own outside the ingress you configure. ## After any rotation ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` The smoke test is the only check that proves the platform still serves an agent rather than merely starting. --- # Uninstall Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/operate/uninstall # Uninstall ## What you are about to delete | | Default | `--keep-data` | `--keep-crds` | |---|---|---|---| | The three Helm releases | Removed | Removed | Removed | | Objects the operators created outside the releases | Removed | Removed | Removed | | Kafka topic finalizers | Cleared | Cleared | Cleared | | Namespaces `gen0sec` and `gen0sec-system` | **Deleted** | Kept | Deleted | | Volumes: Postgres, Kafka, object store | **Deleted** | Kept | Deleted | | Secrets, including the database passwords | **Deleted** | Kept | Deleted | | Operator CRDs | **Removed** | Removed | Kept | :::danger[The default deletes your data, and it asks you to prove you mean this cluster] Without `--keep-data` the databases and the object store go, and nothing can undo that. So that run also **asks you to type the current context name back**, and refuses to run at all with no terminal attached. `--keep-crds` does not trigger that prompt, because a schema the next install reinstalls anyway is no reason to skip asking about the databases. ::: ## Remove everything ```bash $KIT/scripts/uninstall.sh --yes ``` It prints the cluster context, the release list and the volume count, then waits for you to type the context name. Read what it prints before you type. ## Keep the data ```bash $KIT/scripts/uninstall.sh --yes --keep-data ``` Stops after clearing the topic finalizers. The namespaces stay, and so do the volumes and the secrets in them. No context confirmation, because nothing irreversible happens. :::note[Why the secrets have to stay with the volumes] A reinstall over retained volumes needs the original database passwords: the data was written with them and a fresh Postgres operator generates different ones. Keeping the volumes without the secrets leaves you with data you cannot open. ::: ## Keep the CRDs ```bash $KIT/scripts/uninstall.sh --yes --keep-crds ``` Use this when something else in the cluster runs the same operators, or when you want the next install to skip reinstalling schemas. ## How CRD removal is made safe CRDs are cluster-scoped, and **deleting one deletes every resource of that kind in every namespace**. Another Postgres or Kafka cluster run by the same operators would go with them. So the script checks first: if any of those kinds is in use outside the two namespaces it is removing, **it skips the CRDs** and tells you. `--keep-crds` skips them unconditionally. That check is why removing CRDs by default is safe. It is not a blanket delete. If you want to see what would be affected before you run anything: ```bash kubectl get postgresqls,kafkas,kafkatopics,kafkanodepools -A 2>/dev/null | grep -v 'gen0sec' ``` Any output there means the script will leave the CRDs alone. ## The Kafka finalizer, and why order matters Strimzi puts a `strimzi.io/topic-operator` finalizer on all 23 KafkaTopics. Once the operator is gone, nothing clears it, and deleting the namespace leaves it in `Terminating` indefinitely. The script uninstalls the releases first, then clears the finalizers, then deletes the namespaces. That order is the whole reason the script exists. If you already deleted the namespace and it is stuck, this is the fix: ```bash NS=gen0sec-system kubectl get kafkatopics.kafka.strimzi.io -n $NS -o name \ | xargs -r -n1 kubectl patch -n $NS --type=merge \ -p '{"metadata":{"finalizers":null}}' ``` ## Reinstalling over retained volumes Two things bite here. **Kafka.** The volumes survive on purpose, and a recreated cluster generates a new identity. Brokers then crash loop with `Invalid cluster.id`. See [Kafka will not start](./../troubleshoot/symptoms.mdx#kafka-will-not-start). **Postgres.** You need the original role passwords. If you kept the secrets, the operator adopts the existing data. If you did not, the volumes are unreadable. ## Uninstalling by hand The script does more than these commands: it also removes objects the operators created outside the releases, and it handles the CRD safety check. Use it if you can. ```bash helm uninstall g0s -n gen0sec helm uninstall g0s-data -n gen0sec-system helm uninstall g0s-infra -n gen0sec-system # clear the finalizers before touching the namespaces kubectl get kafkatopics.kafka.strimzi.io -n gen0sec-system -o name \ | xargs -r -n1 kubectl patch -n gen0sec-system --type=merge \ -p '{"metadata":{"finalizers":null}}' kubectl delete namespace gen0sec kubectl delete namespace gen0sec-system ``` Do not pass `--wait` to `helm uninstall`. The Kafka topics cannot finish deleting until the finalizers above are cleared, so waiting only burns the timeout. --- # Upgrade Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/operate/upgrade # Upgrade When this page is done you are running the new release, with the same data. Read [Roll back](./rollback.mdx) before you start. The rollback boundary changes what a safe upgrade looks like. ## Before you start - [ ] Read the release notes for every version between yours and the target - [ ] A current backup. See [Back up and restore](./backup.mdx) - [ ] The new release's install kit, verified - [ ] A maintenance window. The application is briefly unavailable while pods roll ## Rules that are not negotiable | Rule | Why | |---|---| | One minor at a time | Skipping a minor is not supported. Each train's migrations assume the previous train ran | | Patches in place | Within a train, upgrade directly | | Same order as install: infra, then data, then platform | Each chart depends on the previous one | | Apply CRDs by hand first | Helm never upgrades CRDs. See step 1 | | Back up first | The migration hook changes your schema, and `helm rollback` does not undo that | ## 1. Apply the new CRDs **Do this before anything else.** Helm installs a chart's CRDs on first install and ignores them from then on. An upgrade that ships a new CRD schema fails with a server-side-apply error, or worse, silently rejects fields it does not know. ```bash cd $B/charts tar xzf gen0sec-infra-*.tgz kubectl apply --server-side --force-conflicts \ -f gen0sec-infra/charts/postgres-operator/crds/ kubectl apply --server-side --force-conflicts \ -f gen0sec-infra/charts/strimzi-kafka-operator/crds/ ``` The output is similar to this: ``` customresourcedefinition.apiextensions.k8s.io/postgresqls.acid.zalan.do serverside-applied ``` :::note[This is not hypothetical] The Postgres operator 1.15 to 2.0 change adds PostgreSQL 18 to the `postgresqls` schema. Without the CRD apply, the `postgresql/core` resource is rejected outright and the database never starts. ::: ## 2. Get the new artifacts Follow [Prepare the artifacts](./../04-prepare-artifacts.mdx) for the new release. Both methods work the same as on a first install, with one addition for the offline bundle. **Offline bundle only.** Mirroring is additive: the new tags land beside the old ones. Your running install keeps working until you upgrade, and a rollback still has its images. Do not reuse the previous bundle's directory. The bundler refuses to write into an existing one, and each bundle's values files are targeted at the version they shipped with. ## 3. Upgrade infra Run every command from the **new** bundle directory, so `charts/` and `values/` are the new release's. ```bash helm upgrade g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml ``` Wait for the operators to finish rolling before continuing. ```bash kubectl -n gen0sec-system rollout status deploy/g0s-infra-postgres-operator kubectl -n gen0sec-system rollout status deploy/strimzi-cluster-operator kubectl -n gen0sec-system rollout status deploy/g0s-infra-synapse-operator kubectl -n gen0sec-system rollout status deploy/dragonfly kubectl -n gen0sec-system rollout status \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" ``` ## 4. Upgrade data ```bash helm upgrade g0s-data $B/charts/gen0sec-data-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml ``` ```bash kubectl -n gen0sec-system wait --for=condition=Ready kafka/core --timeout=15m kubectl -n gen0sec-system wait \ --for=jsonpath='{.status.PostgresClusterStatus}'=Running postgresql/core --timeout=15m ``` ## 5. Upgrade the platform The schema migration runs first, as a **pre-upgrade** hook, before any new pod rolls. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ --set global.imageTag=$VERSION \ --timeout 20m ``` **If the migration fails, Helm aborts and the old application keeps running.** That is the designed behavior and it is why the hook runs before the rollout. ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` Fix the cause, then re-run the same command. See [Migration failures](./../troubleshoot/symptoms.mdx#migration-failures). ## 6. Verify ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` Run the smoke test after every upgrade, not just after installs. It is the only check that proves the platform still serves an agent rather than merely starting. Full checklist: [Verify the install](./../08-verify.mdx). ## If you changed values along the way `helm upgrade` uses the values files you pass, not the ones from last time. Anything you set with `--set` on the previous install must be set again, or it reverts to the chart default. Check what the running release actually has: ```bash helm get values g0s -n gen0sec ``` Keep your overrides in a file rather than in shell history. See [Reference configurations](./../configure/reference-configs.mdx). ## If it fails | Symptom | Start here | |---|---| | CRD schema error on `helm upgrade` | You skipped step 1 | | `Unsupported value` from the Postgres operator | You skipped step 1 | | Migration job fails | [Migration failures](./../troubleshoot/symptoms.mdx#migration-failures) | | Pods roll but the smoke test fails | [Roll back](./rollback.mdx) | | `ImagePullBackOff` on new tags | The new images were not mirrored. Offline bundle method, step 2 | ## Next [Roll back](./rollback.mdx). --- # Prepare the artifacts Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/prepare-artifacts import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; # Prepare the artifacts ## Before you start - [ ] [Requirements](./02-requirements.mdx) met, tools installed, `yq` version checked - [ ] [Network and connectivity](./03-network.mdx) configured by your network team - [ ] Your [registry token](./02-requirements.mdx#the-registry-token), created on the dashboard under **Repository Access** - [ ] Your [Gen0Sec API key with the required scopes](./02-requirements.mdx#your-gen0sec-api-key) ## 0. Set your keys Create your token files. ```bash mkdir -p ~/.gen0sec cd ~/.gen0sec/ && touch .registry-token .download-api-key ``` Then open up an editor of your choice to fill each file with their respective content. Then run ```bash export REGISTRY_TOKEN=$(cat ~/.gen0sec/.registry-token) export G0S_DOWNLOAD_API_KEY=$(cat ~/.gen0sec/.download-api-key) ``` :::tip[Manage your keys securely] By storing keys in files, you make sure they don't get etched into your shell history. If you have history turned off or you know what you're doing, you can also go with ```bash export REGISTRY_TOKEN="your-registry-token" export G0S_DOWNLOAD_API_KEY="your-gen0sec-api-key" ``` ::: ## 1. Set the release Pin the release explicitly. A pinned tag installs the same bytes in six months; a resolved one does not. You can find available release versions on the [Releases page](https://releases.gen0sec.com/cerebellum/). ```bash export TAG=v0.1.0 export VERSION=${TAG#v} ``` To always use the latest version instead: ```bash export TAG=$(curl -fsSL https://releases.gen0sec.com/api/repos/cerebellum/latest | jq -r .tag_name) export VERSION=${TAG#v} ``` :::note[Release candidates are not published here] Only stable releases are mirrored, so an `-rc.N` tag returns 404 no matter how it is spelled. The [release index](https://releases.gen0sec.com/cerebellum/) is the list of what exists. ::: ## 2. Download the install kit ```bash curl -fsSLO https://releases.gen0sec.com/cerebellum/$TAG/cerebellum-install-kit.tar.gz curl -fsSLO https://releases.gen0sec.com/cerebellum/$TAG/cerebellum-install-kit.tar.gz.cosign.bundle ``` ## 3. Verify the signature Run this before you unpack anything. ```bash cosign verify-blob \ --bundle cerebellum-install-kit.tar.gz.cosign.bundle \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/release\.yaml@refs/tags/' \ cerebellum-install-kit.tar.gz ``` The output is similar to this: ``` Verified OK ``` Anything else means stop. Do not unpack the archive. :::note[Why both certificate flags are required] `cosign` refuses to verify without an expected issuer and an expected identity, so a shortened command fails rather than passing weakly. The identity is pinned to one workflow on a tag reference. A broader pattern would accept a signature produced by any workflow in our organization. The signature covers the file's contents, so it proves the bytes are ours regardless of how they reached you. ::: ## 4. Unpack it ```bash tar -xzf cerebellum-install-kit.tar.gz export KIT=$PWD/cerebellum-install-kit-$VERSION cd $KIT ``` The kit contains the bundler, the release manifest, the helper scripts used later in this install, and the sizing overlays. It does not contain images or charts. ## 5. Log in to our registry ```bash export REGISTRY=registry.gen0sec.com export REGISTRY_USER=any printf '%s' "$REGISTRY_TOKEN" | helm registry login $REGISTRY -u "$REGISTRY_USER" --password-stdin printf '%s' "$REGISTRY_TOKEN" | skopeo login $REGISTRY -u "$REGISTRY_USER" --password-stdin ``` Both are needed. `helm` and `skopeo` keep separate credential stores, and the next step uses each of them. `cosign` is a third store again, and it needs no login of its own. The bundler lends it the credential `skopeo` holds. ## 6. Get the charts Pull the charts and their values files. Images are not pulled: your cluster pulls each one from `registry.gen0sec.com` when it needs it. :::note[Confirm your nodes can pull from us before you go further] Our registry challenges pulls that send no `User-Agent`, which on some distributions is what the kubelet's pull path does. This step succeeds either way, because it runs from your workstation. The failure appears at step 3 of [Install](./05-install.mdx), as a 403 on every image. Check it now: [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: ```bash ./bundler.sh assemble-charts release-manifest.yaml export B=$PWD/gen0sec-cerebellum-multiarch-$VERSION ``` The output ends with a per-chart digest confirmation. :::note[What "digest-verified" means here] `assemble-charts` checks two digests per chart: the registry's manifest against the digest this release pins, and the chart archive against the layer named in that manifest. What you install provably came out of the chart this release published. ::: Each chart carries its own `values-onprem.yaml`, extracted into `$B/values/`. That file is what points the install at a registry. The default `values.yaml` leaves the registry blank on purpose, so installing without `-f` will not resolve images. ### 6a. Build the bundle on your workstation ```bash ./bundler.sh create-package release-manifest.yaml ``` This pulls every image and chart the release pins by digest, verifies every signature, then packages the result as `gen0sec-cerebellum-multiarch-$VERSION.tar.gz`. It carries both `linux/amd64` and `linux/arm64`, so it installs on any node in your cluster. Signature verification happens here, on the connected side, because it needs the public transparency log. Everything downstream is protected by checksums instead. :::note[Before you use --arch to halve the transfer] `--arch amd64` or `--arch arm64` extracts one manifest out of each multi-architecture index. That changes every image digest by definition. Per-architecture signatures still verify, because each one is signed individually. **SBOM attestations do not**, because they are attached to the index, and the index is what a single-architecture bundle leaves behind. The bundler warns at the time and annotates the bundle. If your acceptance process checks attestations, stay on the default. If your nodes are mixed, or you are unsure what they run, stay on the default: a missing architecture fails when the kubelet tries to pull, on an isolated cluster, after the transfer. A single-architecture bundle also renames the tarball and the directory inside it, so substitute the name in every command that follows. ::: ### 6b. Move it across In this step you take your bundle onto an intermediary machine inside the network where your internal registry and the target cluster is located you'll install the bundle onto. After transferring the bundle, you can clean up all artifacts it has created. eg. ```bash scp gen0sec-cerebellum-multiarch-$VERSION.tar.gz :~/ # `create-package` ran inside the kit directory, so the bundle is sitting in it. Copying the kit first # would transfer tens of gigabytes twice. `cleanup` removes only what the bundler created. ./bundler.sh cleanup cd .. && scp -r cerebellum-install-kit-$VERSION :~/ # You can even remove the installation kit if you don't intend to keep it. # cd $KIT/.. && rm -rf cerebellum-install-kit.tar.gz $KIT ``` ### 6c. Load it into your registry Log in to your intermediary machine and use the bundler to load the images into your internal registry. ```bash export REGISTRY=registry.internal.example export REGISTRY_USER=your-registry-user export REGISTRY_TOKEN=$(cat path/to/your-registry-token) # Carried across because this is a different machine and Install step 2 needs # it. Unset, the secret it creates is empty and every relay fetch 401s. export G0S_DOWNLOAD_API_KEY=$(cat path/to/your-gen0sec-api-key) export KIT=~/cerebellum-install-kit-$VERSION export VERSION=$(cat $KIT/release-manifest.yaml | yq ".spec.version") tar -xzf gen0sec-cerebellum-multiarch-$VERSION.tar.gz export B=~/gen0sec-cerebellum-multiarch-$VERSION cd $B printf '%s' "$REGISTRY_TOKEN" | skopeo login $REGISTRY -u "$REGISTRY_USER" --password-stdin ./bundler.sh apply release-manifest.yaml -r $REGISTRY ``` `apply` pushes every bundled image into your registry, then rewrites the bundled values files to point at it. The charts are not pushed: you install them from `$B/charts/*.tgz`. The bundle is checked against its own `SHA256SUMS` before anything is pushed, so a corrupted transfer fails here rather than at install time. :::note[For a plain-HTTP registry] Set `MIRROR_TLS_VERIFY=false` before `apply`. This affects the push only, and it is the one place in this install where TLS verification can be disabled. Prefer a certificate from your internal CA. ::: ## Checkpoint Whichever method you used, you should now have the following: | Variable | Points at | |---|---| | `$KIT` | The unpacked install kit | | `$B` | A directory containing `charts/` and `values/` | | `$REGISTRY` | The registry your cluster will pull from | | `$REGISTRY_USER` | The username for that registry (`any` for ours) | | `$REGISTRY_TOKEN` | The password or token for that registry | | `$G0S_DOWNLOAD_API_KEY` | Your Gen0Sec API key, set in step 0. [Install](./05-install.mdx#your-gen0sec-api-key) step 2 puts it in a secret | You are logged in to `$REGISTRY`. Every page after this one assumes exactly that and nothing else, which is why nothing after this point branches. --- # Publish the endpoints Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/publish # Publish the endpoints The ingress controller is installed and running. However it's not configured to serve anything by default. That is deliberate, because creating those routes makes the APIs reachable from wherever the controller is reachable, so it is your decision, not ours. The data plane's LoadBalancer Service already has an external address at this point. Until you complete this page it serves nothing, because no rule names a host. :::note[Ingress and TLS are yours to configure] The platform ships the routing rules. Terminating TLS, issuing the certificate and exposing the LoadBalancer are your cluster's responsibility. For the configuration options, see the [Kubernetes ingress reference](https://docs.gen0sec.com/docs/client/kubernetes-ingress). ::: ## Before you start - [ ] [Install](./05-install.mdx) complete, all pods running - [ ] A DNS name you control, resolving to your ingress controller's address - [ ] A LoadBalancer address for the ingress data plane, or a Service type your cluster can expose - [ ] A TLS certificate for that name, or cert-manager configured ## 1. Choose the address One address serves the dashboard and every agent-facing API. Requests are routed by path prefix. ```bash export ENTRYPOINT_HOST=cerebellum.example.internal ``` **The host is required.** A rule with no host is silently ignored by the ingress controller: the Ingress object is created, no event is logged, and nothing serves traffic. **Use the address people will actually type.** The dashboard derives its sign-in URL from this host, and sign-in fails if the two do not match. ## 2. Publish The kit ships the overlay. Layer it on top of the values file you have been using. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` :::warning[Re-pass every values file you installed with] `helm upgrade` applies only the files you pass it. Anything layered at install time and omitted here reverts to the chart default. On the [Quickstart](./quickstart.mdx) you also passed `-f $KIT/examples/single-node/platform-values.yaml`, so include it in this upgrade too (before the ingress overlay), or every service scales back to its production replica count: ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-node/platform-values.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` ::: :::note[Quote any --set argument containing brackets] `zsh` treats `[0]` as a glob and fails before `helm` runs. Single-quote the whole argument: `--set 'services.ui.ingress.hosts[0].host=...'`. ::: ## 3. Verify the routes exist ```bash kubectl -n gen0sec get ingress ``` The output is similar to this: ``` NAME CLASS HOSTS ADDRESS PORTS AGE g0s-platform synapse cerebellum.example.internal 10.0.0.42 80 30s ``` An empty `HOSTS` column means the host was not applied, and nothing will serve traffic. Then check from outside the cluster. The dashboard is published at `/` and returns a redirect to its sign-in page, so follow it with `-L`: ```bash curl -fsSL -o /dev/null -w '%{http_code}\n' https://$ENTRYPOINT_HOST/ ``` The output is similar to this: ``` 200 ``` :::note[There is no health endpoint on the ingress] `/status/healthz` is served per pod inside the cluster and is deliberately **not** published through the ingress. A request to it from outside falls through to the dashboard catch-all and returns `404`, so do not use it as an external check. The dashboard root above confirms reachability; the [smoke test](./08-verify.mdx#4-the-platform-serves-an-agent) confirms the platform serves agents. ::: ## Find the address to point DNS at The ingress data plane is a `LoadBalancer` Service. ```bash kubectl -n gen0sec-system get svc g0s-infra-synapse -o wide ``` Point your DNS name at the external address. If `EXTERNAL-IP` stays ``, your cluster has no LoadBalancer implementation: change the Service type to something it can expose. It cannot stay cluster-internal. ## The route table You do not configure these. The overlay sets them, read off each service's own router. They are here because they are the answer to "a request landed on the wrong backend", and because a missing route fails quietly rather than loudly. | Path prefix | Served by | |---|---| | `/` | Dashboard (catch-all, shortest prefix, so it is last resort) | | `/v1/indicators`, `/v1/models`, `/v1/geoip`, `/v1/ids-rules` | Relay | | `/v1/penalty-model`, `/v1/traffic-model`, `/v1/ja4-classifier`, `/v1/flow-classifier`, `/v1/jepa-embedder` | Relay, per-model aliases | | `/v1/identity`, `/v1/policy-edges`, `/v1/fleet-bans` | Relay, served locally | | `/v1/authcheck`, `/v1/auth` | Authentication | | `/v1/events`, `/v1/agents`, `/v1/log` | Agent ingest | | `/v1/agents/events` | Configuration push (longer prefix than `/v1/agents`, so it wins) | | `/v1/config` | Configuration | | `/v1/telemetry`, `/v1/logs` | Telemetry | | `/v1/signal`, `/v1/rules` | Signal | | `/v1/ban-ledger` | Ban ledger | | `/v1/threat` | Threat feeds | | `/v1/service-graph` | Service graph | Routing is by **longest matching prefix**, because one address serves every backend. The configuration push stream on `/v1/config` is long-lived, so the chart sets read, write and idle timeouts of 3600 seconds on that route. Do not lower them: agents fall back to polling with no push, which looks like a slow platform rather than a timeout. :::note[Why a missing route is worse than a broken one] The dashboard owns `/`, so any path absent from this table falls through to it and gets a redirect or HTML. HTTP has no way for the caller to tell that apart from a real answer, so the failures are silent: | Missing route | What the agent actually does | |---|---| | `/v1/authcheck` | Receives HTML, cannot distinguish a bad key from a routing mistake, and continues unverified | | `/v1/agents/events` | Logs `SSE connect returned HTTP 404`, then falls back to polling. Looks like a slow platform, not a routing bug | | `/v1/ban-ledger` | Reads the redirect as success. The contribution is lost with no error anywhere | This is why the overlay exists rather than a list of prefixes for you to type. ::: :::note[/v1/ids-rules goes to the relay, not to the in-cluster rules service] IDS rules are relayed from gen0sec. The in-cluster rules service builds rules from a local ingest pipeline that this edition does not run. Route this prefix there and you reach a service with no rules, and the relay is never used. ::: ### If TLS terminates upstream If a load balancer, CDN or tunnel terminates TLS in front of the ingress, the chart cannot infer the external scheme and guesses `http` while the browser is on `https`. The dashboard then builds its sign-in URL with the wrong scheme and rejects every sign-in with `Invalid origin`. Set it explicitly: ```bash --set externalScheme=https ``` Leave it unset when TLS terminates at the ingress, because then the chart reads it from the ingress TLS block. ## The dashboard The dashboard is enabled by default and is published at the root of your address. Sign-in is email and password, held in your own Postgres, so nothing outside the cluster is involved. Its secret, `g0s-db-ui`, was created in [Install](./05-install.mdx) step 6 with all the others. Re-running that script keeps the existing signing secret, so sessions and invite links stay valid. To leave the dashboard out: ```bash --set services.ui.enabled=false ``` ## If it fails | Symptom | Cause | |---|---| | `kubectl get ingress` shows an empty `HOSTS` column | `entrypointHost` was not set. The controller ignores host-less rules silently | | `curl` times out | DNS does not resolve to your ingress controller, or a firewall blocks the port | | The dashboard loads but sign-in redirects to the wrong address | The sign-in URL is derived from `entrypointHost`. It must be the address people type | | A request reaches the wrong service | Longest-prefix routing. Check the `/v1/agents` overlap above | --- # Quickstart on one node Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/quickstart # Quickstart on one node :::danger[Not for production] This runs every component at a single replica on a single node. There is no redundancy anywhere: **losing the node loses the data.** Postgres has no standby, Kafka has one broker at replication factor 1, and the object store has no erasure coding. Use it for evaluation, training and upgrade rehearsals. For anything you depend on, follow [Sizing a production install](./02-requirements.mdx#sizing-a-production-install), which runs on three nodes. ::: ## What you get A working instance you can sign in to, on one machine, without planning a cluster. This is not a separate procedure. It is the same install as the rest of this documentation, with one extra values file per chart. ## What you need | | | |---|---| | One node | 12 vCPU, 32 GB RAM, 200 GB disk | | Kubernetes | k3s | | StorageClass | Any default block StorageClass. k3s installs `local-path` as the default, which is enough | | Tools | The same list as [Requirements](./02-requirements.mdx#tools-on-your-workstation) | | Credentials | The same two as [Requirements](./02-requirements.mdx#credentials-from-gen0sec). The relay is required here too | ## How it differs from production The kit ships an overlay per chart that reduces everything to one replica. | | Production | This quickstart | |---|---|---| | Object store | 3 replicas, erasure coded, distributed | 1 replica, **standalone mode**, and a Deployment rather than a StatefulSet | | Postgres | 3 instances | 1 instance, no replica pooler | | Kafka | 3 brokers, replication factor 3 | 1 broker, every factor 1 | | Application services | 2 or 3 replicas each | 1 each | | Cache | Chart default | 256 MB, one thread | | Storage | Your block StorageClass, 500 GiB per node minimum | The cluster's default StorageClass, about 65 GiB of volumes | | Survives losing a node | Yes | No | :::note[The connection pooler stays on] Only the **replica** pooler is disabled, because there is no replica to pool. Every service still connects through `core-pooler`, exactly as in production, so the secrets that `make-db-secrets.sh` composes are the same shape here. ::: :::note[Two things that surprise people] The object store stops being a StatefulSet and becomes a Deployment of the same name. That is why the readiness checks in [Install](./05-install.mdx) resolve the kind instead of naming it. Volume claims from a previous multi-replica install are not reused. If you are converting an existing install, delete them first. ::: ## Run it Follow the spine exactly as written, starting at [Prepare the artifacts](./04-prepare-artifacts.mdx). Three commands change. Nothing else does. | Spine step | On this install | |---|---| | [Prepare the artifacts](./04-prepare-artifacts.mdx) | Unchanged. Either method works, and the registry method is fewer steps | | [Install](./05-install.mdx) steps 1 and 2 | Unchanged. Same namespaces, same three secrets | | [Install](./05-install.mdx) steps 3, 4 and 7 | Add the overlay. See below | | [Install](./05-install.mdx) steps 5 and 6 | Unchanged. You still create the bucket, and you still run the secret script | | [Publish the endpoints](./06-publish.mdx) | Unchanged. See the note on the address below | | [First sign-in](./07-first-signin.mdx) and [Verify the install](./08-verify.mdx) | Unchanged | Pass the overlay **after** the production values file, so it wins. ```bash helm install g0s-infra $B/charts/gen0sec-infra-*.tgz -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml \ -f $KIT/examples/single-node/infra-values.yaml ``` ```bash helm install g0s-data $B/charts/gen0sec-data-*.tgz -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml \ -f $KIT/examples/single-node/data-values.yaml ``` ```bash helm install g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-node/platform-values.yaml \ --set global.imageTag=$VERSION \ --timeout 20m ``` The wait gates after each one are the same as on [Install](./05-install.mdx). Read the files before you use them: each overlay carries comments explaining what it trades away. :::caution[Your cluster needs a default StorageClass] The overlays leave the storage class empty in all three places — `rustfs.storageclass.name` in `infra-values.yaml`, `postgres.volume.storageClass` and `kafka.storage.class` in `data-values.yaml` — which means "use the cluster's default", the same as the production values file. On k3s that resolves to `local-path` and there is nothing to do. If your cluster has **no** default, every PVC stays `Pending`. `helm install` still reports success, and the only symptom is the wait gate above timing out without naming a cause. Check first: ```bash kubectl get storageclass # exactly one row must be marked (default) ``` To pin a class explicitly instead of relying on the default, set those three keys, or layer a fourth `-f` file last. ::: ## The address [Publish the endpoints](./06-publish.mdx) is unchanged, including the requirement for a host. On k3s the bundled LoadBalancer implementation gives the ingress data plane the node's own address, so there is nothing further to provision. For an evaluation, a name in `/etc/hosts` on the machine you browse from is enough. It has to be the name you actually type in the browser, because the dashboard builds its sign-in URL from it. :::note[About TLS terminatotion] In case your TLS termination does not happen on a higher level, configure Synapse Ingress to work with cert-manager (see the [Kubernetes ingress reference](https://docs.gen0sec.com/docs/client/kubernetes-ingress)). The shipped ingress overlay sets `externalScheme: https` and configures no certificate at the ingress. A single-node evaluation usually has none. Either provision a certificate, or serve plain HTTP: set `externalScheme` to `http` in the ingress overlay and use an `http://` base URL in the sign-in and verify steps below. ::: ## First sign-in Nothing is different here, and it is worth knowing before you get there: the dashboard makes you enroll multi-factor authentication and replace the seeded password before anything opens. See [First sign-in](./07-first-signin.mdx). ## Verify Run the same checks as a production install, from [Verify the install](./08-verify.mdx). The smoke test is the one that matters, because it proves the platform serves an agent rather than merely starting: ```bash # Save the agent key you minted at first sign-in, then read it back below. mkdir -p ~/.gen0sec printf '%s' '' > ~/.gen0sec/agent-key export GEN0SEC_BASE_URL=https://$ENTRYPOINT_HOST export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) $KIT/scripts/verify-deployment.sh ``` A single-node install that passes it is a good rehearsal for a real one. That is most of the value of building this. ## Tearing it down ```bash $KIT/scripts/uninstall.sh --yes ``` It asks you to type the cluster context name back before deleting anything, so it will not run unattended. See [Uninstall](./operate/uninstall.mdx). ## When you are ready for production [Sizing a production install](./02-requirements.mdx#sizing-a-production-install) has the node counts, per-node capacity and storage figures. There is no in-place conversion from this install to a production one: build the production cluster separately. --- # Bundler CLI Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/reference/bundler # Bundler CLI `bundler.sh` ships in the install kit. It builds an offline bundle on a connected host and loads it into your registry on a host that has no internet access. Its only input is the `release-manifest.yaml` inside the kit. No access to our source is needed, so you can build your own bundle from any published release. For the procedure rather than the reference, see [Prepare the artifacts](./../04-prepare-artifacts.mdx). ## Which commands need internet access This is the table an isolated site actually needs. The pipeline is split on internet access, not on environment. | Command | Internet | Talks to | |---|---|---| | `validate` | **Required** | Parses the manifest, resolves tooling, then fetches one image manifest to confirm registry access | | `assemble` | **Required** | Source registry: images and charts | | `assemble-charts` | **Required** | Source registry: charts only | | `assemble-images` | **Required** | Source registry: images only | | `verify-assembly` | **Required** | Source registry for signature artifacts, plus Sigstore trust material | | `create-package` | **Required** | Inherits `assemble` and `verify-assembly` | | `package` | None | Local `tar` | | `verify-package` | None | Local extract and checksum comparison | | `mirror` | None | Destination registry, over your local network | | `retarget` | None | Local files | | `apply` | None | Destination registry. This is `mirror` plus `retarget` | | `cleanup`, `version`, `help` | None | Local | **Split the pipeline at the tarball, not earlier.** `create-package` cannot run on an isolated host: it fails in preflight with no route to the source registry. ## Commands | Command | What it does | |---|---| | `validate` | Preflight. The manifest exists and its schema is sane, every image and chart is digest-pinned, the tooling resolves, and the source registry is readable. Fetches one image manifest, so a bad host, missing login, TLS mismatch or unpublished digest fails here rather than partway through a multi-gigabyte download | | `assemble` | Creates the bundle directory, pulls every image and chart, extracts each chart's production values file, copies in the manifest and itself. Fails if the directory exists | | `assemble-charts` | The charts half. **What the registry install method uses**, since the cluster pulls its own images | | `assemble-images` | The images half. Separable so a failed multi-hour pull can be retried without re-fetching charts | | `verify-assembly` | Keyless signature verification of every pinned image and chart, then writes `SHA256SUMS` and prints the bundle **fingerprint**. Reports every failure and refuses to continue on any | | `package` | Tars the bundle directory into a tarball **beside** it, never inside | | `verify-package` | Unpacks to a temporary directory and re-checks `SHA256SUMS` | | `create-package` | `assemble`, `verify-assembly`, `package`, `verify-package`, in order | | `mirror` | Verifies the bundle against `SHA256SUMS`, then pushes every image to your registry | | `retarget` | Rewrites the registry host in the bundle's own values files, in place, and updates the affected `SHA256SUMS` entries so the bundle stays self-consistent | | `apply` | `mirror` then `retarget`. The usual customer step | | `cleanup` | Removes every directory and archive the bundler created | | `version` | The bundler's version, stamped at release | | `help` | Usage | Every command runs `validate` first, so any of them is safe to invoke on its own. ## Options | Option | Effect | |---|---| | `--arch amd64` or `--arch arm64` | Single-architecture bundle, roughly half the size. **Loses SBOM attestations.** See below | | `-o ` | A different tarball name | | `-r ` | Destination registry for `mirror`, `retarget` and `apply` | | `--dangerously-skip-cosign-verify` | Drops signature verification and its tooling requirement. The bundle is annotated as unverified | ### The --arch trade-off A single-architecture bundle extracts one manifest out of each multi-architecture index, which changes every image digest by definition. | | Default | With `--arch` | |---|---|---| | Image signatures | Verify | **Verify.** Each per-architecture manifest is signed individually | | SBOM attestations | Verify | **Lost.** They hang off the index, which is what is left behind | | Digests match the manifest | Yes | No | | Tarball and directory name | `multiarch` | Named after the architecture, so substitute it everywhere | If your acceptance process checks attestations, or your nodes are mixed, or you are not certain what they run: take the default. A missing architecture fails when the kubelet tries to pull, on an isolated cluster, after the transfer. ## Environment variables | Variable | Purpose | |---|---| | `MIKEFARAH_YQ_PATH` | Path to the Go `yq`, when the one on `PATH` is the Python one | | `BUNDLE_TLS_VERIFY=false` | Disables TLS verification on the **pull** side | | `MIRROR_TLS_VERIFY=false` | Disables TLS verification on the **push** side, for a plain-HTTP registry | The two TLS variables are the only places in the whole install where verification can be disabled. Prefer a certificate from your internal CA. See [Hardening](./../security/hardening.mdx). ## Credentials, and the three stores Three tools, three credential stores. You log in to two of them. | Tool | Store | Login needed | |---|---|---| | `skopeo` | The containers store | Yes, both source and destination | | `helm` | `HELM_REGISTRY_CONFIG` | Yes, source only | | `cosign` | The Docker config | **No.** The bundler lends it the one `skopeo` holds | If you see the bundler say it is reusing skopeo's credential for cosign, that is the expected path and not a warning. ## Behavior worth knowing | | | |---|---| | `assemble` refuses to write into an existing directory | So a stale bundle cannot be silently mixed with a new one | | `retarget` takes no file arguments | Rewriting arbitrary paths handed in on a command line is not something an isolated install should be able to ask for | | `retarget` verifies integrity **before** editing, never after | That order is what stops a checksum update from re-blessing a bundle that arrived altered | | `retarget` is idempotent | Running `apply` twice is safe | | `mirror` runs before `retarget` inside `apply` | So a registry failure leaves your values files untouched | | Mirroring is additive | New tags land beside old ones, so a running install keeps working and a rollback still has its images | | An egress proxy is fine for the bundler | Provided the host can reach the registry and Sigstore. Unlike the relay, these are ordinary CLI tools that honour proxy variables | ## Bundle contents ``` gen0sec-cerebellum--/ ├── oci/ every image, as an OCI layout ├── charts/ installable chart archives ├── values/ each chart's production values, retargeted by `apply` ├── release-manifest.yaml the digests this bundle was built from ├── bundler.sh a copy └── SHA256SUMS over every file above ``` `assemble` extracts each chart's values file as `values/-values-onprem.yaml`. There is no bare `values-onprem.yaml` in the bundle root, which is why the install commands name the file in full. The bundle does **not** carry the install kit's `scripts/` or `examples/`. Those cross the air gap inside the kit directory. See [Install kit contents](./kit.mdx). --- # Release notes Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/reference/changelog # Release notes ## v0.1.0 First on-premises release. ### Installing Two methods, both documented on one procedure that branches once: [install from our registry](./../04-prepare-artifacts.mdx), or [create an offline bundle](./../04-prepare-artifacts.mdx) and mirror it into your own. ### Requires | | | |---|---| | Kubernetes | 1.23 or later | | Nodes | 3 for production, 16 vCPU and 32 GiB each recommended | | Storage | [sized to your retention](./../02-requirements.mdx#sizing-storage-for-your-retention) | | Cluster egress | `api.gen0sec.com:443`, required | ### Credentials you need **Two**, and the second is easy to miss. Both are self-service from the dashboard: 1. A registry token, for pulling charts and images. Created under **Repository Access**. The install kit itself is public and needs no credentials. 2. **Your Gen0Sec API key**, which the relay uses to fetch data from us. Without it the whole fleet receives empty artifacts with no error. See [Credentials from gen0sec](./../02-requirements.mdx#credentials-from-gen0sec). ### Not in this release | | | |---|---| | Zero-egress deployment | [Known limitations](./../troubleshoot/known-limitations.mdx#no-zero-egress-deployment) | | Explicit egress proxy for the relay | [Known limitations](./../troubleshoot/known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy) | | Single sign-on of any kind | Email and password only. Microsoft Entra ID is in development | | Shipped dashboards and alert rules | [Monitor](./../operate/monitor.mdx) lists what to watch meanwhile | | The threat service | Not shipped at this release | --- ## How we version | | | |---|---| | Scheme | Semantic versioning. Each minor is a release train | | Supported | The latest two minor trains | | Upgrades | Sequential minors, patches in place. **Skipping a minor is not supported** | | Release candidates | Not published to `releases.gen0sec.com`. Only stable releases are mirrored | Full policy: [Component versions](./../configure/components.mdx#support-policy). ## Finding a release ```bash curl -fsSL https://releases.gen0sec.com/api/repos/cerebellum/latest | jq -r .tag_name ``` Returns the latest full release. **Pin the tag explicitly for an install you want to reproduce**, so a rebuild in six months fetches the same bytes. ## What every release publishes | Artifact | Where | |---|---| | Install kit and its signature | `https://releases.gen0sec.com/cerebellum//` | | Images and charts | The registry, each pinned to a digest in the kit's release manifest | | Signatures and SBOM attestations | Alongside each image and chart, as OCI referrers | See [Supply chain](./../security/supply-chain.mdx). --- # Install kit contents Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/reference/kit # Install kit contents ## What you download Two files, published per release at `https://releases.gen0sec.com/cerebellum//`. | File | What | |---|---| | `cerebellum-install-kit.tar.gz` | The kit | | `cerebellum-install-kit.tar.gz.cosign.bundle` | Its detached signature | Verify before unpacking. See [Prepare the artifacts](./../04-prepare-artifacts.mdx#3-verify-the-signature). ## Layout ``` cerebellum-install-kit-/ ├── bundler.sh builds and loads the offline bundle ├── release-manifest.yaml every image and chart, pinned to a digest ├── scripts/ │ ├── preflight.sh cluster checks the charts assume but do not create │ ├── make-db-secrets.sh composes the ~20 application secrets │ ├── uninstall.sh the three releases, finalizers, namespaces │ └── verify-deployment.sh external smoke test, read-only ├── examples/ │ ├── single-node/ three overlays, one replica each │ └── single-entrypoint/ one address for the dashboard and every API ├── offline-data/ │ └── README.txt └── SHA256SUMS over every file above ``` **No images and no charts.** Those come from the registry, or from the offline bundle. ## The scripts | Script | Run it | Documented in | |---|---|---| | `preflight.sh` | Before installing | [Install step 1](./../05-install.mdx#1-check-the-cluster) | | `make-db-secrets.sh` | After the databases are ready, before the platform | [Install step 6](./../05-install.mdx#6-create-the-application-secrets) | | `verify-deployment.sh` | After every install and every upgrade | [Verify the install](./../08-verify.mdx#4-the-platform-serves-an-agent) | | `uninstall.sh` | When removing the installation | [Uninstall](./../operate/uninstall.mdx) | `verify-deployment.sh` reads two environment variables: ```bash export GEN0SEC_BASE_URL=https://cerebellum.example.internal export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) ``` ## The examples Everything under `examples/` is a starting point you are meant to read and edit. Each file carries comments marking which values are yours. | Directory | Purpose | |---|---| | `single-node/` | Three overlays that reduce every component to one node. [Quickstart](./../quickstart.mdx) | | `single-entrypoint/` | One address for the dashboard and every agent-facing API. [Publish the endpoints](./../06-publish.mdx) | ## Verifying the kit's own integrity ```bash cd $KIT && sha256sum -c SHA256SUMS ``` `SHA256SUMS` covers the kit **as delivered**. Adding files makes it no longer match, which is expected once you start using it. ## What the offline bundle method has to transfer Two things, and the order matters. | Transfer | Contains | |---|---| | The bundle tarball | Images, charts, values, the manifest, a copy of the bundler | | The kit directory | `scripts/`, `examples/`, and the manifest | The bundle carries its own copy of `bundler.sh`, so loading it needs no tooling from the kit. **The kit still has to cross**, because the install uses its scripts and its examples. Copy the bundle first, then clean up, then the kit. `create-package` runs inside the kit directory, so the bundle is sitting in it, and copying the kit first transfers tens of gigabytes twice. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). Both archives unpack side by side: ``` ~/gen0sec-cerebellum-multiarch-0.1.0/ the bundle ~/cerebellum-install-kit-0.1.0/ the kit ``` ## The offline-data directory The kit creates `offline-data/` containing a single `README.txt`, and nothing else. It is a placeholder for a data-delivery path that is not part of this release: the four datasets the platform needs are fetched at runtime through the relay, not delivered as files. See [Data relay](./../configure/relay.mdx). If you were told to populate it, check with us first. At this release you should not need to. --- # Requirements Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/requirements # Requirements ## Cluster | | Requirement | |---|---| | Kubernetes | 1.23 or later | | Access | `cluster-admin`, for the install only. See [Privileges and RBAC](./security/privileges.mdx) for what it is used for and what it leaves behind | | StorageClass | A default StorageClass on block storage. **Volume expansion recommended** | | Supported architecture | `linux/amd64` or `linux/arm64`. Mixed-architecture clusters are supported | | LoadBalancer | The ingress data plane is exposed as a `LoadBalancer` type Service by default. See the note below | :::note[The ingress needs to be reachable from outside the cluster] The ingress data plane is a `LoadBalancer` type Service, because nothing it programs can serve traffic if it stays cluster-internal. Your cluster needs a LoadBalancer implementation. If your cluster fronts ingress another way, a NodePort, a hardware balancer or an external proxy, change the Service type to match. ::: Cerebellum brings its own data plane - which contains Postgres, Kafka, object store and cache. You do not connect it to existing instances of any of them. :::info[Maintenance] We've deliberately chosen to use well-known solutions from the software industry in our data layer to ease the burden of maintenance. While Cerebellum ships this data layer by default, to make the setup easier, the data in it is the highest importance to you. So make sure you monitor the dataplane and do backups and maintenances regularly. ::: :::note[Container runtime, for the registry install method only] Our registry challenges pulls that carry no `User-Agent`, and on some distributions the kubelet's pull path sends none. containerd's own user agent passes; the problem is its absence. Either configure one on every node, or use the offline bundle method, which pulls into your own registry instead. Settle this before you choose a method. See [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: ### Kubernetes versions 1.23 is the floor because `autoscaling/v2` is the newest API the charts use, and it reached GA in that release. While anything from 1.23 and up should work, we recommend you use the latest stable version. ## Sizing a production install Production runs on **3 nodes**. Several data plane members each form a three-member quorum, so three is the floor for redundancy, not a suggestion. ### CPU & memory per node | | Resources | |---|---| | **Minimum** | 8 vCPU, 16 GiB | | **Recommended** | 16 vCPU, 32 GiB | ### Storage per node We recommend to start your nodes with `500 GiB` **minimum**. Several of the data plane members use PVCs. While the defaults are kept low - as your platform matures the data will grow and you'll need to expand the storage volumes. ### Sizing storage for your retention The shipped sizes are a starting point, not a capacity plan. They are deliberately modest, so for most production workloads you will be growing them rather than shrinking them. Four keys control everything: | Key | Chart | Default | |---|---|---| | `postgres.volume.size` | `gen0sec-data` | 50 GiB | | `kafka.storage.size` | `gen0sec-data` | 50 GiB | | `rustfs.storageclass.dataStorageSize` | `gen0sec-infra` | 50 GiB | | `rustfs.storageclass.logStorageSize` | `gen0sec-infra` | 10 GiB | Which one to grow depends on what you are retaining: | If you need | Grow | |---|---| | Longer event and telemetry history | `postgres.volume.size`. This is the system of record | | A longer Kafka retention window, or a bigger ingest buffer | `kafka.storage.size` | | More room for datasets and locally-produced artifacts | `rustfs.storageclass.dataStorageSize` | Example for longer retention: ```yaml # gen0sec-data postgres: volume: size: 200Gi kafka: storage: size: 200Gi ``` ```yaml # gen0sec-infra rustfs: storageclass: dataStorageSize: 100Gi ``` :::note[Plan for growth] Volume expansion grows a volume if your StorageClass supports it, and it is straightforward. Nothing shrinks one: reducing a size means recreating the volumes and losing what is on them. Starting at the shipped defaults and growing is therefore the recommended direction. It is the reason for low initial limits. Check that your StorageClass has `allowVolumeExpansion: true` before you install. ::: ### What is not on this page **Throughput.** How many agents, events per second or lookups a given topology sustains is not directly derivable from the charts' parameters. We suggest you scale the platform components and storage sizes according to your topology's specific needs. {/* Four workloads declare no CPU or memory requests: the object store, the Kafka brokers, valkey, and the Postgres operator. They are therefore BestEffort or Burstable and are evicted first under node pressure, which is why the per-node recommendation is driven by limits rather than by the requests total. Raised separately as a chart issue. */} ### Evaluating on one node Use the [Quickstart](./quickstart.mdx). It runs everything at a single replica on one node, for evaluation, training and upgrade rehearsals. ## Tools on your workstation Install these on the machine you run the commands from, not on the cluster. | Tool | Version | Used for | Install | |---|---|---|---| | `helm` | 3.16 or later | Installing the charts, and pulling them from the registry | [helm.sh](https://helm.sh/docs/intro/install/) | | `kubectl` | matching your cluster | Everything cluster-side | [kubernetes.io](https://kubernetes.io/docs/tasks/tools/#kubectl) | | `cosign` | 2 or later | Verifying signatures on the install kit, images and charts | [docs.sigstore.dev](https://docs.sigstore.dev/cosign/system_config/installation/) | | `skopeo` | any | Copying images between registries | [github.com/containers/skopeo](https://github.com/containers/skopeo/blob/main/install.md) | | `yq` | 4 or later, **mikefarah/yq only** | Reading the release manifest | [github.com/mikefarah/yq](https://github.com/mikefarah/yq#install) | | `jq` | any | Reading the release API | [jqlang.github.io](https://jqlang.github.io/jq/download/) | | `aws` or `mc` | any | Creating the object store bucket, one step during install | [aws-cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) or [MinIO client](https://min.io/docs/minio/linux/reference/minio-mc.html) | | `curl`, `tar`, `openssl`, `sha256sum` or `shasum` | any | Downloading, unpacking, generating secrets, checking checksums | Your OS packages | ### Check your yq There are two separate tools that are called `yq`. The Python one (`kislyuk/yq`) cannot parse the expressions the bundler uses, and it fails with a syntax error partway through. ```bash yq --version ``` The output must contain `mikefarah`. If it does not, install the Go version and point the bundler at it: ```bash export MIKEFARAH_YQ_PATH=/path/to/mikefarah-yq ``` ## Credentials from Gen0Sec You need **two**, and they are different things. Missing the second one produces a working-looking install whose entire fleet silently receives nothing. | Credential | Used by | Used for | |---|---|---| | **Registry token** | The bundler, on your workstation | Pulling charts and images from our registry | | **Your Gen0Sec API key** | The relay pod, inside your cluster | Fetching threat intelligence, GeoIP, models and IDS rules from Gen0Sec | Both are self-service from the dashboard. Neither requires contacting us. :::note[The install kit itself needs no credentials] The kit is published openly. Downloading and verifying it, in [Prepare the artifacts](./04-prepare-artifacts.mdx), happens before you use either credential — so if that download fails, the token is not what to check. The usual cause is a tag that is not published; the [release index](https://releases.gen0sec.com/cerebellum/) lists the ones that are. ::: ### The registry token Create it on the dashboard, under **Repository Access**: ``` https://app.gen0sec.com//repositoryAccess ``` Your workspace id is in the address bar once you are signed in, and the same page is linked from **Clients**. The token is shown **once**, at creation — store it before leaving the page, because it cannot be retrieved afterwards and a lost one has to be replaced. **The username is literally `any`.** Wherever a command asks for one, type `any`. Only the token is checked, so the username can be anything at all — `any` is simply what these docs use so the commands are copy-pasteable. ### Your Gen0Sec API key You can create your API key on the [SaaS Dashboard](https://app.gen0sec.com) Scopes required: - `ids_rules:read`, - `indicators:read`, - `ml_models:read` :::note[Why this one is separate from the registry token] Your agents get their API keys from your own dashboard, and those keys exist only in your cluster's database, so they cannot authenticate to Gen0Sec. The API key created in this step is used by your in-cluster relay to authenticate against the Gen0Sec SaaS, so it can fetch data related to threat intelligence, GeoIP, ML models and IDS rules. The chart expects it as a secret named `gen0sec-download-proxy`, and the relay pod will not start without it, so a missing key is a failed rollout rather than a silently degraded fleet. You will create this secret in the step 2 of [Install](./05-install.mdx). ::: --- # Architecture and data flows Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/security/architecture # Architecture and data flows Written for a security reviewer. Everything here is observable in a running install, and every claim has a page you can check it against. ## What runs where Two namespaces, three charts, one ingress point. ``` internet │ │ api.gen0sec.com:443 (the only outbound path) │ ┌───────────────────────────┼──────────────────────────────────────────────┐ │ your cluster │ │ │ ▲ │ │ ┌─────────────────────── │ ─────────────────────────────────────────┐ │ │ │ namespace: gen0sec │ │ │ │ │ │ │ │ │ │ dashboard relay(download-api) 13 application services │ │ │ │ │ │ │ └───────────────────────────────┬───────────────────────────────────┘ │ │ │ in-cluster only │ │ ┌───────────────────────────────┼───────────────────────────────────┐ │ │ │ namespace: gen0sec-system │ │ │ │ │ ▼ │ │ │ │ Postgres (3) Kafka (3) object store (4) caches (2) │ │ │ │ Postgres operator Kafka operator ingress operator │ │ │ └───────────────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────┘ ▲ │ one address, TLS, path-prefix routed │ your agents and your users ``` | Namespace | Contains | Reachable from outside | |---|---|---| | `gen0sec` | The application services and the dashboard | Yes, through the ingress you publish | | `gen0sec-system` | Postgres, Kafka, object store, two caches, and the three operators | No | Nothing in `gen0sec-system` is published. The application reaches it by cluster DNS. ## Trust boundaries Four, and they are worth naming separately because they fail differently. | Boundary | Crossed by | Controlled by | |---|---|---| | **Internet to cluster** | The relay, outbound only, one destination | Your firewall. See [Ports and egress](./egress.mdx) | | **Your network to cluster** | Agents and users, inbound, one address | Your ingress and its TLS | | **Namespace to namespace** | The application reading its databases | Cluster DNS. No NetworkPolicy ships. See [Hardening](./hardening.mdx) | | **Cluster to registry** | Image pulls | The image pull secret, one per namespace | ## Data flows ### Inbound: agents and users | Flow | Path | Authentication | |---|---|---| | Agent telemetry and events | Your address, then the ingest services, then Kafka, then Postgres | Agent API key | | Agent configuration pull and push | Your address, then the configuration services | Agent API key | | Agent artifact download | Your address, then the relay, then Gen0Sec | Agent API key, validated locally. The relay uses your installation's own key upstream, so agent keys never leave. See [Data handling](./data-handling.mdx) | | Dashboard sign-in | Your address, then the dashboard, then Postgres | Email and password, held in your Postgres. No external identity provider at this release | ### Outbound: the relay, and nothing else One destination, one purpose: fetching four licensed datasets that are in no artifact we ship. | Dataset | Why it cannot be shipped | |---|---| | CTI indicators | Licensed by upstream providers | | GeoIP databases | Licensed by MaxMind and IPInfo | | ML models | Published by gen0sec, refreshed independently of releases | | IDS rules | Licensed as part of the Emerging Threats ruleset | The relay is a **relay, not a cache**. Nothing is stored on the way through, so there is no local artifact store to size and no staleness window. ### Never outbound | Dataset | What it is | Where it stays | |---|---|---| | `identity` | Workload identity produced in your cluster | Your object store | | `policy-edges` | Pod IPs, namespaces, NetworkPolicy edges | Your object store | Relaying either would push a map of your workload topology to a service you do not run. It is not configurable and there is no flag that enables it. Uploads are not relayed either: the relay is read-only. ## Where data is stored | Store | Contains | Backed up by you | |---|---|---| | Postgres | The system of record: configuration, agents, policy, history | Yes, always | | Object store | Datasets, and the two locally-produced sets above | Yes | | Kafka | In-flight events between ingest and persistence | No, it is transport | | Cache | Nothing durable | No | See [Back up and restore](./../operate/backup.mdx). ## What the platform requires of your cluster | Requirement | Detail | |---|---| | `cluster-admin` during install | For CRDs and cluster-scoped RBAC. See [Privileges and RBAC](./privileges.mdx) | | Custom resource definitions | Postgres and Kafka operator CRDs, cluster-scoped by nature | | Its own ingress controller | Installed by the infra chart, `ingressClassName: synapse` | | One outbound destination | `api.gen0sec.com:443` | | No node access, no host mounts, no privileged containers | See [Hardening](./hardening.mdx) | ## Questions a reviewer usually asks next | Question | Page | |---|---| | What exactly can leave the cluster? | [Data handling](./data-handling.mdx) | | What ports and destinations do I open? | [Ports and egress](./egress.mdx) | | Why does it need `cluster-admin`? | [Privileges and RBAC](./privileges.mdx) | | How do I verify what you shipped me? | [Supply chain](./supply-chain.mdx) | | What is the container security posture? | [Hardening](./hardening.mdx) | | What does not work yet? | [Known limitations](./../troubleshoot/known-limitations.mdx) | --- # Data handling Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/security/data-handling # Data handling ## What leaves your cluster One connection, outbound, to one destination. | | | |---|---| | Destination | `api.gen0sec.com:443` | | Made by | The relay pod only | | Direction | Outbound, request and response | | Purpose | Fetching four licensed datasets | **What is in the request.** A path naming the artifact, and your own Gen0Sec API key. Nothing else. The relay does not add a customer identifier, a cluster identifier, an agent identifier, telemetry or usage data. **What is in the response.** The artifact: a CTI indicator database, a GeoIP database, an ML model, or an IDS rule bundle. ## What never leaves your cluster | Data | What it is | |---|---| | `identity` | Workload identity produced inside your cluster | | `policy-edges` | Pod IP addresses, namespaces, NetworkPolicy edges | | Agent telemetry and events | Everything your agents report | | Configuration and policy | Everything you configure | | Logs | All of it, to container stdout in your cluster | | Database contents | Postgres never leaves | | Dashboard user accounts and passwords | Held in your Postgres | `identity` and `policy-edges` deserve the explicit statement, because they are the two that are technically served by the same service that does the relaying. **They are never relayed.** Doing so would push a map of your workload topology to a service you do not run. It is not configurable, and no flag enables it. The upload surface is not relayed either, because the relay is read-only. ## Who validates agent credentials **Your agents' keys are validated in your cluster, and they never leave it.** Every relayed route sits behind the local authentication middleware, and each one carries the same permission it carries in direct mode. So a key that would be refused when served locally is refused it when relayed. ``` agent --[agent key, authorized locally, permission checked per route]--> relay --[your Gen0Sec API key]--> api.gen0sec.com agent key stops here ``` Two separate concerns, deliberately separated: | Hop | Credential | Checked by | |---|---|---| | Agent to relay | The agent's own key | Your cluster, against your installation, with the artifact's permission | | Relay to Gen0Sec | Your Gen0Sec API key | Gen0Sec | | | | |---|---| | Credentials Gen0Sec receives | One, belonging to your installation | | Credentials Gen0Sec never receives | Your individual agent key | | What Gen0Sec can infer about your fleet | Nothing from the credential. Requests carry no agent or cluster identifier | | Revoking one agent | Local, in your dashboard, immediate | | Revoking your installation's upstream access | Revoke or rotate your Gen0Sec API key | This is the answer to "does our agent inventory leave our perimeter". It does not. :::note[Agent keys are never forwarded upstream, even by accident] On a self-hosted deployment the relay never puts a caller's key in an outbound request, in any configuration. A locally-issued key could not authenticate to Gen0Sec anyway, so forwarding it could only leak a local credential to a third party without making the request succeed. With no Gen0Sec API key configured the relay sends no credential at all rather than sending yours. ::: ## Credentials at rest | Credential | Where it lives | Generated by | |---|---|---| | Database role passwords | Kubernetes secrets | The Postgres operator | | Object store keys | A Kubernetes secret | You, during install | | Registry pull credentials | A Kubernetes secret per namespace | You, during install | | Dashboard session signing key | A Kubernetes secret | Generated once, preserved across regeneration | **No credential appears in a chart, a values file or a rendered manifest.** Every one is referenced by name. See [Secrets](./../configure/secrets.mdx). The install procedure reads every credential from a file or from standard input rather than passing it as a command-line argument, so nothing lands in shell history or in the process list. ## Credentials in transit | Hop | Encrypted | |---|---| | Agents and users to your ingress | Yes, TLS you terminate | | Relay to `api.gen0sec.com` | Yes, TLS | | Services to Postgres | Yes, TLS to the pooler | | Services to the object store | Plain HTTP, in-cluster only | | Services to the cache | Plain, in-cluster only, no credential in the URL | The last two are in-cluster traffic on the pod network. If your threat model includes an attacker on the pod network, add NetworkPolicies and a service mesh. See [Hardening](./hardening.mdx). :::note[One deliberate exception, and why] The dashboard's database connection verifies encryption without verifying the certificate chain, because its client library rejects the pooler's self-signed certificate outright while every other service's library accepts encrypt-without-verify. The traffic is encrypted and in-cluster. Do not copy that setting onto the other services: their driver rejects it. ::: ## Data retention Retention is yours. Nothing expires data on your behalf, and nothing is sent anywhere before it is deleted. | Store | Grows with | Sized by | |---|---|---| | Postgres | Agents, events, history | `postgres.volume.size` | | Kafka | In-flight events, and its retention window | `kafka.storage.size` | | Object store | Datasets and locally-produced artifacts | `rustfs.storageclass.dataStorageSize` | See [Sizing](./../02-requirements.mdx#sizing-a-production-install). ## Deleting everything [Uninstall](./../operate/uninstall.mdx) without `--keep-data` deletes the namespaces and every persistent volume in them: the database, the message log and the object store. There is no copy anywhere else. --- # Ports and egress Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/security/egress # Ports and egress Same as [Network and connectivity](./../03-network.mdx), reproduced here for better visibility. **One canonical source.** If this table and the install page ever disagree, that is a bug in our publishing, not a judgement call for you. Report it. ## Outbound All TCP, all TLS, all outbound. Nothing needs an inbound rule from the internet. | Destination | Port | Needed by | Purpose | Required | |---|---|---|---|---| | `api.gen0sec.com` | 443 | Cluster pods, the relay only | Threat intelligence, GeoIP, ML models, IDS rules | **Always** | | `releases.gen0sec.com` | 443 | Workstation | Download the install kit and its signature | **Always** | | `registry.gen0sec.com` | 443 | Workstation | Pull charts and images | **Always** | | `registry.gen0sec.com` | 443 | Cluster nodes | Image pulls at install time | Registry install only | | `fulcio.sigstore.dev` | 443 | Workstation | Signature certificate verification | **Always** | | `rekor.sigstore.dev` | 443 | Workstation | Transparency log verification | **Always** | | `tuf-repo-cdn.sigstore.dev` | 443 | Workstation | Signature trust root refresh | **Always** | | Your own registry | 443 or 80 | Cluster nodes, and the host that loads the bundle | Image pulls at install time | Offline bundle only | **Only one row applies to the running cluster in steady state:** `api.gen0sec.com:443`, and only the relay pod uses it. Everything else is install-time, and most of it is your workstation rather than your cluster. ## Inbound | Source | Destination | Port | Purpose | |---|---|---|---| | Your agents | Your published address | 443 | Telemetry, events, configuration, artifact download | | Your users | Your published address | 443 | Dashboard | One address serves both, routed by path prefix. See [Publish the endpoints](./../06-publish.mdx#the-route-table). ## In-cluster No NetworkPolicy is required, and none ships. If you enforce default-deny, these are the flows to allow. | From | To | Port | |---|---|---| | `gen0sec` services | `core-pooler.gen0sec-system` (Postgres pooler) | 5432 | | Migration job | `core.gen0sec-system` (Postgres primary) | 5432 | | `gen0sec` services | `core-kafka-bootstrap.gen0sec-system` | 9092 | | `gen0sec` services | `rustfs-svc.gen0sec-system` (object store) | 9000 | | `gen0sec` services | `dragonfly.gen0sec-system` (cache) | 6379 | | Ingress controller | `gen0sec` services | Service ports | See [Hardening](./hardening.mdx) for what we recommend beyond that. ## Notes that save time **The Sigstore hosts are the ones people forget.** Signature verification is the first command of the install, and it fails without them. They are needed on the workstation, not on the cluster. **A proxy does not help for the cluster's one destination.** Explicit forward proxies are not honoured in-cluster. See [Known limitations](./../troubleshoot/known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy). **The offline bundle removes registry egress, not all egress.** The runtime connection stays. See [What this rules out](./../03-network.mdx#what-this-rules-out). --- # Hardening Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/security/hardening # Hardening Two lists: what the charts already do, and what is left to you. The second list is short but it is not empty, and we would rather name it than let you assume otherwise. ## Hardened by default Every application service runs with this security context, and it is not opt-in. ```yaml podSecurityContext: runAsNonRoot: true runAsUser: 65532 runAsGroup: 65532 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: [ALL] ``` | Control | Status | |---|---| | Runs as non-root | Yes, uid 65532 | | Read-only root filesystem | Yes | | Privilege escalation | Disabled | | Linux capabilities | All dropped | | Privileged containers | None | | Host network, PID or IPC | None | | Host path mounts | None | | Node access | None | :::note[Read-only root filesystems were not free] Several services defaulted to writing into a relative path under their working directory, which fails outright on a read-only root. Each one now gets an explicit writable volume mounted where it expects to write. That is why the values file carries per-service volume mounts: they are the cost of this control, not decoration. If you add a service-level override that removes one, that service will fail to start. ::: ## Pod Security Admission The application pods satisfy the **restricted** profile. Label the namespace to enforce it: ```bash kubectl label namespace gen0sec \ pod-security.kubernetes.io/enforce=restricted \ pod-security.kubernetes.io/enforce-version=latest ``` **Do not label `gen0sec-system` restricted yet.** The Postgres and Kafka pods there are created by upstream operators and have not been assessed against that profile. Use `baseline` if you need something, and tell us if you require restricted there. ## What you should add None of these ship, and all of them are worth doing. ### NetworkPolicies **No NetworkPolicy ships with the charts.** On a cluster without default-deny, that means the platform namespaces are as open as everything else. Add default-deny in both namespaces, then allow exactly these: | From | To | Port | |---|---|---| | `gen0sec` services | `core-pooler.gen0sec-system` | 5432 | | Migration job | `core.gen0sec-system` | 5432 | | `gen0sec` services | `core-kafka-bootstrap.gen0sec-system` | 9092 | | `gen0sec` services | `rustfs-svc.gen0sec-system` | 9000 | | `gen0sec` services | `dragonfly.gen0sec-system` | 6379 | | Ingress controller | `gen0sec` services | Service ports | | Relay pod | `api.gen0sec.com:443` | 443, egress | **Egress default-deny is the high-value one here.** The relay is meant to be the only pod with a route out. A NetworkPolicy is what turns that from a design intention into an enforced property, and it is the single strongest control you can add to this deployment. ### TLS everywhere it is not already | Hop | Today | Worth adding | |---|---|---| | Ingress to users and agents | TLS you terminate | Already yours | | Relay to Gen0Sec | TLS | Nothing to do | | Services to Postgres | TLS to the pooler | Nothing to do | | Services to the object store | Plain HTTP, in-cluster | A service mesh, if your threat model includes the pod network | | Services to the cache | Plain, in-cluster, no credential in the URL | Enable cache authentication, then move the connection string into each service's secret | ### Resource limits on the four workloads that have none The object store, the Kafka brokers, the ingress cache and the Postgres operator declare no CPU or memory requests. That makes them BestEffort or Burstable, so the kubelet evicts them first under node pressure. For the object store, holding your data, that is the wrong priority order. Set requests on them, or set a namespace `LimitRange` so nothing lands unbounded. See [Sizing](./../02-requirements.mdx#sizing-a-production-install). ### Secrets from an external manager `make-db-secrets.sh` is fine for evaluation. In production, have your secret manager produce secrets of the same shape so credentials never exist as plain Kubernetes secrets you created by hand. Every name and key is listed in [Secrets](./../configure/secrets.mdx). ## Settings you should never enable Two environment variables exist and both disable a security control. Neither is set by the charts, and neither appears anywhere in the install procedure. | Setting | What it does | When it is acceptable | |---|---|---| | `TLS_INSECURE_SKIP_VERIFY` | Stops the relay verifying the upstream certificate | Never in production. It defeats the protection on your only outbound connection | | `MIRROR_TLS_VERIFY=false` | Stops the bundler verifying your registry's certificate when pushing | Only for a plain-HTTP registry on a trusted network, and preferably not then | If you find either one set in a running deployment, treat it as a finding. ## Verifying the posture Confirm no pod runs as root or privileged: ```bash kubectl -n gen0sec get pods -o json | jq -r ' .items[] | .metadata.name + " runAsNonRoot=" + ((.spec.securityContext.runAsNonRoot // false) | tostring)' ``` Confirm nothing holds cluster-admin: ```bash kubectl get clusterrolebinding -o json \ | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | .metadata.name' ``` Confirm only the relay can reach the internet, once you have added egress policy: ```bash kubectl -n gen0sec run netcheck --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/busybox:1.36.1-glibc -- wget -T5 -qO- https://example.com ; echo "exit=$?" ``` A non-zero exit is the result you want. --- # Privileges and RBAC Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/security/privileges # Privileges and RBAC ## The short answer `cluster-admin` is required **to install**, not to run. It is needed for two things that are cluster-scoped by nature: 1. Installing custom resource definitions. 2. Creating the cluster roles the three operators need. Once installed, no component in the platform holds cluster-admin. ## What is cluster-scoped ### Custom resource definitions From the bundled operators, not written by us. | CRD group | Installed by | Kinds | |---|---|---| | `acid.zalan.do` | Postgres operator | `postgresqls`, `operatorconfigurations`, `postgresteams` | | `kafka.strimzi.io` | Kafka operator | `kafkas`, `kafkatopics`, `kafkanodepools`, `strimzipodsets`, and the Connect and Bridge kinds we do not use | CRDs are cluster-scoped in Kubernetes. There is no namespaced alternative. :::note[Deleting a CRD deletes every resource of that kind, cluster-wide] Including any outside these two namespaces. If anything else in your cluster runs a Postgres or Kafka operator, removing these CRDs would take its databases and clusters with them. [Uninstall](./../operate/uninstall.mdx) removes them by default, but checks first: if any of those kinds is in use outside the two namespaces it is removing, it skips the CRDs and says so. `--keep-crds` skips them unconditionally. ::: ### Cluster roles | Cluster role | Belongs to | What it is for | |---|---|---| | `g0s-postgres-operator` | Postgres operator | Managing Postgres clusters and their pods | | `g0s-postgres-operator-endpoints` | Postgres operator | The `endpoints` permission its own upstream role omits, while still creating a master Endpoints object per cluster | | `postgres-pod` | Postgres pods | What a database pod itself needs | | `strimzi-cluster-operator-global` | Kafka operator | Cluster-scoped reconciliation | | `strimzi-cluster-operator-namespaced` | Kafka operator | Namespaced resources, bound per namespace | | `strimzi-cluster-operator-leader-election` | Kafka operator | Leader election | | `strimzi-cluster-operator-watched` | Kafka operator | Watching the namespaces it manages | | `strimzi-kafka-broker` | Kafka brokers | What a broker pod needs | | `strimzi-entity-operator` | Topic and user operator | Managing topics and users | | `strimzi-kafka-client` | Kafka clients | Client-side permissions | | `g0s-synapse-operator` | Ingress operator | Watching Ingress objects and services | Eleven, and nine of them come straight from the two upstream operators. :::note[The one we added, and why] `g0s-postgres-operator-endpoints` exists because the Postgres operator ships a cluster role without `endpoints`, yet creates a master Endpoints object for every cluster it manages. Without that rule, `postgresql/core` fails with `endpoints is forbidden` and the database never starts. See [Postgres will not start](./../troubleshoot/symptoms.mdx#postgres-will-not-start). ::: ## What the application itself holds The application services in `gen0sec` are not operators and do not watch the cluster. | | | |---|---| | Cluster-scoped permissions | None | | Namespaced permissions | Reading their own configuration and secrets | | Service accounts | One per service | | Node access | None | | Host mounts | None | | Privileged containers | None | See [Hardening](./hardening.mdx) for the container security context. ## Can I install without cluster-admin? Partly, if you split the install between two roles. This is a supported pattern but it is not the documented golden path, so plan for one extra iteration. | Step | Needs | Who | |---|---|---| | Apply CRDs | Cluster-scoped write on `customresourcedefinitions` | Your platform team, once per release | | Create cluster roles and bindings | Cluster-scoped RBAC write | Your platform team, once | | Install the three charts | Namespace-admin on both namespaces | The installer | Render what a chart would create, and hand that to whoever holds the privilege: ```bash helm template g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system -f $B/values/gen0sec-infra-values-onprem.yaml \ | yq eval-all 'select(.kind == "ClusterRole" or .kind == "ClusterRoleBinding")' ``` ## Verifying after install Confirm nothing holds more than it should: ```bash kubectl get clusterrolebinding -o json \ | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | .metadata.name + " -> " + ([.subjects[]?.name] | join(","))' ``` No entry in that output should reference a `gen0sec` service account. If one does, tell us: that is a defect, not a configuration choice. ## What an uninstall leaves behind | | Removed | Reason | |---|---|---| | Namespaced resources | Yes | Helm owns them | | Objects the operators created outside the releases | Yes | The script removes them; `helm uninstall` alone does not | | CRDs | Yes, unless in use elsewhere or `--keep-crds` | Removal is cluster-wide destructive, so the script checks before it acts | | Cluster roles and bindings | Depends on the chart that created them | Check what remains and remove by hand if nothing else uses the operators | [Uninstall](./../operate/uninstall.mdx) reports what it removed and what it left. --- # Provenance, signatures and SBOM Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/security/supply-chain # Provenance, signatures and SBOM Everything we publish is signed, and every signature is verifiable by you without asking us for a key. This page also includes the one case where verification cannot follow the artifacts. ## What is signed, and how | Artifact | Signed | Attested | |---|---|---| | Install kit tarball | Detached cosign bundle, published beside it | | | Container images | `cosign sign --recursive`, so the index **and** each per-architecture manifest | SPDX JSON SBOM, one per architecture | | Helm charts | `cosign sign` on the OCI chart artifact | SPDX JSON SBOM | **Keyless, with GitHub OIDC.** There is no long-lived private key to steal, and no public key for you to fetch and trust. Each signature is bound to a workflow identity recorded in a public transparency log, so verification asserts *who built it*, not *who holds a key*. SBOMs are generated with syft and attached as SPDX JSON attestations. :::note[SLSA provenance is deliberately off] BuildKit's provenance attestations add manifests to the multi-architecture index carrying a placeholder platform. That breaks tools which walk an index by platform, including our own mirroring. We ship the SPDX SBOM attestation instead, which is the thing customers actually verify. If you require SLSA provenance, tell us: it is a build-flag change plus mirroring work, not a redesign. ::: ## Verify the install kit Do this before you unpack it. It is step 3 of [Prepare the artifacts](./../04-prepare-artifacts.mdx). ```bash cosign verify-blob \ --bundle cerebellum-install-kit.tar.gz.cosign.bundle \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/release\.yaml@refs/tags/' \ cerebellum-install-kit.tar.gz ``` **Both certificate flags are required.** `cosign` refuses to verify without an expected issuer and an expected identity, so a shortened command fails rather than passing weakly. The identity is pinned to **one workflow, on a tag reference**. A looser pattern would accept a signature produced by any workflow, on any branch, in our organization. That is the difference between "built by our release pipeline from a tag" and "built by somebody at Gen0Sec". ## Verify an image or a chart Against our registry, or against yours after mirroring: ```bash cosign verify \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/' \ registry.gen0sec.com/gen0sec/auth-api:0.1.0 ``` ## Read the SBOM ```bash cosign download attestation \ --predicate-type https://spdx.dev/Document \ registry.gen0sec.com/gen0sec/auth-api:0.1.0 \ | jq -r '.payload | @base64d | fromjson | .predicate' ``` The predicate is an SPDX JSON document. Feed it to whatever consumes SBOMs in your pipeline. ## Verification across the air gap This is the part nobody else documents, so it gets its own section. ### What crosses, and what does not Signature verification needs the public transparency log, and an isolated network does not have one. So the bundler verifies **once, on the connected staging host, before packaging**, and protects everything downstream with checksums. ``` connected staging host isolated network ────────────────────── ──────────────── pull every artifact by digest cosign verify each one (cannot re-verify: no transparency log) write SHA256SUMS -> verify-package re-checks SHA256SUMS print the bundle fingerprint -> mirror re-checks it before pushing package ``` | Stage | Guarantee | |---|---| | Before packaging | Full cryptographic signature verification, every image and every chart | | The tarball | `SHA256SUMS` over every file, plus a **fingerprint**, which is a sha256 of that list | | On the isolated side | `verify-package` and `mirror` both re-check `SHA256SUMS` and refuse to proceed on a mismatch | **Record the fingerprint** that verification prints on the connected side, out of band. Comparing it on the isolated side is what closes the loop between the two networks. :::note[Signature re-verification inside the isolated network is not supported] It would need the transparency log, or a pinned trust bundle carried across and kept current. We do not pretend otherwise. If your policy requires cryptographic verification at the destination rather than checksum verification, tell us. That is a different design, and we would rather discuss it now than have you find the gap during an audit. ::: ### The one case where attestations are lost `--arch amd64` or `--arch arm64` halves the transfer. It also extracts **one manifest out of each multi-architecture index**, which changes every image digest by definition. | | Multi-architecture bundle (default) | Single-architecture bundle | |---|---|---| | Image signatures | Verify | **Verify.** Each per-architecture manifest is signed individually | | SBOM attestations | Verify | **Lost.** They hang off the index, and the index is what is left behind | | Digests match the release manifest | Yes | No, by definition | The bundler warns at the time and annotates the bundle rather than letting this be a silent difference. **If your acceptance process checks attestations, stay on the default.** ## Digest pinning `release-manifest.yaml` in the install kit pins **every image and chart to a digest**, not to a tag. Tags move. Digests do not. That is what makes a release reproducible: the bundler resolves nothing at build time, it fetches exactly the digests the manifest names, and it fails if one is missing. ```bash yq '.spec.images[] | [.image, .imageTag, .digest] | @tsv' $KIT/release-manifest.yaml ``` ## If verification fails | Symptom | Cause | |---|---| | `no matching signatures` | Wrong identity pattern, or the artifact is not ours | | Certificate fetch errors | The Sigstore hosts are blocked. See [Network and connectivity](./../03-network.mdx#egress-list) | | The image verifies, the SBOM does not | Single-architecture bundle. Expected, see above | | `SHA256SUMS` mismatch on the isolated side | The transfer altered the bundle. Re-transfer, do not proceed | **A failed verification is not a step to skip.** The bundler has a flag that drops the check, and a bundle built with it is annotated as unverified. Use it only if cosign genuinely cannot be installed on your staging host, and record that you did. --- # Collect diagnostics Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/troubleshoot/diagnostics # Collect diagnostics Collect this before you contact us. Everything here is something we would otherwise ask for, so sending it means the first reply is an answer rather than a request for more output. There is no single collection command at this release. This page is the list. ## Set up Run the commands from a directory you can archive afterwards. ```bash NS_SYS=gen0sec-system NS_APP=gen0sec OUT=gen0sec-diagnostics mkdir -p $OUT && cd $OUT ``` Cluster and storage: ```bash kubectl version -o yaml > cluster-version.yaml kubectl get nodes -o wide > nodes.txt kubectl describe nodes > nodes-describe.txt kubectl get storageclass -o yaml > storageclasses.yaml kubectl get pvc -A -o wide > pvc.txt ``` Releases and workloads: ```bash helm list -A > helm-releases.txt for ns in $NS_SYS $NS_APP; do kubectl -n $ns get all -o wide > workloads-$ns.txt kubectl -n $ns get events --sort-by=.lastTimestamp > events-$ns.txt kubectl -n $ns get pods -o yaml > pods-$ns.yaml done ``` Logs from anything not ready: ```bash for ns in $NS_SYS $NS_APP; do for p in $(kubectl -n $ns get pods -o json \ | jq -r '.items[] | select(any(.status.containerStatuses[]?; .ready == false)) | .metadata.name'); do kubectl -n $ns logs $p --all-containers --tail=2000 > logs-$ns-$p.txt 2>&1 kubectl -n $ns logs $p --all-containers --previous --tail=2000 > logs-$ns-$p-previous.txt 2>&1 done done ``` Operators and custom resources: ```bash kubectl -n $NS_SYS logs deploy/g0s-infra-postgres-operator --tail=2000 > logs-postgres-operator.txt kubectl -n $NS_SYS logs deploy/strimzi-cluster-operator --tail=2000 > logs-kafka-operator.txt kubectl -n $NS_SYS logs deploy/g0s-infra-synapse-operator --tail=2000 > logs-ingress-operator.txt kubectl -n $NS_SYS get postgresql,kafka,kafkanodepool,kafkatopic -o yaml > custom-resources.yaml ``` The migration job, which is the most common single cause: ```bash kubectl -n $NS_APP logs job/g0s-db-migrate --tail=2000 > logs-db-migrate.txt 2>&1 ``` Images actually in use, including init containers: ```bash for ns in $NS_SYS $NS_APP; do kubectl -n $ns get pods -o yaml | grep -E '^\s+image:' | sort -u > images-$ns.txt done ``` Routing and the relay: ```bash kubectl -n $NS_APP get ingress -o yaml > ingress.yaml kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' > relay-healthz.json 2>&1 ``` Then pack it: ```bash cd .. && tar -czf gen0sec-diagnostics-$(date +%Y%m%d-%H%M%S).tar.gz gen0sec-diagnostics ``` :::danger[Read the archive before you send it] It is plain text and it is yours. Nothing here redacts anything, so check it, and remove any sensitive data it might have, particularly if you added `helm get values`, which prints whatever you set. None of the commands above read secret contents, database rows or object store objects. Logs can contain request paths and key prefixes. ::: :::note[There should be one command for this] A single `collect-diagnostics.sh` in the install kit would replace this page with two lines, and the list above is the specification for it. A single diagnostics tool can accidentally collect data that's sensitive and may fail to redact that information, so that's why we ask you to assemble your own report. Once we're 100% sure we can provide a diagnostics tool that doesn't leak any information of your sensitive data, we'll do so. ::: ## What to tell us alongside it Four lines, and they save a round trip: 1. Which install method you used. 2. The release version. 3. The last command you ran, verbatim. 4. What you expected to happen. --- # Known limitations in this release Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/troubleshoot/known-limitations # Known limitations If you hit something that is not on this page, we'd like to hear about it! ## No zero-egress deployment **What it means.** Your cluster must reach `api.gen0sec.com` on port 443. That connection carries threat intelligence, GeoIP, ML models and IDS rules, which are licensed upstream data that is in no artifact we ship. **What you can remove.** All dependency on Gen0Sec-hosted registries. The [offline bundle method](./../04-prepare-artifacts.mdx) mirrors every image and chart into your own registry, so your cluster pulls nothing from us. **What you cannot remove.** The runtime connection. There is no supported configuration with zero cluster egress. **If your policy forbids all egress**, please contact us. See [Network and connectivity](./../03-network.mdx). ## Our registry challenges pulls that send no user agent **By design on our side, and it can decide which install method you use.** **Symptom.** Every image pull from `registry.gen0sec.com` returns **403 with a CAPTCHA challenge**, and pods sit in `ImagePullBackOff`. A manual `docker pull` or `skopeo copy` from the same node **succeeds**. That discrepancy is the whole diagnosis. If the CLI works and the kubelet does not, this is it. **Cause.** On some Kubernetes distributions, the component that services the kubelet's pull sets no `User-Agent` header. Go's `net/http` then sends its own default, `Go-http-client/1.1`, and our edge treats that as a scanner signature and issues a challenge. A kubelet cannot solve a CAPTCHA, so the pull never succeeds and never will. It is not your credentials, not the pull secret, not TLS, and not an allowlist. ### What passes and what does not Measured against our registry: | User-Agent | Result | |---|---| | `containerd/1.7.0` | 200 | | `containerd/v2.1.4` | 200 | | `docker/29.4.0` | 200 | | `curl/8.7.1` | 200 | | `skopeo/1.24.0` | 200 | | `Go-http-client/1.1` | **403, challenged** | | `Go-http-client/2.0` | **403, challenged** | **containerd's own user agent is fine.** Any real one is. The failure is specifically the absence of a user agent, which Go turns into `Go-http-client`. ### Confirming it The blocked response identifies what it matched: ``` "user_agent": "Go-http-client/1.1", "ja4_fingerprint": null, "captcha_validated": false ``` Compare a manual pull against what the kubelet gets: ```bash # on the node: succeeds, because the CLI sends its own user agent skopeo inspect docker://registry.gen0sec.com/gen0sec/auth-api:0.1.0 ``` ```bash # what the kubelet sees kubectl -n gen0sec describe pod | grep -A3 Failed ``` Success from the CLI plus 403 from the kubelet confirms it. ### Two ways forward **Option 1: use the offline bundle method.** Nothing to configure on your nodes, and your own registry does not inspect user agents. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). **Option 2: make the pull path send a user agent**, on every node. :::note[Settle this before you choose an install method] If you cannot change container runtime configuration on your nodes, whether by policy or because your distribution manages it, use the offline bundle method. This constraint forces that choice, and it is better made before you start than at the first failed pull. ::: ## `--set entrypointHost` has no effect Every route publishes on the `gen0sec.internal` placeholder whatever address you pass, and nothing reports it. Set `entrypointHost` inside `$KIT/examples/single-entrypoint/ingress-values.yaml` instead. ## `--set externalScheme` has no effect Set `services.ui.ingress.externalScheme` instead. ## The relay cannot use an explicit egress proxy **Symptom.** You set `HTTPS_PROXY` on the relay. Nothing changes, and nothing errors. **Cause.** Every HTTP client in the platform builds its own transport without proxy support, so `HTTP_PROXY` and `HTTPS_PROXY` are ignored inside the cluster. This is by design. **What works instead.** | Option | Notes | |---|---| | Allow `api.gen0sec.com:443` directly | One firewall exception that bypasses the proxy | | Redirect transparently at the network layer | The client does not need to be proxy-aware | A TLS-intercepting proxy also cannot work: the charts expose no way to mount a CA bundle into the platform containers, and the relay rejects the intercepted certificate. ## The object store bucket is not created for you **Symptom.** Services start and then fail to read or write the object store. `download-api` answers `500 NoSuchBucket`. **Cause.** Nothing in the charts creates the bucket. **What to do.** [Install](./../05-install.mdx#5-create-the-object-store-bucket) step 5 creates it. It is a first-class step, not an optional one. ## Three services report readiness from a weaker probe `download-api`, `ids-rules-api` and `service-graph-api` serve a richer `/status/healthz` and, in doing so, never register `/status/readyz`. Requesting it returns 404, so a readiness probe pointed at it fails forever while liveness stays healthy, and the pod sits at `0/1` with zero restarts. **What that means for you.** Do not repoint those services at `/status/readyz`. It will fail. ## Helm never upgrades CRDs Not our defect, but it will bite you on the first upgrade that contains CRDs, so it is here. Helm installs a chart's CRDs on first install and ignores them from then on. An upgrade that ships a new CRD schema fails with a server-side-apply error, or silently rejects new fields. [Upgrade](./../operate/upgrade.mdx) applies them by hand as step 1. Do not skip it. ## Kafka volumes survive an uninstall `deleteClaim: false` is deliberate, so `helm uninstall` leaves the Kafka volumes behind. Reinstalling against them produces `Invalid cluster.id ... Expected X, but read Y`, because the new cluster generates a new identity. Delete the volume claims first, or restore the original cluster. See [Kafka will not start](./symptoms.mdx#kafka-will-not-start). ## Migration `force` means the opposite of what people expect `migrate force ` marks that version **applied** and continues with the next one. To re-run a migration that failed, force the version **before** it. Migrations are not idempotent. Forcing to the wrong version re-runs DDL that already exists and fails with `already exists`. --- # Symptom index Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/troubleshoot/symptoms # Symptom index Everything here was reproduced on a real cluster. ## Images ### ImagePullBackOff Three different causes, and the first thing to establish is which. ```bash kubectl -n get pod -o jsonpath='{.spec.containers[*].image}' kubectl -n describe pod | tail -20 ``` | The image reference is | Cause | Fix | |---|---|---| | Correct, and the error mentions authorization | The `gen0sec-registry` secret is missing from that namespace | It is needed in **both** namespaces. See [Install step 2](./../05-install.mdx#the-image-pull-secret) | | Pointing at the wrong registry | A values file was not passed with `-f` | Re-run with `-f $B/values/-values-onprem.yaml` | | A path that does not exist in your registry | Mirror path mismatch, on the offline bundle method | See the next entry | ### 403 with a CAPTCHA challenge from our registry All images, not one, and only on the registry install method. **The tell:** a manual `docker pull` or `skopeo copy` from the same node succeeds while the kubelet's pull returns 403. ```bash # succeeds: the CLI sends its own user agent skopeo inspect docker://registry.gen0sec.com/gen0sec/auth-api:0.1.0 ``` ```bash # 403: what the kubelet gets kubectl -n gen0sec describe pod | grep -A3 Failed ``` On some distributions the component servicing the kubelet's pull sets no `User-Agent`, so Go sends `Go-http-client/1.1`, which our edge challenges as a scanner signature. A kubelet cannot solve a CAPTCHA, so it never recovers. Not credentials, not TLS, not the pull secret. Fix: set a user agent in the container runtime on every node, or use the offline bundle method. Both in [Known limitations](./known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ### ImagePullBackOff on a pod you did not configure Operator-launched pods (Postgres, the connection pooler, Kafka brokers, the entity operator) get their image reference from the **operator**, not from the chart. If one of those is stuck, the mirror path is wrong, not the pull secret. Confirm the path exists in your registry: ```bash curl -su : https://$REGISTRY/v2/_catalog?n=200 | tr ',' '\n' ``` Two paths look surprising and are not negotiable: | Image | Must be mirrored to | |---|---| | Kafka broker | `/gen0sec/strimzi/kafka:1.0.0-kafka-4.1.1` | | Entity operator | `/gen0sec/strimzi/operator:1.0.0` | Everything is under `gen0sec/`. Kafka additionally keeps the upstream `strimzi/` path inside it, because the operator's `defaultImageRegistry` replaces only the registry host and keeps the repository path. A Kafka pod asking for `/strimzi/kafka` with no `gen0sec/` means that setting lost the namespace. ### 401 Unauthorized pulling into a Postgres or pooler pod The Postgres operator does not copy `imagePullSecrets` onto the database pods it creates. They inherit them from their ServiceAccount, which `values-onprem.yaml` configures through `configKubernetes.pod_service_account_definition`. If you overrode `configKubernetes` wholesale, you dropped it. Merge instead of replacing. ### manifest unknown A chart is asking for a tag that was never published. Check what it rendered: ```bash helm get manifest -n | grep -E '^\s+image:' | sort -u ``` Then compare against [Component versions](./../configure/components.mdx). ## Postgres ### Postgres will not start `postgresql/core` sits in `CreateFailed`. ```bash kubectl -n gen0sec-system logs deploy/g0s-infra-postgres-operator | grep -i 'could not create' ``` | Log line contains | Cause | Fix | |---|---|---| | `endpoints is forbidden` | The operator's own ClusterRole omits `endpoints`, but it creates a master Endpoints object per cluster | The infra chart adds the missing rule. Confirm: `kubectl get clusterrole g0s-infra-postgres-operator-endpoints` | | `Unsupported value: "18"` | The CRDs in the cluster are older than the operator chart | Helm never upgrades CRDs. Apply them first. See [Upgrade](./../operate/upgrade.mdx) | | Nothing, and no volume is bound | No default StorageClass, or not enough capacity | See [Sizing](./../02-requirements.mdx#sizing-a-production-install) | After fixing the cause the operator does not retry promptly. Force it: ```bash kubectl -n gen0sec-system rollout restart deploy/g0s-infra-postgres-operator ``` ### Migration failures The migration job runs as a Helm hook. On failure Helm aborts and deploys nothing, so the cluster is in a safe state. ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` | Log line contains | Cause | Fix | |---|---|---| | `relation "..." does not exist` | The chain is running against a database that is not at the version it expects. Usually a forced version, or a `db-migrate` image older than the chart | Read the version table in the next entry before you force anything, then [collect diagnostics](./diagnostics.mdx) | | `role "..." does not exist` | A database role is missing. The migrations `GRANT` to roles by name and roll back if one is absent | The data chart creates them. Confirm: `kubectl -n gen0sec-system get secret \| grep credentials` | | `already exists` | A migration was forced to the wrong version and re-ran DDL | See the next entry | ### The schema is marked dirty The migration tool marks the schema dirty when a migration fails halfway, and refuses to continue until you say what happened. ```bash kubectl -n gen0sec-system exec -it core-0 -c postgres -- \ psql -U postgres -d ax_prod -c 'select * from schema_migrations' ``` `force ` means "**this version is applied**, continue with the next one". So to re-run a failed migration, force the version **before** it. Migrations are not idempotent. ## Kafka ### Kafka will not start Broker crash loops with `Invalid cluster.id ... Expected X, but read Y`. The volume survived a previous `helm uninstall`, which is deliberate, and the recreated cluster generated a new identity. Either restore the original cluster or wipe the volumes: ```bash kubectl -n gen0sec-system delete kafkanodepool kafka kubectl -n gen0sec-system delete pvc -l strimzi.io/cluster=core helm upgrade g0s-data ... ``` **This deletes the message data on those volumes.** ### No KafkaTopic resources exist although the chart rendered them Helm can skip re-applying objects that were deleted out of band. Re-apply the stored manifest: ```bash helm get manifest g0s-data -n gen0sec-system | kubectl apply -f - ``` ## Object store and cache ### Services cannot reach the object store The object store Service is `rustfs-svc`, with the suffix. `S3_ENDPOINT` and `ARXIGNIS_DATA_URL` must match it. ```bash kubectl -n gen0sec-system get svc | grep rustfs kubectl -n gen0sec get deploy g0s-download-api \ -o jsonpath='{.spec.template.spec.containers[0].env}' | tr ',' '\n' | grep -E 'S3_ENDPOINT|DATA_URL' ``` See [Install step 7](./../05-install.mdx#7-install-the-platform). ### 500 NoSuchBucket The bucket does not exist. [Install step 5](./../05-install.mdx#5-create-the-object-store-bucket) creates it. Nothing in the charts does. ### Everything works but reads are slow The shared cache is never fatal. Missing or unreachable, each service logs a warning once and runs on its in-process cache only. Confirm from the service side, not the pod list: ```bash kubectl -n gen0sec logs deploy/g0s-auth-api | grep -i "L2 cache" ``` `L2 cache (Redis) connected to server` means the cache is in use. `falling back to L1-only cache` means `REDIS_URL` is wrong or unreachable. ```bash kubectl -n gen0sec-system get svc dragonfly ``` The Service is deliberately not prefixed with the release name, because `REDIS_URL` names it directly. ### The cache pod crash loops ```bash kubectl -n gen0sec-system logs deploy/dragonfly --previous ``` | Log shows | Cause | Fix | |---|---|---| | `io_uring` errors | Blocked by the host kernel or a seccomp profile | Add `--force_epoll` to `dragonfly.extraArgs` in the infra values | | Read-only filesystem errors | The pod runs with a read-only root, which is why the chart disables snapshotting | Remove any flag you added that re-enables it | ### The cache is being evicted or OOM-killed `maxMemory` is the eviction threshold and the memory limit is the kill threshold. Keep the first comfortably below the second, or the kubelet kills the pod before it gets a chance to evict. ## Install and workstation ### The bundler fails with a yq syntax error You have the wrong `yq`. Two unrelated tools share the name. ```bash yq --version ``` The output must contain `mikefarah`. If not, install the Go version and point the bundler at it with `MIKEFARAH_YQ_PATH`. See [Requirements](./../02-requirements.mdx#check-your-yq). ### zsh: no matches found, before helm runs `zsh` expands `[0]` in a `--set` argument as a glob. Single-quote the whole argument: ```bash --set 'services.ui.ingress.hosts[0].host=example.internal' ``` ### The namespace hangs in Terminating Strimzi puts a finalizer on all 23 KafkaTopics, and once the operator is gone nothing clears it. ```bash kubectl get kafkatopics.kafka.strimzi.io -n gen0sec-system -o name \ | xargs -r -n1 kubectl patch -n gen0sec-system --type=merge \ -p '{"metadata":{"finalizers":null}}' ``` `uninstall.sh` does this for you, in the right order. See [Uninstall](./../operate/uninstall.mdx). ### A pod reports Running but nothing works A pod in `CrashLoopBackOff` still reports `status.phase: Running`. Read the `READY` column, never a field selector on phase. ```bash kubectl -n gen0sec get pods ``` ### Agents get 401 on every relayed artifact Your Gen0Sec API key is wrong, or the secret holds a trailing newline. ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` ``` artifact relayed from upstream path=/v1/ids-rules/version status=401 ``` Recreate the secret with `printf`, not `echo`. `--from-file` stores the bytes exactly, so a newline becomes part of the key. See [Data relay](./../configure/relay.mdx#getting-the-key-right). A missing secret does not produce this: the relay pod fails to start instead. See the next entry. ### CreateContainerConfigError on the relay pod The pod references a secret that does not exist. Almost always `gen0sec-download-proxy`: `downloadProxy.apiKeySecret.name` is set but the secret was never created. ```bash kubectl -n gen0sec describe pod -l app.kubernetes.io/name=download-api | tail -20 kubectl -n gen0sec get secret gen0sec-download-proxy ``` Create it. See [Install step 2](./../05-install.mdx#your-gen0sec-api-key). This failure is the visible one. The dangerous case is the opposite: leaving `apiKeySecret.name` unset renders no secret reference at all, the pod starts, and the failure is the silent 401 above. ### The relay reports 502 on every relayed route The upstream is unreachable. A 502 is deliberate rather than a 404 or 500: a polling agent reads it as "ask again later" and keeps the artifact it already holds. ```bash kubectl -n gen0sec exec deploy/g0s-download-api -- curl -fsS localhost:9999/status/healthz ``` Check that `api.gen0sec.com:443` is allowed. See [Ports and egress](./../security/egress.mdx). If you route egress through an explicit proxy, that is the cause: proxies are not honoured in-cluster. See [Known limitations](./known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy). ## Routing ### An agent gets HTML, or a redirect, from an API The request reached the dashboard's catch-all instead of the intended service. Routing is by longest matching prefix and the dashboard owns `/`, so any path missing from the route table lands there. ```bash kubectl -n gen0sec get ingress -o wide ``` Compare against [Publish the endpoints](./../06-publish.mdx#the-route-table). The failures this causes are quiet by design of HTTP, not by ours: | Missing route | What the agent sees | |---|---| | `/v1/authcheck` | A redirect and HTML. The agent cannot distinguish a bad key from a routing mistake and continues unverified | | `/v1/agents/events` | `SSE connect returned HTTP 404`, then silent fallback to polling, which looks like a slow platform | | `/v1/ban-ledger` | A redirect the agent reads as success. The contribution is lost silently | ### kubectl get ingress shows an empty HOSTS column The ingress controller ignores host-less rules: the object is created, nothing is logged, and no traffic is served. Set `entrypointHost`. See [Publish the endpoints](./../06-publish.mdx). ## Air gap and mirroring ### Confirming the cluster really is isolated From a pod, not from a node. ```bash kubectl -n gen0sec run netcheck --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/busybox:1.36.1-glibc -- wget -T5 -qO- https://ghcr.io/ ; echo "exit=$?" ``` A non-zero exit is the expected result. If it succeeds, the cluster still has egress and an install that appears to work may be pulling from the internet. ### An image is missing for one architecture You built the bundle with `--arch`. A single-architecture bundle fails when the kubelet tries to pull on a node of the other architecture, which on an isolated cluster is after the transfer. Rebuild without `--arch`, or with the architecture your nodes actually run. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). ## Still stuck [Collect diagnostics](./diagnostics.mdx), then send us the archive. --- # Verify the install Source: https://docs.gen0sec.com/docs/cerebellum/0.1.0/verify # Verify the install Running pods does not mean a working platform. A worker with no Service, a misconfigured credential, or a missing route all leave every pod `Ready` while no data moves. This page defines what "installed successfully" means. ## Before you start - [ ] [First sign-in](./07-first-signin.mdx) complete - [ ] An agent API key was created ## 1. Every pod is ready ```bash kubectl -n gen0sec-system get pods kubectl -n gen0sec get pods ``` Every pod reports `Running` and `n/n` in the `READY` column. On a correct install there are no exceptions: nothing is expected to sit unready. :::note[Check readiness, not phase] A pod in `CrashLoopBackOff` still reports `status.phase: Running`. A `--field-selector` on phase reports a broken install as clean, which is why the command above reads the `READY` column instead. ::: ## 2. The migration job is gone ```bash kubectl -n gen0sec get job g0s-db-migrate ``` The output is similar to this: ``` Error from server (NotFound): jobs.batch "g0s-db-migrate" not found ``` `NotFound` is the pass. Helm deletes the job the moment it succeeds. If it is still there, it failed: `kubectl -n gen0sec logs job/g0s-db-migrate`. ## 3. The relay is connected ```bash kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' ``` The output is similar to this: ```json { "status": "ready", "mode": "proxy", "upstream": "https://api.gen0sec.com", "external_relays": ["geoip", "ids-rules"] } ``` `"mode": "proxy"` confirms the relay came up. If the upstream is unreachable, relayed routes answer 502, which a polling agent reads as "ask again later" rather than "the artifact is gone". Then confirm it is actually authorized upstream, which the mode alone does not tell you: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means your Gen0Sec API key is wrong. ## 4. The platform serves an agent This is the check that matters. It makes the same requests an agent makes, from outside the cluster, over your published address. ```bash export GEN0SEC_BASE_URL=https://$ENTRYPOINT_HOST export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) $KIT/scripts/verify-deployment.sh ``` It checks four things: 1. The API refuses unauthenticated callers. 2. Configuration is served. 3. Threat intelligence and IDS rule bundles are published. 4. A downloaded bundle matches the SHA-256 its manifest advertises. It is read-only and safe to run against production. It exits non-zero and names the failing check. **Run it after every install and every upgrade.** It is the cheapest way to catch a deployment that looks healthy and serves nothing. ## Offline bundle only: confirm nothing pulls from outside ```bash kubectl -n gen0sec-system get pods -o yaml | grep -E '^\s+image:' | sort -u kubectl -n gen0sec get pods -o yaml | grep -E '^\s+image:' | sort -u ``` Every image listed is under your registry. Anything else means a values file was not passed with `-f`. :::note[Why both namespaces, and why read the YAML] The application runs in `gen0sec`; only the operators, object store and cache run in `gen0sec-system`. Checking one namespace covers well under half the images. Reading the YAML rather than a container template also catches init containers, which is where the object store's init image is. ::: ## Definition of installed successfully All five are true: 1. Every pod in both namespaces reports `Running` and `n/n` ready. 2. `kubectl -n gen0sec get job g0s-db-migrate` returns `NotFound` or appears as `Completed`. 3. `download-api` reports `"mode": "proxy"` with your upstream, and its log shows no `status=401` or `status=402` on relayed paths. 4. `verify-deployment.sh` exits zero. 5. On the offline bundle method, every image reference is under your registry. If all five hold, the install is done. If any fail, start at the [symptom index](./troubleshoot/symptoms.mdx). ## You may be interested in... - [Upgrade](./operate/upgrade.mdx), so you know how to move before you need to - [Roll back](./operate/rollback.mdx), and where the data boundary is - [Known limitations in this release](./troubleshoot/known-limitations.mdx) --- # Overview Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/ # Overview ## What is Cerebellum? Cerebellum is the enterprise version of the Gen0Sec platform that you can host in your own Kubernetes cluster. ## Charts | Installation order | Chart | Namespace | What it installs | |---|---|---|---| | 1. | `gen0sec-infra` | `gen0sec-system` | Three operators (Postgres, Kafka, ingress), the object store, and two caches: one shared by the application, one for the ingress data plane | | 2. | `gen0sec-data` | `gen0sec-system` | The Postgres cluster, the Kafka cluster, topics and database roles | | 3. | `gen0sec-platform` | `gen0sec` | Schema migrations, then the application services and the dashboard | The installation order is mandatory, as each chart creates resources the next one depends on. ## Installation methods ### Installation from registry (Recommended) Choose it when, - your cluster can access our registry, - you don't mind depending on an external registry ### Install via creating an offline bundle Choose it when, - your cluster don't have access to our registry, - you'd rather host mirrored images yourself, - you're facing issues pulling images from our registry :::note[Same content] Worry not, the offline bundle builds the same release and installs the same charts. ::: If you are unsure, go with **Install from registry**. You can always switch later. :::info[One thing that can decide this for you] Our registry challenges pulls that arrive with no `User-Agent`, and on some Kubernetes distributions the kubelet's pull path sends none. That is deliberate policy on our side, and the result is a 403 the kubelet cannot recover from. If you cannot change container runtime configuration on your nodes, use **Create an offline bundle**. Your own registry does not inspect user agents. See [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: You will need to decide the installation method in the [Prepare the artifacts](./04-prepare-artifacts.mdx) step. ## Connectivity :::warning[What 'offline' means] The 'offline' part in the offline bundler is used to describe the installation method, not the resulting product. ::: It's important to note that even by creating an offline bundle, doesn't mean you can operate the platform in an airgapped cluster that is entirely isolated from reaching the public internet. Cerebellum needs exactly one outbound destination from your cluster: - `api.gen0sec.com` on port `443`. That single connection carries **threat intelligence, GeoIP databases, ML models and IDS rules**. All four are licensed by their upstream providers and are not included in any of the artifacts we ship. Every agent in your fleet reaches it through one in-cluster service, so it is one destination to allow and one to audit. Nothing else in either namespace has a route out. Read [Network and connectivity](./03-network.mdx) before you start. It is the page to send to whoever runs your firewall. ## How long it takes Set aside **half a day for a first install**, and about an hour once you have done one. The install is 26 numbered steps across five pages, and most of the elapsed time is spent waiting on image pulls and on two clusters becoming ready, not on typing. The waits are gated explicitly, so the ceiling is knowable. These are the timeouts the commands themselves carry — the upper bound before a step is treated as failed, not a prediction: | Stage | You are waiting on | Gate | |---|---|---| | [Prepare the artifacts](./04-prepare-artifacts.mdx) | Downloads and signature verification. From the registry this is minutes; building an offline bundle copies every image and is the longest single stage | None | | [Install](./05-install.mdx) infra | Three operators and the object store | None | | [Install](./05-install.mdx) data | Kafka ready, then Postgres `Running` | `15m` each | | [Install](./05-install.mdx) platform | Schema migrations, then every pod ready | `20m` + `10m` | | [Publish](./06-publish.mdx) | The ingress rollout | `20m` | | [First sign-in](./07-first-signin.mdx) | You: MFA enrolment, a forced password change, the onboarding wizard | None | | [Verify](./08-verify.mdx) | Four checks you run | None | What actually decides your number is how fast your nodes pull images. Everything else is bounded by the table above. :::note[The Quickstart is not a shortcut] It is the same procedure with one extra values file per chart, on one node. It is quicker because a single-node cluster pulls and schedules less — not because it skips steps. ::: ## Before you install 1. [Requirements](./02-requirements.mdx). Cluster, tools, and your access token. 2. [Network and connectivity](./03-network.mdx). What to allow, and why. 3. [Prepare the artifacts](./04-prepare-artifacts.mdx). Prepare for the installation. If you want a working instance to look at before you plan a real install, use the [Quickstart](./quickstart.mdx). It builds a throwaway single-node install and tears it down again. --- # Component versions Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/configure/components # Component versions :::note[Read this from your release, not from this page] Every release pins its own versions, and the authoritative list is in the install kit you downloaded. ```bash yq '.spec.images[] | [.image, .imageTag, .digest] | @tsv' $KIT/release-manifest.yaml yq '.spec.charts[] | [.chart, .version, .digest] | @tsv' $KIT/release-manifest.yaml ``` This page describes what those versions mean and which of them have surprising requirements. ::: {/* This page should be generated from release/components.yaml and release-manifest.yaml in CI, per the structure proposal section 7. Hand-maintained, it drifts within one release and is then worse than nothing, because operators trust it. */} ## Third-party components | Component | Version | Notes | |---|---|---| | Postgres operator | 2.0.1 | Zalando. **1.15.x cannot be used**: its schema rejects PostgreSQL 18 | | PostgreSQL | 18, with extensions | Includes AGE, TimescaleDB, `pg_cron`, `pg_net`, `pglogical`, `pgsodium` | | Connection pooler | 2.0.1 | Operator-launched. Services connect through it, never directly | | Logical backup | 2.0.1 | Only used when logical backups are enabled | | Kafka operator | 1.0.0 | Strimzi. v1 CRD API only | | Apache Kafka | 4.1.1 | **KRaft, no ZooKeeper.** Operator-launched | | Object store | Chart 0.10.0 | S3-compatible. Its Service has a `-svc` suffix | | Shared cache | 1.38.1 | Redis protocol. Application services degrade to in-process caching without it | | Ingress cache | Valkey | Deployed by the ingress chart and used only by it. Kept separate on purpose, so a shared-cache outage cannot take the ingress data plane with it. One replica: a restart costs a cold start, not data | | Ingress operator | Commit-tagged | `ingressClassName: synapse`. Built in a sibling repository, so the tag is a commit hash rather than a version | | Platform services | The release version | Resolved from the registry host, namespace and image tag | ## Mirror paths, for the offline bundle method Every mirrored image lands under `gen0sec/`. Two paths look wrong and are not negotiable. | Image | Must be at | |---|---| | Kafka broker | `/gen0sec/strimzi/kafka:1.0.0-kafka-4.1.1` | | Kafka entity operator | `/gen0sec/strimzi/operator:1.0.0` | **Why Kafka keeps an extra path segment.** The Kafka operator's image-registry setting replaces only the registry *host* and preserves the repository path. So the chart points it at `/gen0sec` rather than ``, and the upstream `strimzi/` segment survives inside our namespace. :::note[This exact thing shipped wrong once] With the namespace omitted, the chart asked for `/strimzi/operator` while the release mirrored `/gen0sec/strimzi/operator`. One path segment apart, and the entire Kafka stack failed to pull on a cluster with no fallback. Every chart is now rendered in CI and the build fails on any image reference not backed by the release manifest, so the two cannot drift again. That gate is also why the table above is trustworthy: it catches a chart asking for a tag nobody publishes, which is how the ingress operator once requested a version that was never released. ::: `bundler.sh mirror` reproduces these paths exactly. You do not compute them. ## Not everything a chart renders is mirrored The vendored Kafka chart hardcodes an image map covering four Kafka versions plus Connect and Bridge build tooling. This edition ships one Kafka version and no Connect clusters, so the rest are deliberately absent from the bundle. That is roughly 1.5 GB of images nothing ever pulls. **If you enable Kafka Exporter, Cruise Control, Kafka Bridge or Connect builds, mirror their images first.** The release does not carry them. ## Kubernetes | | | |---|---| | Supported | 1.23 or later | See [Requirements](./../02-requirements.mdx#kubernetes-versions). ## Support policy | | | |---|---| | Supported releases | The latest two minor trains | | Upgrade path | Sequential minors. **Skipping a minor is not supported.** Patches in place | | Upgrade order | Always infra, then data, then platform | | CRDs | Helm never upgrades them. Apply them before the chart. See [Upgrade](./../operate/upgrade.mdx) | | Downgrade | Not supported across a minor that ran a schema migration. Within a train, use rollback. See [Roll back](./../operate/rollback.mdx) | | Kubernetes upgrades | Independent of ours. Stay within the supported range | ## Verifying what you are actually running The manifest says what a release pins. This says what your cluster pulled: ```bash kubectl -n gen0sec-system get pods -o yaml | grep -E '^\s+image:' | sort -u kubectl -n gen0sec get pods -o yaml | grep -E '^\s+image:' | sort -u ``` Reading the YAML rather than a container template also catches init containers, which is where the object store's init image is. On the offline bundle method, every reference should be under your own registry. Anything else means a values file was not passed. --- # Sign-in and identity Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/configure/identity # Sign-in and identity ## What is available | Method | Status | Where credentials live | |---|---|---| | Email and password | **Available** | Your Postgres | | Multi-factor authentication | **Available, and required during onboarding** | Your Postgres | | Microsoft Entra ID | **In development, not supported at the moment ** | | **Nothing outside your cluster is involved in sign-in.** Accounts, password hashes and sessions are all in your own database. :::note[Single sign-on is not available yet] Microsoft Entra ID is in development. No generic OIDC provider, no SAML and no LDAP integration exists either. If single sign-on is a requirement for you, tell us which provider, because that shapes what ships first. ::: ## The first account Registration is invite-only and invites need an existing member, so a fresh install seeds one account with a **published password**. Signing in, completing onboarding and changing that password is a required install step, not an optional one. See [First sign-in](./../07-first-signin.mdx). ## The published address decides whether sign-in works The dashboard derives its sign-in base URL from the host you publish it on. If that does not match the address people actually type in the browser, every sign-in is rejected as an invalid origin. Two rules: 1. **Use the address people type.** Not an internal service name, not a load balancer's own hostname. 2. **If TLS terminates upstream of the ingress**, set the scheme explicitly, because the chart cannot infer it and guesses `http` while the browser is on `https`. ```bash --set externalScheme=https ``` Leave that unset when TLS terminates at the ingress. See [Publish the endpoints](./../06-publish.mdx#if-tls-terminates-upstream). ## Sessions The signing secret lives in `g0s-db-ui`, created with all the other database secrets in [Install step 6](./../05-install.mdx#6-create-the-application-secrets). | Action | Effect on sessions | |---|---| | Re-running `make-db-secrets.sh` | None. The existing signing secret is preserved deliberately | | Rotating the signing secret | Every session ends and every outstanding invite link stops working | | Upgrading | None | ## Turning the dashboard off entirely If your operators use the API only: ```bash --set services.ui.enabled=false ``` The agent-facing APIs are unaffected. Note that agent API keys are issued through the dashboard, so turn it off only once you have the keys you need. ## If it fails | Symptom | Cause | |---|---| | Sign-in rejected as an invalid origin | The published host does not match the address typed, or the scheme is wrong. See above | | Sessions drop after a re-run of the secret script | The signing secret changed. It should not; check whether the secret was deleted first | | Every sign-in fails after an upgrade | Check that `externalScheme` and the host survived the upgrade. `helm get values g0s -n gen0sec` | --- # Reference configurations Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/configure/reference-configs # Reference configurations Two named configurations rather than a blank values file. Each ships in the install kit, and each is reproduced in full on this page, so you can read it before you download anything. | Configuration | For | Redundancy | Storage | |---|---|---|---| | [Single node](#single-node) | Evaluation, training, upgrade rehearsal | **None** | Smallest | | [Production](#production) | Anything you depend on | Full | Smallest production grade | ## Single node **Not for production.** Everything at one replica on one node. Losing the node loses the data. Ships as three overlays in the kit: ``` $KIT/examples/single-node/infra-values.yaml $KIT/examples/single-node/data-values.yaml $KIT/examples/single-node/platform-values.yaml ``` Layer each one **after** the production values file, so it wins. Full procedure: [Quickstart](./../quickstart.mdx). | | Production | Single node | |---|---|---| | Object store | 3 replicas, erasure coded, StatefulSet | 1 replica, standalone, **Deployment** | | Postgres | 3 instances, both poolers on | 1 instance, replica pooler off | | Kafka | 3 brokers, replication factor 3 | 1 broker, every factor 1 | | Services | 2 or 3 replicas | 1 each | | StorageClass | Yours, set explicitly | The cluster's default (empty in the overlay) | ### What the overlays set Verbatim from the v0.1.1 kit, so you can read them without unpacking it first.
infra-values.yaml — object store and cache ```yaml title="$KIT/examples/single-node/infra-values.yaml" # ============================================================================ # gen0sec-infra — SINGLE-NODE EVALUATION overlay # ============================================================================ # Layered ON TOP of values-onprem.yaml (pass both, in this order): # helm install g0s-infra charts/gen0sec-infra -n gen0sec-system \ # -f values/gen0sec-infra-values-onprem.yaml \ # -f examples/single-node/infra-values.yaml # # Shrinks the footprint to fit one worker node. NOT for production: the object # store loses erasure coding and every component becomes a single point of # failure. Use it for evaluation, training, and dry-running an upgrade. # ============================================================================ rustfs: # Distributed mode needs >= 2 nodes for erasure coding; on one node the pods # would schedule but the cluster is not fault-tolerant anyway, so run the # single-process server and be explicit about it. mode: standalone: enabled: true distributed: enabled: false replicaCount: 1 storageclass: # Empty = the cluster's default StorageClass, matching values-onprem.yaml. On # k3s that resolves to local-path, which this overlay used to hardcode — but a # literal `local-path` exists on no other distribution, so the PVCs bound # nowhere and only surfaced as pods Pending past the rollout timeout. name: "" dataStorageSize: 20Gi logStorageSize: 5Gi dragonfly: # Cache only — shrink it so the node's RAM goes to Postgres and Kafka. Keep # maxMemory under resources.limits.memory so it evicts instead of being killed. maxMemory: 256mb proactorThreads: 1 resources: requests: { cpu: 100m, memory: 128Mi } limits: { cpu: 500m, memory: 512Mi } ```
data-values.yaml — Postgres and Kafka ```yaml title="$KIT/examples/single-node/data-values.yaml" # ============================================================================ # gen0sec-data — SINGLE-NODE EVALUATION overlay # ============================================================================ # helm install g0s-data charts/gen0sec-data -n gen0sec-system \ # -f values/gen0sec-data-values-onprem.yaml \ # -f examples/single-node/data-values.yaml # # One Postgres instance, one Kafka node, replication factor 1. Any node loss is # data loss — evaluation only. # ============================================================================ postgres: numberOfInstances: 1 # With a single instance there is no replica to pool. enableReplicaConnectionPooler: false volume: size: 20Gi # Empty = the cluster's default StorageClass (see infra-values.yaml). storageClass: "" resources: requests: { cpu: 500m, memory: 1Gi } limits: { cpu: "4", memory: 6Gi } parameters: shared_buffers: 512MB effective_cache_size: 1536MB max_connections: "400" max_worker_processes: "16" max_parallel_workers: "8" timescaledb.max_background_workers: "8" kafka: replicas: 1 storage: size: 20Gi # Empty = the cluster's default StorageClass (see infra-values.yaml). class: "" # min.insync.replicas must stay < replicas, so every factor drops to 1 # together with the node count. config: offsets.topic.replication.factor: 1 transaction.state.log.replication.factor: 1 transaction.state.log.min.isr: 1 default.replication.factor: 1 min.insync.replicas: 1 log.compression.type: zstd topicDefaults: partitions: 1 replicas: 1 ```
platform-values.yaml — one replica per service ```yaml title="$KIT/examples/single-node/platform-values.yaml" # ============================================================================ # gen0sec-platform — SINGLE-NODE EVALUATION overlay # ============================================================================ # helm install g0s charts/gen0sec-platform -n gen0sec \ # -f values/gen0sec-platform-values-onprem.yaml \ # -f examples/single-node/platform-values.yaml \ # --set global.imageTag=$VERSION --timeout 20m # # One replica of everything: no HA, and a rolling update is a short outage per # service. Resource requests are unchanged — they are already small. # ============================================================================ services: auth-api: { replicaCount: 1 } ban-ledger-api: { replicaCount: 1 } config-api: { replicaCount: 1 } data-api: { replicaCount: 1 } download-api: { replicaCount: 1 } feeds-api: { replicaCount: 1 } ids-rules-api: { replicaCount: 1 } rules-validator: { replicaCount: 1 } service-graph-api: { replicaCount: 1 } signal-api: { replicaCount: 1 } telemetry-api: { replicaCount: 1 } threat-api: { replicaCount: 1 } agent-status: { replicaCount: 1 } ui: { replicaCount: 1 } ```
## Production The default. `values-onprem.yaml` in the bundle **is** the production configuration, so there is no overlay to apply: what you install in [Install](./../05-install.mdx) is this. What you set yourself: ```yaml # gen0sec-infra rustfs: storageclass: name: your-block-storage-class # gen0sec-data postgres: volume: storageClass: your-block-storage-class kafka: storage: class: your-block-storage-class ``` For requirements see [Sizing](./../02-requirements.mdx#sizing-a-production-install). ## Extending retention The shipped sizes are deliberately modest, so growing is the normal direction. Which key you grow depends on what you are retaining. See [Sizing storage for your retention](./../02-requirements.mdx#sizing-storage-for-your-retention). eg. ```yaml # gen0sec-data postgres: volume: size: 200Gi storageClass: your-block-storage-class kafka: storage: size: 200Gi class: your-block-storage-class ``` ```yaml # gen0sec-infra rustfs: storageclass: name: your-block-storage-class dataStorageSize: 100Gi ``` :::note[You can also grow later] Volume expansion grows a volume if your StorageClass has `allowVolumeExpansion: true`. Starting at the defaults and expanding when you need to is a reasonable plan, and it is the direction that works: nothing shrinks a volume. ::: ## Publishing endpoints Independent of the three above, and needed by all of them: ``` $KIT/examples/single-entrypoint/ingress-values.yaml ``` One address for the dashboard and every agent-facing API, routed by path prefix. See [Publish the endpoints](./../06-publish.mdx) for the route table and the two values you set.
The header of that file, with both customer inputs and why they exist The rest of the file is the per-service route list. It is reproduced as a table on [Publish the endpoints](./../06-publish.mdx#the-route-table) rather than twice here. ```yaml title="$KIT/examples/single-entrypoint/ingress-values.yaml (head)" # ============================================================================ # gen0sec-platform — SINGLE-ENTRYPOINT INGRESS overlay # ============================================================================ # Publishes the dashboard and the agent-facing APIs through ONE address, served # by the Synapse proxy that gen0sec-infra deploys for the `synapse` IngressClass. # # helm upgrade --install g0s charts/gen0sec-platform -n gen0sec \ # -f values/gen0sec-platform-values-onprem.yaml \ # -f examples/single-entrypoint/ingress-values.yaml # # Layered ON TOP of values-onprem.yaml, which leaves every ingress disabled. # That default is deliberate — turning this on makes the APIs reachable from # wherever the proxy Service is reachable — so this is opt-in, not the default. # # A HOST IS REQUIRED — set `entrypointHost` below, it is aliased into every # rule. Host-less rules do NOT work with this controller even though the chart # renders them: synapse's upstreams.yaml is keyed by host, and the operator # skips any rule with an empty host (controllers/ingress_controller.go, "if # host == \"\" || rule.HTTP == nil { continue }"). Such a rule produces an # Ingress object that its own controller silently ignores, with no event and no # log line. # # The dashboard needs this host to be the address people actually type: # BETTER_AUTH_URL is derived from it (services.ui.urlFromIngress), and sign-in # breaks otherwise. # # ROUTING IS BY PATH PREFIX, because one address serves every backend. The # prefixes below are the real route roots, read off each service's router -- # they are not a naming convention and do not survive being guessed at. # # One overlap to keep in mind, resolved by longest-prefix matching: # /v1/agents/events config-api serves this; data-api owns /v1/agents # If a request lands on the wrong backend, that is the first thing to check. # # /v1/ids-rules goes to download-api, NOT the in-cluster ids-rules-api: in proxy # mode IDS rules are relayed from the platform, and ids-rules-api serves a local # ET ingest that this edition does not run. # ============================================================================ # CUSTOMER-INPUT: the single address every rule below is published under. entrypointHost: &entrypointHost gen0sec.internal # CUSTOMER-INPUT: the scheme users actually reach that address on. # # Only needed when TLS terminates UPSTREAM of the Ingress -- a cloud load # balancer, a CDN, a tunnel. The chart otherwise infers the scheme from the # Ingress's own `tls:` block, which is absent in that setup, so it guesses http # while the browser is on https. The dashboard hands this to Better Auth as its # base URL and every sign-in is then rejected with "Invalid origin". # # Leave it unset when TLS terminates at the Ingress (i.e. you set ingress.tls). externalScheme: &externalScheme https ```
## Keeping your overrides Put them in a file under version control, not in `--set` arguments. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f my-site/platform.yaml \ --set global.imageTag=$VERSION \ --timeout 20m ``` `helm upgrade` uses only what you pass it. Anything set with `--set` on a previous install and omitted now reverts to the chart default, silently. A file makes that impossible to forget. Two things that are awkward to keep in a file, because they are per-release or per-secret: | Keep as `--set` | Why | |---|---| | `global.imageTag` | Changes every release | | `downloadProxy.apiKeySecret.name` | Or put it in your file. Just never omit it. See [Data relay](./relay.mdx) | ## What the kit contains ```bash ls $KIT/examples/*/ ``` Everything under `examples/` is a starting point you are meant to read and edit, not a black box. Each file carries comments explaining which values are yours to set. --- # Threat intelligence and the data relay Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/configure/relay # Threat intelligence and the data relay The relay is one in-cluster service, `download-api`. It is the only component with a route out, and it is how your fleet gets threat intelligence. This is not a choice you make. It is a requirement. See [Known limitations](./../troubleshoot/known-limitations.mdx#no-zero-egress-deployment). ## What it relays Read-only, upstream-produced content only. | Class | Routes | Why it is not shipped | |---|---|---| | CTI indicators | `/v1/indicators/version`, `/v1/indicators/download` | Built by the upstream intelligence pipeline | | ML models | `/v1/models/*` and the per-model aliases | Trained and published by Gen0Sec | | GeoIP | `/v1/geoip/manifest`, `/v1/geoip/{file}/download` | Licensed by MaxMind and IPInfo. The IPInfo Lite bundle is the default source | | IDS rules | `/v1/ids-rules/*` | Licensed as part of the Emerging Threats ruleset | **It is a relay, not a cache.** Nothing is stored on the way through. There is no local artifact store to size, no synchronization job to supervise, and no staleness window. An agent that polls gets whatever upstream has at that moment. **The relayed URLs are identical to the direct ones**, so your agents need no configuration change. ## What it never relays | Class | Routes | Why | |---|---|---| | Workload identity | `/v1/identity/*` | Produced in your cluster, describes your workloads | | Policy edges | `/v1/policy-edges/*` | Pod IPs, namespaces, NetworkPolicy edges | | Fleet bans | `/v1/fleet-bans/*` | Your own agents' ban decisions | These three are tenant-scoped: they describe your infrastructure or your decisions. Relaying them would push that to a service you do not run. They keep the local object store path behind the local authorization check, in relay mode exactly as without it. Uploads are never relayed. The relay is read-only, so `PUT` and `POST` on the download surface are served locally or not at all. ## How authorization works Two hops, two credentials, checked in two places. | Hop | Credential | Checked by | |---|---|---| | Agent to relay | The agent's own key, issued by your dashboard | Your cluster. Relayed routes sit behind the local auth middleware and each carries the same permission it has in direct mode | | Relay to Gen0Sec | Your Gen0Sec API key | Gen0Sec | **A caller's key is never forwarded upstream on a self-hosted deployment.** A locally-issued key cannot authenticate to gen0sec. Forwarding it would leak a local credential to a third party without making the request succeed. With no Gen0Sec API key configured, the relay sends no credential at all. ## The setting that must be right ```yaml downloadProxy: enabled: true upstream: "https://api.gen0sec.com" apiKeySecret: name: gen0sec-download-proxy # named by default key: DOWNLOAD_PROXY_API_KEY ``` ### Why the key is required Your agents' keys are issued by **your** dashboard and exist only in **your** database, so they cannot authenticate to gen0sec. The relay presents your own Gen0Sec API key instead, and that is the only credential the upstream will accept from your installation. Without it, the relay sends no credential, Gen0Sec answers 401, and no relayed artifact reaches your fleet. :::note[The chart names the secret by default, so this fails loudly now] `apiKeySecret.name` defaults to `gen0sec-download-proxy`. If that secret does not exist, the relay pod never starts: `CreateContainerConfigError`, visible immediately. That is deliberate. An empty name renders no environment variable at all, the relay starts, and the only symptom is every relayed class answering 401 while agents report empty rulesets rather than errors. A failed rollout at install time is the cheaper failure. Blank the name only if you relay through a gateway that supplies its own credential. ::: ### Getting the key right ```bash printf '%s' '' > ~/.gen0sec/relay-key kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-file=DOWNLOAD_PROXY_API_KEY=$HOME/.gen0sec/relay-key ``` The value renders as a `secretKeyRef`, so it never appears in a manifest or in a values file. ## All settings | Key | Default | Meaning | |---|---|---| | `downloadProxy.enabled` | `true` | Turning it off is not supported | | `downloadProxy.upstream` | `https://api.gen0sec.com` | The upstream base URL. A malformed value **fails startup** rather than degrading | | `downloadProxy.external` | `geoip,ids-rules` | Which external classes to relay. An unknown name fails startup | | `downloadProxy.timeout` | `60s` | Upstream request timeout, matching the router's own | | `downloadProxy.apiKeySecret.name` | `""` | **Set this.** See above | | `downloadProxy.apiKeySecret.key` | `DOWNLOAD_PROXY_API_KEY` | The key within the secret | :::note[Failing startup is deliberate] A malformed upstream is not treated as "fall back to reading from the local object store". That would leave every agent poll erroring against a store this deployment does not populate. Failing fast is the safer wrong answer. ::: ## Verifying ```bash kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' ``` The output is similar to this: ```json { "status": "ready", "mode": "proxy", "upstream": "https://api.gen0sec.com", "external_relays": ["geoip", "ids-rules"] } ``` **That tells you the relay started, not that it is authorized.** For that, read the log: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means your Gen0Sec API key is missing or wrong. ## Response codes worth knowing | Code | Means | What an agent does | |---|---|---| | 200 | Relayed successfully | Uses the artifact | | **401** | The upstream rejected the credential | **Reports an empty ruleset, not an error.** This is the silent failure | | **502** | The upstream is unreachable | Reads it as "ask again later" and keeps what it holds. Correct degradation | | 404 | The artifact does not exist upstream | Treats it as withdrawn | The 401 and 502 rows are the reason this page exists. ## Preview a change before applying it ```bash helm template g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ --set downloadProxy.apiKeySecret.name=gen0sec-download-proxy \ | grep -A3 DOWNLOAD_PROXY ``` --- # Secrets Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/configure/secrets # Secrets **No chart contains a credential.** Every one is referenced by name. This page is explains what must exist, with which keys, before the platform starts. ## The secrets you create | Secret | Namespace | Keys | Created in | |---|---|---|---| | `gen0sec-registry` | **Both** | `.dockerconfigjson` | [Install step 2](./../05-install.mdx#the-image-pull-secret) | | `gen0sec-object-store` | `gen0sec-system` | `RUSTFS_ACCESS_KEY`, `RUSTFS_SECRET_KEY` | [Install step 2](./../05-install.mdx#the-object-store-credentials) | | `gen0sec-download-proxy` | `gen0sec` | `DOWNLOAD_PROXY_API_KEY` | [Install step 2](./../05-install.mdx#your-gen0sec-api-key) | The pull secret is needed in **both** namespaces. The application runs in `gen0sec`; the operators, object store and cache run in `gen0sec-system`. ## The twenty the script creates `make-db-secrets.sh` composes one secret per service from the passwords the Postgres operator generated. | Secret | Contains | Role it uses | Endpoint | |---|---|---|---| | `g0s-db-` | `DATABASE_URL`, plus object store keys for services that use it | Its own per-service role, read-only where it only reads | The connection pooler | | `gen0sec-db-migrate` | `DATABASE_URL` | `postgres`, superuser | The primary, unpooled | | `g0s-db-ui` | `DATABASE_URL`, `BETTER_AUTH_SECRET`, object store keys | `frontend` | The connection pooler | Every service connects as its own role. The migration job needs a superuser on the primary because it runs `CREATE EXTENSION`, which an owner role cannot do. :::note[The session signing key is preserved] `g0s-db-ui` carries the dashboard's session signing secret. Re-running the script keeps the existing value, so sessions and invite links stay valid across a re-run. Nothing else regenerates it. ::: Four services need no database connection and get no `DATABASE_URL`: the GeoIP updater, the GeoIP database updater, the model publisher and the rules validator. ## Where the database passwords come from The Postgres operator creates a credentials secret per role it provisions, named after the role: ``` .core.credentials.postgresql.acid.zalan.do ``` Underscores in the role become dashes in the secret name. Compose each connection string from that role's password: ``` postgres://:@core-pooler.gen0sec-system.svc.cluster.local:5432/ax_prod?sslmode=require ``` | Consumer | Role | Endpoint | |---|---|---| | Each service | Its own per-service role | `core-pooler` | | Migration job | `postgres` | `core`, the primary, with the on-prem marker appended | | Dashboard | `frontend` | `core-pooler` | :::note[One deliberate exception] The dashboard's connection string uses `sslmode=no-verify`. Its client library verifies the certificate chain for `sslmode=require` and rejects the pooler's self-signed certificate; the Go library every other service uses reads the same value as encrypt-without-verify. **Do not copy `no-verify` onto the other services.** Their driver rejects it outright. ::: ## For production: use your secret manager `make-db-secrets.sh` is fine for evaluation. In production, treat its output as a **specification** rather than a tool, and have your secret manager produce secrets of the same shape. Two reasons: 1. The script writes plain Kubernetes secrets from operator-generated passwords. Your manager should be the source of truth, with the cluster holding copies. 2. Rotation is then a manager operation rather than a re-run plus a rollout. An external secret can template the connection string from the operator-generated password, so nothing is copied by hand: ```yaml # shape, not a working manifest: adapt to your operator data: - secretKey: DATABASE_URL remoteRef: key: g0s-auth-api-ro.core.credentials.postgresql.acid.zalan.do property: password # then template: postgres://:{{ .password }}@core-pooler...:5432/ax_prod?sslmode=require ``` To see the exact list your install needs, read it out of the values file rather than from this page: ```bash grep -oE 'secretRef:\s*\{?\s*name:\s*[a-z0-9-]+' \ $B/values/gen0sec-platform-values-onprem.yaml | sort -u ``` Or inspect your live installation inside your cluster. ## Rotation | Credential | How | |---|---| | Database role passwords | Rotate through the Postgres operator, then restart the consumers | | Object store keys | Update **both** `gen0sec-object-store` and every `g0s-db-` that carries them, then restart the consumers | | Registry pull credentials | Recreate `gen0sec-registry` in both namespaces. Existing pods keep running; new pulls use the new value | | Your Gen0Sec API key | Recreate `gen0sec-download-proxy`, then restart the relay | | Dashboard session key | Rotating invalidates every session and every outstanding invite link | :::note[One credential pair, surfaced two ways] The object store keys appear as `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` to the object store itself, and as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to the services that read it. Same values, two naming conventions. Rotating means updating both places. ::: ## Verifying Nothing should be missing before you install the platform: ```bash $KIT/scripts/preflight.sh ``` After install, confirm the count: ```bash kubectl -n gen0sec get secret | wc -l # ~20 service secrets plus the ones you created kubectl -n gen0sec-system get secret | grep credentials | wc -l # expect 6 ``` ## What is never in a secret | | Where it is instead | |---|---| | The registry host and namespace | The values file | | The relay upstream URL | The values file | | The cache connection string | The values file, deliberately with no credential in it | If you enable authentication on the cache, remove that line from the values file and put the connection string in each service's secret instead. --- # Values reference Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/configure/values # Values reference :::note[This page is the reference] It covers the keys you actually set, which is a much shorter list than every key a chart accepts, and states which of them are required. For anything not listed here, read the chart's own `values.yaml` in the bundle. There is no generated key-by-key reference. The charts carry no JSON schema yet, so nothing can be generated from one; when that changes this page will link to it. ::: ## How values are layered Three files, applied in order, last one wins. ``` chart values.yaml the chart's own defaults, registry deliberately blank values-onprem.yaml shipped in the bundle, points everything at a registry your overrides what you set ``` **Always pass `-f values-onprem.yaml`.** The default `values.yaml` leaves the registry blank on purpose, so installing without it cannot resolve images. Put your overrides in a file rather than in `--set` arguments, because `helm upgrade` uses only what you pass it. Anything set with `--set` last time and omitted this time reverts to the chart default. ```bash helm get values g0s -n gen0sec # what the running release actually has ``` ## gen0sec-infra Top-level keys: `postgres-operator`, `strimzi-kafka-operator`, `synapse-operator`, `synapse`, `rustfs`, `dragonfly`. | Key | Default | Set it when | |---|---|---| | `rustfs.storageclass.name` | `""`, meaning the cluster default | You want a specific StorageClass | | `rustfs.storageclass.dataStorageSize` | `50Gi` | Sizing the object store. **Four drives per replica** | | `rustfs.storageclass.logStorageSize` | `10Gi` | Rarely | | `rustfs.replicaCount` | `3` | Fewer nodes. Erasure coding assumes they spread | | `rustfs.clusterDomain` | `cluster.local` | Your cluster uses a custom DNS domain | | `dragonfly.extraArgs` | `[]` | Adding `--force_epoll` when `io_uring` is blocked | | `dragonfly.maxMemory` | Chart default | Keep it comfortably under the memory limit, or the kubelet kills the pod before it can evict | See [Sizing storage for your retention](./../02-requirements.mdx#sizing-storage-for-your-retention). ## gen0sec-data Top-level keys: `postgres`, `kafka`, `kafka2pg`. | Key | Default | Set it when | |---|---|---| | `postgres.numberOfInstances` | `3` | Never below 3 in production | | `postgres.volume.size` | `50Gi` | Sizing the database. This is the system of record, so it is the one to grow first | | `postgres.volume.storageClass` | `""` | You want a specific StorageClass. **Fast block storage** | | `postgres.resources` | 1 vCPU / 2 GiB requested, 8 vCPU / 12 GiB limit | Tuning. The limit sets the per-node minimum | | `kafka.replicas` | `3` | Fewer nodes. **Drop the replication factors together**, or brokers refuse writes | | `kafka.storage.size` | `50Gi` | Sizing the message log and the ingest buffer | | `kafka.storage.class` | `""` | You want a specific StorageClass | :::danger[Kafka replicas and replication factor move together] `min.insync.replicas` must stay below `replicas`. Reducing brokers without reducing the factors gives you a cluster that schedules and then refuses every write. ::: ## gen0sec-platform Top-level keys: `global`, `migration`, `downloadProxy`, `services`. ### global | Key | Default | Notes | |---|---|---| | `global.registry.host` | `registry.gen0sec.com` | Rewritten by the bundler on the offline path | | `global.registry.namespace` | `gen0sec` | Every image lives under this | | `global.imageTag` | Empty | **Required.** Set it to the release version | | `global.imagePullSecrets` | `gen0sec-registry` | Must exist in both namespaces | | `global.commonEnv.S3_ENDPOINT` | The object store service URL | Must match the object store Service, which has a `-svc` suffix | | `global.commonEnv.ARXIGNIS_DATA_URL` | The same URL plus `/platform-data` | **The bucket name must match the bucket you created** | | `global.commonEnv.REDIS_URL` | The cache service URL | No credential in it, deliberately | | `global.podSecurityContext` | Non-root, uid 65532 | Do not weaken. See [Hardening](./../security/hardening.mdx) | | `global.securityContext` | Read-only root, no capabilities | Do not weaken | | `global.ingress.className` | `synapse` | The ingress controller the infra chart installs | ### Publishing Set by the [single-entrypoint overlay](./../06-publish.mdx), not by `values-onprem.yaml`, which leaves every ingress disabled. Both are top level, and both are aliased into every rule the overlay writes — which is why they are listed here rather than under `services`. | Key | Default | Notes | |---|---|---| | `entrypointHost` | `gen0sec.internal` | **Required to publish anything.** The one address that serves the dashboard and every agent-facing API. A rule with an empty host renders, and the controller then skips it with no event and no log line. It must also be the address people actually type: the dashboard derives its sign-in URL from it | | `externalScheme` | Unset | Set to `https` **only when TLS terminates upstream** of the Ingress — a cloud load balancer, a CDN, a tunnel. With no `tls:` block to read, the chart otherwise infers `http` while the browser is on `https`, and every sign-in is rejected with `Invalid origin`. Leave it unset when TLS terminates at the Ingress | ### downloadProxy | Key | Default | Notes | |---|---|---| | `downloadProxy.enabled` | `true` | Turning it off is not supported | | `downloadProxy.apiKeySecret.name` | `""` | **Required. Empty is a silently broken deployment** | Full detail on [Data relay](./relay.mdx). This is the single most consequential key in the chart. ### services Twenty-one entries. Each one accepts the same shape, and you rarely need to touch any of it. | Key pattern | Notes | |---|---| | `services..enabled` | `threat-api` is not shipped at this release | | `services..replicaCount` | 2 or 3 in production, 1 in the quickstart overlay | | `services..resources` | Requests and limits. See [Sizing](./../02-requirements.mdx#sizing-a-production-install) | | `services..env` | Per-service environment, including `PORT` | | `services..ingress` | Off by default. Set by the entrypoint overlay | | `services..volumes`, `.volumeMounts` | **Do not remove these.** They exist because the read-only root filesystem needs a writable path where each service expects one | | `services.ui.enabled` | `true`. The dashboard | | `services.ui.entraId.enabled` | `false`. Entra ID is in development and not supported at this release. See [Sign-in and identity](./identity.mdx) | :::danger[Removing a service's volume mount breaks that service] Several services default to writing into a path relative to their working directory, which fails on a read-only root filesystem. Each one has an explicit writable volume mounted where it writes. Those mounts are load-bearing, not leftovers. ::: ## Checking what a change will do Always render before you apply. ```bash helm template g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f my-overrides.yaml \ | less ``` To diff against what is running: ```bash helm get manifest g0s -n gen0sec > running.yaml helm template g0s ... > proposed.yaml diff running.yaml proposed.yaml ``` ## Named starting points Two tested configurations rather than a blank page: [Reference configurations](./reference-configs.mdx). --- # First sign-in Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/first-signin ## Why this page exists Registration is invite-only, and invites can only be created by an existing member. A fresh install can be logged in by a pre-seeded admin account. ## The seeded account | | | |---|---| | Email | `admin@gen0sec.local` | | Password | `Gen0Sec!ChangeMe` | :::danger[This password is published, and it is the same on every install] It is a constant in our migration file, so anyone who has read our documentation knows it. The dashboard will not let you past it: the account is created with a must-change flag, so setting a new password is a step you cannot skip or postpone. What that does not do is protect the account between the moment you publish the address and the moment you first sign in. **Whoever signs in first is who sets the new password.** Sign in before you give the address to anyone else. If you have already shared it, sign in now and treat the interval as exposed. ::: ## 1. Sign in Open your published address and sign in with the details above. ```bash echo https://$ENTRYPOINT_HOST ``` If sign-in is rejected as an invalid origin, the published host or scheme does not match what you typed. See [Sign-in and identity](./configure/identity.mdx#the-published-address-decides-whether-sign-in-works). ## 2. Complete first-run setup The dashboard holds you at each of these in turn, and it will not let you skip ahead. A deep link to any other page returns you to the step you are on. 1. **Enroll multi-factor authentication.** Nothing else opens until you confirm a six-digit code. 2. **Set a new password.** Step 3 below. 3. **Create your organization.** 4. **Finish the remaining onboarding steps.** The wizard creates your first API key, then walks the WAF, access rule and monitoring surfaces. Multi-factor comes first, so the published password stops being enough to sign in before you have even reached the password screen. ## 3. Mandatory password change Because the password is preseeded, after setting up MFA, you'll be asked to change your password. :::note[The change signs out this account everywhere else] Every other session on the seeded account ends when the password changes. On a first install there are none, which is the point: if the address was reachable before you signed in, this is what closes that off. ::: The seed is also idempotent on the email address. Re-running the migration or the secret script neither duplicates the account nor re-flags it, and it does not reset a password you have already changed. ## 4. Invite the people who need access Invite your operators rather than sharing the seeded account. ## 5. Mint an agent API key Create an agent API key in the dashboard, then save it where the verification step reads it from: ```bash mkdir -p ~/.gen0sec printf '%s' 'yor-agent-key' > ~/.gen0sec/agent-key ``` ## If it fails | Symptom | Cause | |---|---| | The seeded account does not exist | The migration ran without the on-prem marker. Secrets were provisioned without `make-db-secrets.sh` | | Sign-in rejected as an invalid origin | The published host or scheme does not match the address you typed | | The dashboard loads but onboarding will not advance | Check `kubectl -n gen0sec logs deploy/g0s-ui` | | The password is rejected | It is case-sensitive and contains a `!`. Watch your shell if you pasted it | | The new password is rejected as too weak | 12 characters, one symbol, and three of the four character classes. The server applies the same rules as the form | | The password screen appears again on a later sign-in | The change did not complete. The flag clears only on success | --- # Install Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/install # Install ## Before you start - [ ] [Prepare the artifacts](./04-prepare-artifacts.mdx) complete - [ ] `$KIT`, `$B`, `$REGISTRY`, `$REGISTRY_USER`, `$REGISTRY_TOKEN` are set in your shell - [ ] `$G0S_DOWNLOAD_API_KEY` is set. Step 2 writes it into a secret, and an unset variable produces an empty one that fails silently - [ ] You are logged in to the registry ## 1. Check the cluster ```bash $KIT/scripts/preflight.sh ``` This checks the prerequisites the charts assume but do not create: a reachable cluster, a StorageClass, and the namespaces and secrets created in step 2. On a fresh cluster it reports the namespaces and secrets as missing, which is expected at this point. Pass a StorageClass name to pin it instead of accepting the default: ```bash $KIT/scripts/preflight.sh gen0sec-system gen0sec my-block-storage ``` ## 2. Create the namespaces and the two bootstrap secrets ```bash for ns in gen0sec-system gen0sec; do kubectl create namespace $ns --dry-run=client -o yaml | kubectl apply -f - done ``` Written to converge, so re-running step 2 after a failure further down is safe rather than a wall of `AlreadyExists`. ### The image pull secret Both namespaces need it. The application runs in `gen0sec`; the operators, object store and cache run in `gen0sec-system`. ```bash for ns in gen0sec-system gen0sec; do kubectl -n $ns create secret docker-registry gen0sec-registry \ --docker-server=$REGISTRY \ --docker-username=$REGISTRY_USER \ --docker-password=$REGISTRY_TOKEN \ --dry-run=client -o yaml | kubectl apply -f - done ``` ### The object store credentials :::danger[Run this once, and only once] Unlike the rest of step 2 this does **not** converge. It mints new random keys every time it runs, and the object store's existing data was written with the old ones — recreating the secret on a second pass strands that data behind credentials nothing holds any more. The guard below stops before that happens. ```bash kubectl -n gen0sec-system get secret gen0sec-object-store >/dev/null 2>&1 \ && echo "already exists — skip to step 3" \ || echo "not present — run the block below" ``` ::: ```bash umask 077 S=$(mktemp -d) openssl rand -hex 16 | tr -d '\n' > "$S/RUSTFS_ACCESS_KEY" openssl rand -hex 32 | tr -d '\n' > "$S/RUSTFS_SECRET_KEY" kubectl -n gen0sec-system create secret generic gen0sec-object-store --from-file="$S" ``` **Save both files somewhere safe before you delete them.** Nothing regenerates these keys, and the data in the object store is written with them. ```bash mkdir -p ~/gen0sec-object-store-keys cp "$S"/RUSTFS_* ~/gen0sec-object-store-keys/ # or load them into your secret manager rm -rf "$S" ``` Step 5 reads them back from that directory. ### Your Gen0Sec API key The relay presents this key to Gen0Sec when it fetches threat intelligence, GeoIP, models and IDS rules. The chart expects this exact secret name, so the relay pod will not start without it. See [Credentials from gen0sec](./02-requirements.mdx#your-gen0sec-api-key). ```bash # Refuses to run rather than creating an empty secret. An empty key is accepted # by Kubernetes and by the chart, and only surfaces later as every relay fetch # returning 401 across the whole fleet — with nothing in the install log. : "${G0S_DOWNLOAD_API_KEY:?not set — see Prepare the artifacts, step 0}" kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-literal=DOWNLOAD_PROXY_API_KEY=$G0S_DOWNLOAD_API_KEY ``` ## 3. Install infra ```bash helm install g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml ``` This installs three operators (Postgres, Kafka, ingress), the object store and the shared cache. ### Wait for all five The next step creates resources these operators have to reconcile. Do not continue until every one is available. ```bash kubectl -n gen0sec-system rollout status deploy/g0s-infra-postgres-operator kubectl -n gen0sec-system rollout status deploy/strimzi-cluster-operator kubectl -n gen0sec-system rollout status deploy/g0s-infra-synapse-operator kubectl -n gen0sec-system rollout status deploy/dragonfly kubectl -n gen0sec-system rollout status \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" ``` The output for each is similar to this: ``` deployment "g0s-infra-postgres-operator" successfully rolled out ``` :::note[Why the last one resolves its own kind] The object store is a StatefulSet in the production configuration and a Deployment in the single-node configuration. Resolving the kind means the same command works in both. ::: ## 4. Install data ```bash helm install g0s-data $B/charts/gen0sec-data-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml ``` This creates the Postgres cluster, the Kafka cluster, its topics, and the database roles. ### Wait for both clusters Both take several minutes. Nothing after this works until Postgres reports `Running`. ```bash kubectl -n gen0sec-system wait --for=condition=Ready kafka/core --timeout=15m kubectl -n gen0sec-system wait \ --for=jsonpath='{.status.PostgresClusterStatus}'=Running postgresql/core --timeout=15m ``` ## 5. Create the object store bucket The object store starts empty and nothing creates the bucket for you. Create it now, before the platform install. Services that read or write the object store fail until it exists. This step is required on every install. The two datasets your cluster produces about itself, `identity` and `policy-edges`, live here and are never sent anywhere. See [Network and connectivity](./03-network.mdx#what-never-leaves-your-cluster). Run it as a throwaway pod in the cluster. The pod reads the keys straight from the `gen0sec-object-store` secret, so they never reach your shell and never touch your local AWS configuration. ```bash kubectl -n gen0sec-system run s3-init --rm -i --restart=Never \ --image=amazon/aws-cli:2.15.0 \ --overrides='{"spec":{"containers":[{"name":"s3-init","image":"amazon/aws-cli:2.15.0", "env":[{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"name":"gen0sec-object-store","key":"RUSTFS_ACCESS_KEY"}}}, {"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"name":"gen0sec-object-store","key":"RUSTFS_SECRET_KEY"}}}, {"name":"AWS_EC2_METADATA_DISABLED","value":"true"}, {"name":"AWS_REGION","value":"us-east-1"}], "args":["--endpoint-url","http://rustfs-svc.gen0sec-system.svc.cluster.local:9000","s3","mb","s3://platform-data"]}]}}' ``` Expect `make_bucket: platform-data`. `BucketAlreadyOwnedByYou` is also a pass, so this step converges: running it twice is harmless. :::note[Offline installs: mirror this one image first] `amazon/aws-cli` is **not** in the offline bundle. The bundle carries only what `release-manifest.yaml` pins, and its `SHA256SUMS` covers the whole tree — the bundler refuses a package containing anything the checksums do not list — so it cannot be added after the fact. Mirror it alongside the bundle, then point the pod at your copy: ```bash skopeo copy docker://amazon/aws-cli:2.15.0 docker://$REGISTRY/amazon/aws-cli:2.15.0 ``` Substitute `$REGISTRY/amazon/aws-cli:2.15.0` for `amazon/aws-cli:2.15.0` in both the `--image` flag and the `image` field of the overrides above. They must match: the `--image` flag sets the pod's image, and the override object sets the container's, and Kubernetes uses the latter. ::: Verify by running the same command with `"s3","ls"` in place of `"s3","mb","s3://platform-data"`. The bucket should be listed: ``` 2026-09-01 14:02:11 platform-data ``` An empty result means the bucket was not created, and the services in step 7 will fail with `500 NoSuchBucket`. Do not continue. :::note[If you changed the bucket name] `platform-data` is the value of `global.commonEnv.S3_BUCKET`. If you overrode it, use your name instead. ::: :::caution[Do not export the object-store keys into your shell] The in-cluster method above is recommended because the credentials never enter your shell. That matters most on managed Kubernetes. **On EKS, exporting them breaks `kubectl`.** The kubeconfig authenticates through an `aws eks get-token` exec plugin that reads `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` — the same two variables. Setting them to the object-store keys makes every later command in that shell fail with: ``` error: You must be logged in to the server (Unauthorized) ``` The error names kubectl, not the variables, which makes it hard to attribute. The in-cluster method avoids it entirely. If you use the workstation method below, scope the credentials to the one command with `env ...` rather than exporting them. **If the object store returns `401`,** the keys are not the ones this install generated. Keys left over from a previous install of the same cluster do this, and so does a stale copy in `~/gen0sec-object-store-keys/`. Credentials are genuinely validated: wrong keys return `InvalidAccessKeyId`, and none at all returns `403`. :::
Workstation method, for reference This needs no S3 client image in the cluster, which is why it was preferred: on the offline bundle method an in-cluster image has to be mirrored first, so the bucket comes to depend on a step that exists only to create the bucket. Use it if that trade-off suits you — but read the caution above first, because exporting the object-store keys will break `kubectl` on EKS. ### 5a. Open a port forward In a second terminal: ```bash kubectl -n gen0sec-system port-forward svc/rustfs-svc 9000:9000 ``` Leave it running. The output is similar to this: ``` Forwarding from 127.0.0.1:9000 -> 9000 ``` ### 5b. Create the bucket Back in your first terminal, use the two keys you generated in step 2. Scope them to the one command with `env` rather than exporting them, so they cannot clobber the variables your kubeconfig's exec plugin reads: ```bash env AWS_ACCESS_KEY_ID="$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ AWS_SECRET_ACCESS_KEY="$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" \ AWS_REGION=us-east-1 AWS_EC2_METADATA_DISABLED=true \ aws --endpoint-url http://localhost:9000 s3 mb s3://platform-data ``` The output is similar to this: ``` make_bucket: platform-data ``` `BucketAlreadyOwnedByYou` is also a pass. With the MinIO client instead: ```bash mc alias set g0s http://localhost:9000 \ "$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ "$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" mc mb g0s/platform-data mc ls g0s ``` ### 5c. Verify it exists ```bash env AWS_ACCESS_KEY_ID="$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ AWS_SECRET_ACCESS_KEY="$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" \ AWS_REGION=us-east-1 AWS_EC2_METADATA_DISABLED=true \ aws --endpoint-url http://localhost:9000 s3 ls ``` The output is similar to this: ``` 2026-09-01 14:02:11 platform-data ``` An empty result means the bucket was not created, and the services in step 7 will fail with `500 NoSuchBucket`. Do not continue. Then stop the port forward with `Ctrl-C` in the second terminal.
## 6. Create the application secrets ```bash $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml ``` This composes around twenty secrets from the passwords the Postgres operator generated in step 4. That is why it cannot run earlier. Each service gets its own database role through the connection pooler. The migration job gets a superuser connection direct to the primary, because it runs `CREATE EXTENSION`. :::note[In production, treat this script's output as a specification] `make-db-secrets.sh` is fine for evaluation. For production, have your secret manager produce secrets of the same shape. Every name and key is listed in [Secrets](./configure/secrets.mdx). ::: ## 7. Install the platform ```bash helm install g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ --set global.imageTag=$VERSION \ --timeout 20m ``` :::note[Nothing to set for the relay] The chart already points at `gen0sec-download-proxy` by default. You created that secret in step 2, and if it is missing the relay pod fails to start with `CreateContainerConfigError` rather than coming up half-configured. ::: Schema migrations run first, as a pre-install hook. If they fail, Helm aborts and nothing is deployed. That is deliberate: a partially migrated database is worse than no install. ### Check the migration ```bash kubectl -n gen0sec get job g0s-db-migrate ``` The output is similar to this: ``` Error from server (NotFound): jobs.batch "g0s-db-migrate" not found ``` **`NotFound` is the success case.** The job is a Helm hook with `hook-delete-policy: hook-succeeded`, so Helm deletes it the moment it passes. If the job is still there, it may have failed: ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` ### Wait for the pods ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m ``` :::note[Check readiness, not phase] A pod in `CrashLoopBackOff` still reports `status.phase: Running`. A `--field-selector` on phase reports a broken install as clean. ::: :::note[Two values that must match your cluster] Both are set in `values-onprem.yaml` and both point at the in-cluster object store. Check them if anything cannot read its data. | Key | Must be | |---|---| | `global.commonEnv.S3_ENDPOINT` | `http://rustfs-svc.gen0sec-system.svc.cluster.local:9000` | | `global.commonEnv.ARXIGNIS_DATA_URL` | The same endpoint, plus the bucket: `.../platform-data` | `ARXIGNIS_DATA_URL` is where in-cluster services read their datasets from. It points at your object store, not at Gen0Sec. **The bucket name at the end of it must match the bucket you created in step 5.** If you used a different name there, change it here too. The Service is `rustfs-svc`, with the `-svc` suffix. That suffix is the single most common cause of services that come up and then cannot read anything. ::: ## If it fails | Symptom | Start here | |---|---| | `ImagePullBackOff` | [ImagePullBackOff](./troubleshoot/symptoms.mdx#imagepullbackoff) | | Install aborts at `g0s-db-migrate` | [Migration failures](./troubleshoot/symptoms.mdx#migration-failures) | | `postgresql/core` never reaches `Running` | [Postgres will not start](./troubleshoot/symptoms.mdx#postgres-will-not-start) | | Kafka broker crash loops | [Kafka will not start](./troubleshoot/symptoms.mdx#kafka-will-not-start) | --- # Network and connectivity Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/network # Network and connectivity ## Why the platform must reach Gen0Sec No platform is useful without data, neither is Gen0Sec. Because the datasets are licensed per customer, they are not in the install kit, not in the offline bundle, and not in any container image. The platform fetches them at runtime from `api.gen0sec.com`. | Dataset | Licensed by | |---|---| | CTI indicators | Upstream threat intelligence providers | | GeoIP databases | MaxMind, IPInfo | | IDS rules | Emerging Threats ruleset | | ML models | Trained and published by Gen0Sec | One in-cluster service, `download-api`, does the fetching. It relays each request upstream and streams the response straight back. It is a relay, not a cache, so there is no artifact bucket to size, no sync job to supervise and no staleness window. An agent that polls gets whatever upstream has at that moment. This also means you get the most recent data we can provide. ## One hop, not many Your agents never talk to the internet. They talk to `download-api`, and `download-api` is the only service in either namespace with a route out. ``` agents -> download-api -> api.gen0sec.com:443 ^ | the only egress point in the deployment ``` That means one destination to allow, one to audit. It also means the relayed URLs are identical to the direct ones, so agents need no configuration change. ## What never leaves your cluster Two datasets describe your own infrastructure, and they are never relayed: | Dataset | What it contains | Where it stays | |---|---|---| | `identity` | Workload identity produced in your cluster | Your object store | | `policy-edges` | Pod IPs, namespaces, NetworkPolicy edges | Your object store | No hidden switches, no configuration options, it is not implemented. Your data is yours and yours only. Uploads are not relayed either. The relay is read-only, so `PUT` and `POST` on the download surface are served locally or not at all. ## How authorization works **Your agents' API keys never leave your cluster.** Agents authenticate to your installation with keys your own dashboard issued, held in your own database. On a correct configuration the relay does not forward those keys upstream. It presents **your own Gen0Sec API key**, the one issued with your subscription, on the second hop only. ``` agent --[your agent's key]--> relay --[your Gen0Sec API key]--> api.gen0sec.com agent key stops here ``` | | | |---|---| | What Gen0Sec sees | One credential, belonging to your installation | | What Gen0Sec never sees | Any individual agent key, or how many you have issued | | Revoking an agent | Entirely local, in your dashboard | | Revoking your installation's access | Revoke or rotate your Gen0Sec API key | :::danger[This key is required, and omitting it fails silently] With no Gen0Sec API key configured, the relay falls back to forwarding the caller's own key. That is correct for the hosted product, where the same platform issued it. Here it is always wrong: your keys exist only in your cluster, so Gen0Sec rejects every one with a 401, and **the agent reports an empty ruleset rather than an error**. The install may look healthy, but the fleet quietly stops receiving threat intelligence, GeoIP, models and IDS rules. Set up in [Install](./05-install.mdx) step 2, enabled in step 7, and checked by [Verify the install](./08-verify.mdx). ::: Routes that are not relayed, including `identity` and `policy-edges`, keep the local authorization check against your own database. ## Egress list All connections are outbound, TCP, TLS. Nothing needs an inbound rule from the internet. | Destination | Port | Needed by | Purpose | Required | |---|---|---|---|---| | `api.gen0sec.com` | 443 | Cluster pods (`download-api` only) | Threat intelligence, GeoIP, ML models, IDS rules | **Always** | | `releases.gen0sec.com` | 443 | Your workstation | Download the install kit and its signature | **Always** | | `registry.gen0sec.com` | 443 | Your workstation | `helm pull` for charts, `skopeo copy` for images | **Always** | | `registry.gen0sec.com` | 443 | Cluster nodes (kubelet) | Image pulls at install time | Registry install only | | `fulcio.sigstore.dev` | 443 | Your workstation | `cosign` certificate verification | **Always** | | `rekor.sigstore.dev` | 443 | Your workstation | `cosign` transparency log verification | **Always** | | `tuf-repo-cdn.sigstore.dev` | 443 | Your workstation | `cosign` trust root refresh | **Always** | | Your own registry | 443 or 80 | Cluster nodes, and the host that runs `bundler.sh apply` | Image pulls at install time | Offline bundle only | {/* ASSUMPTION: this list is derived from the release workflow, the bundler and the chart values. Confirm it is complete, and confirm the Sigstore host set, which changes with cosign versions. */} :::note[The Sigstore hosts are the ones people forget] `cosign verify-blob` checks a signature against a public transparency log. Without those three destinations it fails, and it fails on the first command of the install. Allow them on the workstation, not on the cluster. ::: ### Inside the cluster No NetworkPolicy is required for the install. If you enforce a default-deny policy, the two namespaces need to reach each other: the application in `gen0sec` talks to Postgres, Kafka, the object store and the cache in `gen0sec-system`. ## What this rules out `api.gen0sec.com` on port 443 is required for the platform to serve data. An agent fleet with the platform hosted on a cluster that is permitted no outbound connections at all cannot function correctly. The offline bundle method removes every dependency on gen0sec-hosted registries: your cluster pulls images only from your own. It does not remove the runtime connection, because the mentioned datasets above are not ours to ship. If your policy allows no cluster egress, talk to us. We would rather hear it now than during your evaluation. > **Air-gapped.** Often used to mean "no connection of any kind". Under that definition Cerebellum > is not deployable. If it means "no dependency on vendor-hosted registries and one audited > outbound destination", the offline bundle method fits, and this page is the list your firewall team > needs. --- # Back up and restore Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/operate/backup # Back up and restore ## What to back up Not everything in the deployment is equally valuable. Backing up the wrong things is how backup jobs become expensive and untested. | Component | Back up | If you lose it | |---|---|---| | **Postgres** | **Yes, always** | Everything. Configuration, agents, policy, history. This is the system of record | | **Object store** | **Yes** | The datasets and the artifacts your cluster produced about itself. Datasets are re-fetchable, `identity` and `policy-edges` are not | | Kafka | No | In-flight messages only. It is a transport, not a store. Producers re-send | | Shared cache | No | Nothing. It repopulates | | Secrets | **Yes** | Access to your own volumes. See the warning below | :::danger[Back up the volumes along with the secrets, or the volumes are useless] The database passwords are generated by the Postgres operator and stored as Kubernetes secrets. Data is written with those credentials. A volume backup without the matching secret backup restores data you cannot open. ::: ## Postgres The Postgres operator ships a logical backup image, already mirrored as part of the release. Enable it in the infra values rather than scripting `pg_dump` yourself. ```yaml # gen0sec-infra values postgres-operator: configLogicalBackup: logical_backup_schedule: "0 2 * * *" logical_backup_provider: s3 ``` Verify a backup ran: ```bash kubectl -n gen0sec-system get cronjob | grep logical-backup kubectl -n gen0sec-system get job | grep logical-backup ``` :::note[A backup you have not restored is not a backup] Restore into a scratch namespace once, before you need to. The [Quickstart](./../quickstart.mdx) cluster is a reasonable place to rehearse it, and rehearsing a restore is also a rehearsal of an upgrade rollback. ::: ## Object store Two prefixes matter, and they matter differently. | Prefix | Recoverable without a backup? | |---|---| | `identity/`, `policy-edges/` | **No.** Produced by your cluster about your cluster | | Dataset prefixes | Yes. They are fetched from Gen0Sec | Back up the whole bucket with any S3-compatible tool, with an in-cluster job or manually, over a port forward as in [Install step 5](./../05-install.mdx#5-create-the-object-store-bucket): ```bash kubectl -n gen0sec-system port-forward svc/rustfs-svc 9000:9000 ``` ```bash aws --endpoint-url http://localhost:9000 s3 sync s3://platform-data ./platform-data-backup ``` Restore is the same command with the arguments reversed. ## Secrets ```bash kubectl -n gen0sec-system get secret -o yaml > secrets-gen0sec-system.yaml kubectl -n gen0sec get secret -o yaml > secrets-gen0sec.yaml ``` **These files contain live credentials in base64.** Treat them exactly as you treat the database itself: encrypted at rest, access controlled, and out of any shared drive. If you run an external secret manager, you do not need this: your manager is already the source of truth and the cluster holds copies. See [Secrets](./../configure/secrets.mdx). ## Recovery point and recovery time | | Value | Determined by | |---|---|---| | **Recovery point** | Your backup interval | The logical backup schedule. Nightly gives up to 24 hours of loss | | **Recovery time** | Measure it | Database size and your storage throughput. Rehearse it and record your own figure | We do not publish an RTO as it depends entirely on your configuration specifics and the data size. Time your own rehearsal and use that number in your own runbook. ## Restoring Order matters, and it is the install order. 1. Install infra and data as on a [first install](./../05-install.mdx), so the operators and empty clusters exist. 2. Restore the secrets, before anything tries to use the volumes. 3. Restore Postgres from the logical backup. 4. Restore the object store bucket. 5. Install the platform. **The migration hook runs against restored data**, so the restored schema must match the release you are installing. Restore into the version it came from, then upgrade. 6. Run the smoke test: `$KIT/scripts/verify-deployment.sh`. :::note[Do not restore an old database into a new release] The migration chain expects to run forward from the schema version in the dump. Restoring a v0.1 backup and installing v0.3 skips the intermediate migrations. Restore v0.1, then upgrade one minor at a time. See [Upgrade](./upgrade.mdx). ::: --- # Monitor Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/operate/monitor # Monitor ## What the platform exposes today | Signal | Where | Notes | |---|---|---| | Health and readiness | `/status/healthz` and `/status/readyz` on each service | Three services serve only `healthz`. See [Known limitations](./../troubleshoot/known-limitations.mdx#three-services-report-readiness-from-a-weaker-probe) | | Relay mode and upstream | `/status/healthz` on the relay | Returns the mode, the upstream and the active relay classes | | Prometheus metrics | Worker port 8080 | Exposed, but no Service is created for workers, so scraping needs a direct pod target | | Kubernetes state | Pods, Deployments, custom resources | The primary signal | **No ServiceMonitor, alert rules or dashboards ship with the charts at the moment.** If you run Prometheus, you build the scrape configuration. ## The four checks worth alerting on If you set up nothing else, set up these. ### 1. Pods not ready ```bash kubectl -n gen0sec get pods --no-headers \ | awk '{split($2,r,"/"); if (r[1] != r[2]) print}' ``` Any output is a problem. **Read the ready column, never a field selector on phase:** a pod in `CrashLoopBackOff` still reports `status.phase: Running`. ### 2. The database cluster is not Running ```bash kubectl -n gen0sec-system get postgresql core \ -o jsonpath='{.status.PostgresClusterStatus}' ``` Expect `Running`. Anything else and nothing downstream works. ### 3. The relay lost its upstream ```bash kubectl -n gen0sec exec deploy/g0s-download-api -- \ curl -fsS localhost:9999/status/healthz ``` Expect `"status": "ready"` and `"mode": "proxy"`. An unreachable upstream answers **502** on relayed routes rather than 404 or 500. That is deliberate: a polling agent reads 502 as "ask again later" and keeps the artifact it already holds, instead of treating it as withdrawn. So a broken upstream degrades your fleet slowly rather than immediately, which is safer and also easier to miss. Alert on it. ### 4. The platform stopped serving agents ```bash $KIT/scripts/verify-deployment.sh ``` Read-only, safe against production, exits non-zero and names the failing check. Running it on a schedule is the single most useful synthetic check available, because it is the only one that proves data actually moves. ## Signals that are noisy on purpose Do not page on these. | Signal | Why it is not an incident | |---|---| | `falling back to L1-only cache` | The shared cache is never fatal. Services run on their in-process cache. Reads get slower, nothing breaks | | A 502 from a relayed route, once | Upstream hiccup. Agents retry and keep what they have | | The migration job absent | That is success. Helm deletes it on completion | ## Logs ```bash kubectl -n gen0sec logs deploy/ --tail=200 kubectl -n gen0sec-system logs deploy/g0s-infra-postgres-operator --tail=200 ``` All services log to standard output as structured JSON, so any collector that reads container stdout works without configuration. Log level is set per service through the values file. See [Values reference](./../configure/values.mdx). ## What a healthy install looks like Five statements, the same five as [Verify the install](./../08-verify.mdx#definition-of-installed-successfully). If you build a dashboard, build it to answer these: 1. Every pod in both namespaces is `Running` and fully ready. 2. `postgresql/core` reports `Running`. 3. `kafka/core` reports `Ready`. 4. The relay reports `"mode": "proxy"` with your upstream. 5. The smoke test exits zero. --- # Roll back Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/operate/rollback # Roll back Read the boundary section first. It decides whether a rollback is safe or whether you need a restore. ## The boundary, stated plainly **`helm rollback` reverts manifests. It does not revert your database.** That single sentence is the whole risk. Helm restores the previous Deployments, images and configuration. Any schema change the upgrade's migration hook applied is still applied. If the old application cannot read the new schema, rolling back the manifests gives you old code against a new database. | Situation | Rollback is | |---|---| | Patch upgrade within a train, no migration ran | Safe. Just do it | | Minor upgrade whose migrations only added things | Safe. Old code ignores what it does not know | | Minor upgrade whose migrations removed or renamed something the old code reads | **Not safe.** Restore from backup instead | | You are not sure which | Check the release notes. If they do not say, ask us before rolling back | :::note[Why we prefer expand and contract] Migrations that add in one train and remove in a later one mean a one-release rollback never needs a schema change. That is the design intent, and it is what makes most rollbacks safe. It is not a guarantee for every release, which is why this page exists rather than a one-line reassurance. ::: ## Before you start - [ ] You know which revision you are going back to - [ ] You have read the release notes for the version you are leaving - [ ] The previous release's images are still in your registry **Offline bundle method:** mirroring is additive, so the old tags are still there unless you deleted them. Do not prune your registry between an upgrade and the point at which you are confident. ## 1. Find the revision ```bash helm history g0s -n gen0sec ``` The output is similar to this: ``` REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Mon Sep 1 10:14:02 2026 superseded gen0sec-platform-0.1.0 0.1.0 Install complete 2 Mon Sep 1 15:41:18 2026 deployed gen0sec-platform-0.2.0 0.2.0 Upgrade complete ``` ## 2. Roll back the platform Start here. In most cases this is the only step you need, because the application is what broke. ```bash helm rollback g0s 1 -n gen0sec ``` ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` If the smoke test passes, stop. Do not roll back infra or data unless they are the problem. ## 3. Roll back the schema, only if you must Needed when the upgrade applied a change the old code cannot work with. Check the release notes first. ```bash kubectl -n gen0sec run db-migrate-down --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/db-migrate: \ --env DATABASE_URL="$OWNER_DSN" -- down ``` `$OWNER_DSN` is the superuser connection string from the `gen0sec-db-migrate` secret. See [Secrets](./../configure/secrets.mdx). :::danger[A down migration can lose data] Reversing a migration that dropped a column recreates the column, not its contents. If the upgrade removed data, only a restore brings it back. See [Back up and restore](./backup.mdx). ::: ## 4. Roll back infra and data, only if they are the problem Same order as the upgrade, in reverse: platform, then data, then infra. ```bash helm rollback g0s-data 1 -n gen0sec-system helm rollback g0s-infra 1 -n gen0sec-system ``` :::note[CRDs do not roll back] Helm never manages CRDs after the first install, in either direction. A rolled-back operator runs against the newer CRD schema. That works when the change was additive, which is the usual case, and it is another reason to roll back the platform alone where possible. ::: ## When a restore is the right answer instead Rollback is for "the new version misbehaves". Restore is for "the data is wrong". | You want to | Use | |---|---| | Undo a bad release | Rollback | | Undo a destructive migration | Restore | | Recover from data corruption | Restore | | Return to a point in time | Restore | [Back up and restore](./backup.mdx). ## If it fails | Symptom | Cause | |---|---| | `ImagePullBackOff` after rollback | The previous version's images were pruned from your registry | | Old pods start, then error on database queries | Schema mismatch. Go to step 3, or restore | | `helm rollback` reports no such revision | `helm history` was truncated. `helm get manifest g0s --revision N` to confirm what exists | ## Next [Uninstall](./uninstall.mdx), or [Back up and restore](./backup.mdx). --- # Rotate credentials Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/operate/rotate # Rotate credentials Credentials rotates differently, and some of them have a consequence worth knowing before you start. | Credential | Rotating it interrupts | Reversible | |---|---|---| | [Database role passwords](#database-role-passwords) | The services that use them, briefly | Yes | | [Object store keys](#object-store-keys) | Everything that reads the object store | Yes, **if you kept the old values** | | [Registry pull credentials](#registry-pull-credentials) | Nothing running. New pulls only | Yes | | [Your Gen0Sec API key](#your-gen0sec-api-key) | Artifact delivery, briefly | Yes | | [Dashboard session signing key](#dashboard-session-signing-key) | **Every session and every invite link** | No | If you run an external secret manager, rotation happens there and the cluster picks up the new value. This page is for installs holding plain Kubernetes secrets. See [Secrets](./../configure/secrets.mdx). ## Database role passwords Rotate through the Postgres operator, not by editing the secret. The operator owns these values, and an edited secret is overwritten on its next reconciliation. After the operator has regenerated the credentials, recompose the application secrets and restart the consumers: ```bash $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy ``` The script preserves the dashboard's session signing key, so re-running it does not sign people out. ## Object store keys These are the ones you generated during install, and **nothing regenerates them**. Rotating means updating two places. ```bash umask 077 S=$(mktemp -d) openssl rand -hex 16 | tr -d '\n' > "$S/RUSTFS_ACCESS_KEY" openssl rand -hex 32 | tr -d '\n' > "$S/RUSTFS_SECRET_KEY" kubectl -n gen0sec-system delete secret gen0sec-object-store kubectl -n gen0sec-system create secret generic gen0sec-object-store --from-file="$S" ``` Then restart the object store, and recompose the application secrets, because the services receive the same values under different names: ```bash kubectl -n gen0sec-system rollout restart \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy ``` Save the new files where you kept the old ones, then remove the temporary directory. :::danger[Do not lose these] Data in the object store was written with them. A retained volume without the matching keys is data you cannot open. See [Back up and restore](./backup.mdx). ::: ## Registry pull credentials Recreate the secret in **both** namespaces. Running pods keep running; the new value applies to the next pull. ```bash umask 077 AUTH=$(printf '%s:%s' "$REG_USER" "$(cat $REG_PASS_FILE)" | base64 | tr -d '\n') printf '{"auths":{"%s":{"auth":"%s"}}}' "$REGISTRY" "$AUTH" > dockerconfig.json unset AUTH for ns in gen0sec-system gen0sec; do kubectl -n $ns delete secret gen0sec-registry kubectl -n $ns create secret generic gen0sec-registry \ --type=kubernetes.io/dockerconfigjson \ --from-file=.dockerconfigjson=dockerconfig.json done rm -f dockerconfig.json ``` Rotate before the old credential expires, not after. An expired pull secret is invisible until something reschedules. ## Your Gen0Sec API key ```bash printf '%s' '' > $RELAY_KEY_FILE kubectl -n gen0sec delete secret gen0sec-download-proxy kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-file=DOWNLOAD_PROXY_API_KEY=$RELAY_KEY_FILE kubectl -n gen0sec rollout restart deploy/g0s-download-api ``` Confirm it worked, because a wrong key fails silently: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means the new key is wrong, or has a trailing newline. Use `printf`, not `echo`. See [Data relay](./../configure/relay.mdx). ## Dashboard session signing key :::danger[This signs everyone out] Rotating it invalidates every active session and every outstanding invite link. There is no partial rotation and no grace period. ::: Only do this if you believe the key is compromised. Delete it and let the secret script regenerate it: ```bash kubectl -n gen0sec delete secret g0s-db-ui $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy/g0s-ui ``` The script preserves an existing key, which is why deleting the secret first is the step that actually rotates it. ## Certificates | Certificate | Owned by | |---|---| | Ingress TLS | You. Rotate in your ingress or through cert-manager | | Postgres pooler | The Postgres operator, self-signed, rotated by it | | Relay to Gen0Sec | Public certificate on our side. Nothing to do | The platform presents no certificate of its own outside the ingress you configure. ## After any rotation ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` The smoke test is the only check that proves the platform still serves an agent rather than merely starting. --- # Uninstall Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/operate/uninstall # Uninstall ## What you are about to delete | | Default | `--keep-data` | `--keep-crds` | |---|---|---|---| | The three Helm releases | Removed | Removed | Removed | | Objects the operators created outside the releases | Removed | Removed | Removed | | Kafka topic finalizers | Cleared | Cleared | Cleared | | Namespaces `gen0sec` and `gen0sec-system` | **Deleted** | Kept | Deleted | | Volumes: Postgres, Kafka, object store | **Deleted** | Kept | Deleted | | Secrets, including the database passwords | **Deleted** | Kept | Deleted | | Operator CRDs | **Removed** | Removed | Kept | :::danger[The default deletes your data, and it asks you to prove you mean this cluster] Without `--keep-data` the databases and the object store go, and nothing can undo that. So that run also **asks you to type the current context name back**, and refuses to run at all with no terminal attached. `--keep-crds` does not trigger that prompt, because a schema the next install reinstalls anyway is no reason to skip asking about the databases. ::: ## Remove everything ```bash $KIT/scripts/uninstall.sh --yes ``` It prints the cluster context, the release list and the volume count, then waits for you to type the context name. Read what it prints before you type. ## Keep the data ```bash $KIT/scripts/uninstall.sh --yes --keep-data ``` Stops after clearing the topic finalizers. The namespaces stay, and so do the volumes and the secrets in them. No context confirmation, because nothing irreversible happens. :::note[Why the secrets have to stay with the volumes] A reinstall over retained volumes needs the original database passwords: the data was written with them and a fresh Postgres operator generates different ones. Keeping the volumes without the secrets leaves you with data you cannot open. ::: ## Keep the CRDs ```bash $KIT/scripts/uninstall.sh --yes --keep-crds ``` Use this when something else in the cluster runs the same operators, or when you want the next install to skip reinstalling schemas. ## How CRD removal is made safe CRDs are cluster-scoped, and **deleting one deletes every resource of that kind in every namespace**. Another Postgres or Kafka cluster run by the same operators would go with them. So the script checks first: if any of those kinds is in use outside the two namespaces it is removing, **it skips the CRDs** and tells you. `--keep-crds` skips them unconditionally. That check is why removing CRDs by default is safe. It is not a blanket delete. If you want to see what would be affected before you run anything: ```bash kubectl get postgresqls,kafkas,kafkatopics,kafkanodepools -A 2>/dev/null | grep -v 'gen0sec' ``` Any output there means the script will leave the CRDs alone. ## The Kafka finalizer, and why order matters Strimzi puts a `strimzi.io/topic-operator` finalizer on all 23 KafkaTopics. Once the operator is gone, nothing clears it, and deleting the namespace leaves it in `Terminating` indefinitely. The script uninstalls the releases first, then clears the finalizers, then deletes the namespaces. That order is the whole reason the script exists. If you already deleted the namespace and it is stuck, this is the fix: ```bash NS=gen0sec-system kubectl get kafkatopics.kafka.strimzi.io -n $NS -o name \ | xargs -r -n1 kubectl patch -n $NS --type=merge \ -p '{"metadata":{"finalizers":null}}' ``` ## Reinstalling over retained volumes Two things bite here. **Kafka.** The volumes survive on purpose, and a recreated cluster generates a new identity. Brokers then crash loop with `Invalid cluster.id`. See [Kafka will not start](./../troubleshoot/symptoms.mdx#kafka-will-not-start). **Postgres.** You need the original role passwords. If you kept the secrets, the operator adopts the existing data. If you did not, the volumes are unreadable. ## Uninstalling by hand The script does more than these commands: it also removes objects the operators created outside the releases, and it handles the CRD safety check. Use it if you can. ```bash helm uninstall g0s -n gen0sec helm uninstall g0s-data -n gen0sec-system helm uninstall g0s-infra -n gen0sec-system # clear the finalizers before touching the namespaces kubectl get kafkatopics.kafka.strimzi.io -n gen0sec-system -o name \ | xargs -r -n1 kubectl patch -n gen0sec-system --type=merge \ -p '{"metadata":{"finalizers":null}}' kubectl delete namespace gen0sec kubectl delete namespace gen0sec-system ``` Do not pass `--wait` to `helm uninstall`. The Kafka topics cannot finish deleting until the finalizers above are cleared, so waiting only burns the timeout. --- # Upgrade Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/operate/upgrade # Upgrade When this page is done you are running the new release, with the same data. Read [Roll back](./rollback.mdx) before you start. The rollback boundary changes what a safe upgrade looks like. ## Before you start - [ ] Read the release notes for every version between yours and the target - [ ] A current backup. See [Back up and restore](./backup.mdx) - [ ] The new release's install kit, verified - [ ] A maintenance window. The application is briefly unavailable while pods roll ## Rules that are not negotiable | Rule | Why | |---|---| | One minor at a time | Skipping a minor is not supported. Each train's migrations assume the previous train ran | | Patches in place | Within a train, upgrade directly | | Same order as install: infra, then data, then platform | Each chart depends on the previous one | | Apply CRDs by hand first | Helm never upgrades CRDs. See step 1 | | Back up first | The migration hook changes your schema, and `helm rollback` does not undo that | ## 1. Apply the new CRDs **Do this before anything else.** Helm installs a chart's CRDs on first install and ignores them from then on. An upgrade that ships a new CRD schema fails with a server-side-apply error, or worse, silently rejects fields it does not know. ```bash cd $B/charts tar xzf gen0sec-infra-*.tgz kubectl apply --server-side --force-conflicts \ -f gen0sec-infra/charts/postgres-operator/crds/ kubectl apply --server-side --force-conflicts \ -f gen0sec-infra/charts/strimzi-kafka-operator/crds/ ``` The output is similar to this: ``` customresourcedefinition.apiextensions.k8s.io/postgresqls.acid.zalan.do serverside-applied ``` :::note[This is not hypothetical] The Postgres operator 1.15 to 2.0 change adds PostgreSQL 18 to the `postgresqls` schema. Without the CRD apply, the `postgresql/core` resource is rejected outright and the database never starts. ::: ## 2. Get the new artifacts Follow [Prepare the artifacts](./../04-prepare-artifacts.mdx) for the new release. Both methods work the same as on a first install, with one addition for the offline bundle. **Offline bundle only.** Mirroring is additive: the new tags land beside the old ones. Your running install keeps working until you upgrade, and a rollback still has its images. Do not reuse the previous bundle's directory. The bundler refuses to write into an existing one, and each bundle's values files are targeted at the version they shipped with. ## 3. Upgrade infra Run every command from the **new** bundle directory, so `charts/` and `values/` are the new release's. ```bash helm upgrade g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml ``` Wait for the operators to finish rolling before continuing. ```bash kubectl -n gen0sec-system rollout status deploy/g0s-infra-postgres-operator kubectl -n gen0sec-system rollout status deploy/strimzi-cluster-operator kubectl -n gen0sec-system rollout status deploy/g0s-infra-synapse-operator kubectl -n gen0sec-system rollout status deploy/dragonfly kubectl -n gen0sec-system rollout status \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" ``` ## 4. Upgrade data ```bash helm upgrade g0s-data $B/charts/gen0sec-data-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml ``` ```bash kubectl -n gen0sec-system wait --for=condition=Ready kafka/core --timeout=15m kubectl -n gen0sec-system wait \ --for=jsonpath='{.status.PostgresClusterStatus}'=Running postgresql/core --timeout=15m ``` ## 5. Upgrade the platform The schema migration runs first, as a **pre-upgrade** hook, before any new pod rolls. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ --set global.imageTag=$VERSION \ --timeout 20m ``` **If the migration fails, Helm aborts and the old application keeps running.** That is the designed behavior and it is why the hook runs before the rollout. ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` Fix the cause, then re-run the same command. See [Migration failures](./../troubleshoot/symptoms.mdx#migration-failures). ## 6. Verify ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` Run the smoke test after every upgrade, not just after installs. It is the only check that proves the platform still serves an agent rather than merely starting. Full checklist: [Verify the install](./../08-verify.mdx). ## If you changed values along the way `helm upgrade` uses the values files you pass, not the ones from last time. Anything you set with `--set` on the previous install must be set again, or it reverts to the chart default. Check what the running release actually has: ```bash helm get values g0s -n gen0sec ``` Keep your overrides in a file rather than in shell history. See [Reference configurations](./../configure/reference-configs.mdx). ## If it fails | Symptom | Start here | |---|---| | CRD schema error on `helm upgrade` | You skipped step 1 | | `Unsupported value` from the Postgres operator | You skipped step 1 | | Migration job fails | [Migration failures](./../troubleshoot/symptoms.mdx#migration-failures) | | Pods roll but the smoke test fails | [Roll back](./rollback.mdx) | | `ImagePullBackOff` on new tags | The new images were not mirrored. Offline bundle method, step 2 | ## Next [Roll back](./rollback.mdx). --- # Prepare the artifacts Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/prepare-artifacts import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; # Prepare the artifacts ## Before you start - [ ] [Requirements](./02-requirements.mdx) met, tools installed, `yq` version checked - [ ] [Network and connectivity](./03-network.mdx) configured by your network team - [ ] Your [registry token](./02-requirements.mdx#the-registry-token), created on the dashboard under **Repository Access** - [ ] Your [Gen0Sec API key with the required scopes](./02-requirements.mdx#your-gen0sec-api-key) ## 0. Set your keys Create your token files. ```bash mkdir -p ~/.gen0sec cd ~/.gen0sec/ && touch .registry-token .download-api-key ``` Then open up an editor of your choice to fill each file with their respective content. Then run ```bash export REGISTRY_TOKEN=$(cat ~/.gen0sec/.registry-token) export G0S_DOWNLOAD_API_KEY=$(cat ~/.gen0sec/.download-api-key) ``` :::tip[Manage your keys securely] By storing keys in files, you make sure they don't get etched into your shell history. If you have history turned off or you know what you're doing, you can also go with ```bash export REGISTRY_TOKEN="your-registry-token" export G0S_DOWNLOAD_API_KEY="your-gen0sec-api-key" ``` ::: ## 1. Set the release Pin the release explicitly. A pinned tag installs the same bytes in six months; a resolved one does not. You can find available release versions on the [Releases page](https://releases.gen0sec.com/cerebellum/). ```bash export TAG=v0.1.1 export VERSION=${TAG#v} ``` To always use the latest version instead: ```bash export TAG=$(curl -fsSL https://releases.gen0sec.com/api/repos/cerebellum/latest | jq -r .tag_name) export VERSION=${TAG#v} ``` :::note[Release candidates are not published here] Only stable releases are mirrored, so an `-rc.N` tag returns 404 no matter how it is spelled. The [release index](https://releases.gen0sec.com/cerebellum/) is the list of what exists. ::: ## 2. Download the install kit ```bash curl -fsSLO https://releases.gen0sec.com/cerebellum/$TAG/cerebellum-install-kit.tar.gz curl -fsSLO https://releases.gen0sec.com/cerebellum/$TAG/cerebellum-install-kit.tar.gz.cosign.bundle ``` ## 3. Verify the signature Run this before you unpack anything. ```bash cosign verify-blob \ --bundle cerebellum-install-kit.tar.gz.cosign.bundle \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/release\.yaml@refs/tags/' \ cerebellum-install-kit.tar.gz ``` The output is similar to this: ``` Verified OK ``` Anything else means stop. Do not unpack the archive. :::note[Why both certificate flags are required] `cosign` refuses to verify without an expected issuer and an expected identity, so a shortened command fails rather than passing weakly. The identity is pinned to one workflow on a tag reference. A broader pattern would accept a signature produced by any workflow in our organization. The signature covers the file's contents, so it proves the bytes are ours regardless of how they reached you. ::: ## 4. Unpack it ```bash tar -xzf cerebellum-install-kit.tar.gz export KIT=$PWD/cerebellum-install-kit-$VERSION cd $KIT ``` The kit contains the bundler, the release manifest, the helper scripts used later in this install, and the sizing overlays. It does not contain images or charts. ## 5. Log in to our registry ```bash export REGISTRY=registry.gen0sec.com export REGISTRY_USER=any printf '%s' "$REGISTRY_TOKEN" | helm registry login $REGISTRY -u "$REGISTRY_USER" --password-stdin printf '%s' "$REGISTRY_TOKEN" | skopeo login $REGISTRY -u "$REGISTRY_USER" --password-stdin ``` Both are needed. `helm` and `skopeo` keep separate credential stores, and the next step uses each of them. `cosign` is a third store again, and it needs no login of its own. The bundler lends it the credential `skopeo` holds. ## 6. Get the charts Pull the charts and their values files. Images are not pulled: your cluster pulls each one from `registry.gen0sec.com` when it needs it. :::note[Confirm your nodes can pull from us before you go further] Our registry challenges pulls that send no `User-Agent`, which on some distributions is what the kubelet's pull path does. This step succeeds either way, because it runs from your workstation. The failure appears at step 3 of [Install](./05-install.mdx), as a 403 on every image. Check it now: [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: ```bash ./bundler.sh assemble-charts release-manifest.yaml export B=$PWD/gen0sec-cerebellum-multiarch-$VERSION ``` The output ends with a per-chart digest confirmation. :::note[What "digest-verified" means here] `assemble-charts` checks two digests per chart: the registry's manifest against the digest this release pins, and the chart archive against the layer named in that manifest. What you install provably came out of the chart this release published. ::: Each chart carries its own `values-onprem.yaml`, extracted into `$B/values/`. That file is what points the install at a registry. The default `values.yaml` leaves the registry blank on purpose, so installing without `-f` will not resolve images. ### 6a. Build the bundle on your workstation ```bash ./bundler.sh create-package release-manifest.yaml ``` This pulls every image and chart the release pins by digest, verifies every signature, then packages the result as `gen0sec-cerebellum-multiarch-$VERSION.tar.gz`. It carries both `linux/amd64` and `linux/arm64`, so it installs on any node in your cluster. Signature verification happens here, on the connected side, because it needs the public transparency log. Everything downstream is protected by checksums instead. :::note[Before you use --arch to halve the transfer] `--arch amd64` or `--arch arm64` extracts one manifest out of each multi-architecture index. That changes every image digest by definition. Per-architecture signatures still verify, because each one is signed individually. **SBOM attestations do not**, because they are attached to the index, and the index is what a single-architecture bundle leaves behind. The bundler warns at the time and annotates the bundle. If your acceptance process checks attestations, stay on the default. If your nodes are mixed, or you are unsure what they run, stay on the default: a missing architecture fails when the kubelet tries to pull, on an isolated cluster, after the transfer. A single-architecture bundle also renames the tarball and the directory inside it, so substitute the name in every command that follows. ::: ### 6b. Move it across In this step you take your bundle onto an intermediary machine inside the network where your internal registry and the target cluster is located you'll install the bundle onto. After transferring the bundle, you can clean up all artifacts it has created. eg. ```bash scp gen0sec-cerebellum-multiarch-$VERSION.tar.gz :~/ # `create-package` ran inside the kit directory, so the bundle is sitting in it. Copying the kit first # would transfer tens of gigabytes twice. `cleanup` removes only what the bundler created. ./bundler.sh cleanup cd .. && scp -r cerebellum-install-kit-$VERSION :~/ # You can even remove the installation kit if you don't intend to keep it. # cd $KIT/.. && rm -rf cerebellum-install-kit.tar.gz $KIT ``` ### 6c. Load it into your registry Log in to your intermediary machine and use the bundler to load the images into your internal registry. ```bash export REGISTRY=registry.internal.example export REGISTRY_USER=your-registry-user export REGISTRY_TOKEN=$(cat path/to/your-registry-token) # Carried across because this is a different machine and Install step 2 needs # it. Unset, the secret it creates is empty and every relay fetch 401s. export G0S_DOWNLOAD_API_KEY=$(cat path/to/your-gen0sec-api-key) export KIT=~/cerebellum-install-kit-$VERSION export VERSION=$(cat $KIT/release-manifest.yaml | yq ".spec.version") tar -xzf gen0sec-cerebellum-multiarch-$VERSION.tar.gz export B=~/gen0sec-cerebellum-multiarch-$VERSION cd $B printf '%s' "$REGISTRY_TOKEN" | skopeo login $REGISTRY -u "$REGISTRY_USER" --password-stdin ./bundler.sh apply release-manifest.yaml -r $REGISTRY ``` `apply` pushes every bundled image into your registry, then rewrites the bundled values files to point at it. The charts are not pushed: you install them from `$B/charts/*.tgz`. The bundle is checked against its own `SHA256SUMS` before anything is pushed, so a corrupted transfer fails here rather than at install time. :::note[For a plain-HTTP registry] Set `MIRROR_TLS_VERIFY=false` before `apply`. This affects the push only, and it is the one place in this install where TLS verification can be disabled. Prefer a certificate from your internal CA. ::: ## Checkpoint Whichever method you used, you should now have the following: | Variable | Points at | |---|---| | `$KIT` | The unpacked install kit | | `$B` | A directory containing `charts/` and `values/` | | `$REGISTRY` | The registry your cluster will pull from | | `$REGISTRY_USER` | The username for that registry (`any` for ours) | | `$REGISTRY_TOKEN` | The password or token for that registry | | `$G0S_DOWNLOAD_API_KEY` | Your Gen0Sec API key, set in step 0. [Install](./05-install.mdx#your-gen0sec-api-key) step 2 puts it in a secret | You are logged in to `$REGISTRY`. Every page after this one assumes exactly that and nothing else, which is why nothing after this point branches. --- # Publish the endpoints Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/publish # Publish the endpoints The ingress controller is installed and running. However it's not configured to serve anything by default. That is deliberate, because creating those routes makes the APIs reachable from wherever the controller is reachable, so it is your decision, not ours. The data plane's LoadBalancer Service already has an external address at this point. Until you complete this page it serves nothing, because no rule names a host. :::note[Ingress and TLS are yours to configure] The platform ships the routing rules. Terminating TLS, issuing the certificate and exposing the LoadBalancer are your cluster's responsibility. For the configuration options, see the [Kubernetes ingress reference](https://docs.gen0sec.com/docs/client/kubernetes-ingress). ::: ## Before you start - [ ] [Install](./05-install.mdx) complete, all pods running - [ ] A DNS name you control, resolving to your ingress controller's address - [ ] A LoadBalancer address for the ingress data plane, or a Service type your cluster can expose - [ ] A TLS certificate for that name, or cert-manager configured ## 1. Choose the address One address serves the dashboard and every agent-facing API. Requests are routed by path prefix. ```bash export ENTRYPOINT_HOST=cerebellum.example.internal ``` **The host is required.** A rule with no host is silently ignored by the ingress controller: the Ingress object is created, no event is logged, and nothing serves traffic. **Use the address people will actually type.** The dashboard derives its sign-in URL from this host, and sign-in fails if the two do not match. ## 2. Publish The kit ships the overlay. Layer it on top of the values file you have been using. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` :::warning[Re-pass every values file you installed with] `helm upgrade` applies only the files you pass it. Anything layered at install time and omitted here reverts to the chart default. On the [Quickstart](./quickstart.mdx) you also passed `-f $KIT/examples/single-node/platform-values.yaml`, so include it in this upgrade too (before the ingress overlay), or every service scales back to its production replica count: ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-node/platform-values.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` ::: :::note[Quote any --set argument containing brackets] `zsh` treats `[0]` as a glob and fails before `helm` runs. Single-quote the whole argument: `--set 'services.ui.ingress.hosts[0].host=...'`. ::: ## 3. Verify the routes exist ```bash kubectl -n gen0sec get ingress ``` The output is similar to this: ``` NAME CLASS HOSTS ADDRESS PORTS AGE g0s-platform synapse cerebellum.example.internal 10.0.0.42 80 30s ``` An empty `HOSTS` column means the host was not applied, and nothing will serve traffic. Then check from outside the cluster. The dashboard is published at `/` and returns a redirect to its sign-in page, so follow it with `-L`: ```bash curl -fsSL -o /dev/null -w '%{http_code}\n' https://$ENTRYPOINT_HOST/ ``` The output is similar to this: ``` 200 ``` :::note[There is no health endpoint on the ingress] `/status/healthz` is served per pod inside the cluster and is deliberately **not** published through the ingress. A request to it from outside falls through to the dashboard catch-all and returns `404`, so do not use it as an external check. The dashboard root above confirms reachability; the [smoke test](./08-verify.mdx#4-the-platform-serves-an-agent) confirms the platform serves agents. ::: ## Find the address to point DNS at The ingress data plane is a `LoadBalancer` Service. ```bash kubectl -n gen0sec-system get svc g0s-infra-synapse -o wide ``` Point your DNS name at the external address. If `EXTERNAL-IP` stays ``, your cluster has no LoadBalancer implementation: change the Service type to something it can expose. It cannot stay cluster-internal. ## The route table You do not configure these. The overlay sets them, read off each service's own router. They are here because they are the answer to "a request landed on the wrong backend", and because a missing route fails quietly rather than loudly. | Path prefix | Served by | |---|---| | `/` | Dashboard (catch-all, shortest prefix, so it is last resort) | | `/v1/indicators`, `/v1/models`, `/v1/geoip`, `/v1/ids-rules` | Relay | | `/v1/penalty-model`, `/v1/traffic-model`, `/v1/ja4-classifier`, `/v1/flow-classifier`, `/v1/jepa-embedder` | Relay, per-model aliases | | `/v1/identity`, `/v1/policy-edges`, `/v1/fleet-bans` | Relay, served locally | | `/v1/authcheck`, `/v1/auth` | Authentication | | `/v1/events`, `/v1/agents`, `/v1/log` | Agent ingest | | `/v1/agents/events` | Configuration push (longer prefix than `/v1/agents`, so it wins) | | `/v1/config` | Configuration | | `/v1/telemetry`, `/v1/logs` | Telemetry | | `/v1/signal`, `/v1/rules` | Signal | | `/v1/ban-ledger` | Ban ledger | | `/v1/threat` | Threat feeds | | `/v1/service-graph` | Service graph | Routing is by **longest matching prefix**, because one address serves every backend. The configuration push stream on `/v1/config` is long-lived, so the chart sets read, write and idle timeouts of 3600 seconds on that route. Do not lower them: agents fall back to polling with no push, which looks like a slow platform rather than a timeout. :::note[Why a missing route is worse than a broken one] The dashboard owns `/`, so any path absent from this table falls through to it and gets a redirect or HTML. HTTP has no way for the caller to tell that apart from a real answer, so the failures are silent: | Missing route | What the agent actually does | |---|---| | `/v1/authcheck` | Receives HTML, cannot distinguish a bad key from a routing mistake, and continues unverified | | `/v1/agents/events` | Logs `SSE connect returned HTTP 404`, then falls back to polling. Looks like a slow platform, not a routing bug | | `/v1/ban-ledger` | Reads the redirect as success. The contribution is lost with no error anywhere | This is why the overlay exists rather than a list of prefixes for you to type. ::: :::note[/v1/ids-rules goes to the relay, not to the in-cluster rules service] IDS rules are relayed from gen0sec. The in-cluster rules service builds rules from a local ingest pipeline that this edition does not run. Route this prefix there and you reach a service with no rules, and the relay is never used. ::: ### If TLS terminates upstream If a load balancer, CDN or tunnel terminates TLS in front of the ingress, the chart cannot infer the external scheme and guesses `http` while the browser is on `https`. The dashboard then builds its sign-in URL with the wrong scheme and rejects every sign-in with `Invalid origin`. Set it explicitly: ```bash --set externalScheme=https ``` Leave it unset when TLS terminates at the ingress, because then the chart reads it from the ingress TLS block. ## The dashboard The dashboard is enabled by default and is published at the root of your address. Sign-in is email and password, held in your own Postgres, so nothing outside the cluster is involved. Its secret, `g0s-db-ui`, was created in [Install](./05-install.mdx) step 6 with all the others. Re-running that script keeps the existing signing secret, so sessions and invite links stay valid. To leave the dashboard out: ```bash --set services.ui.enabled=false ``` ## If it fails | Symptom | Cause | |---|---| | `kubectl get ingress` shows an empty `HOSTS` column | `entrypointHost` was not set. The controller ignores host-less rules silently | | `curl` times out | DNS does not resolve to your ingress controller, or a firewall blocks the port | | The dashboard loads but sign-in redirects to the wrong address | The sign-in URL is derived from `entrypointHost`. It must be the address people type | | A request reaches the wrong service | Longest-prefix routing. Check the `/v1/agents` overlap above | --- # Quickstart on one node Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/quickstart # Quickstart on one node :::danger[Not for production] This runs every component at a single replica on a single node. There is no redundancy anywhere: **losing the node loses the data.** Postgres has no standby, Kafka has one broker at replication factor 1, and the object store has no erasure coding. Use it for evaluation, training and upgrade rehearsals. For anything you depend on, follow [Sizing a production install](./02-requirements.mdx#sizing-a-production-install), which runs on three nodes. ::: ## What you get A working instance you can sign in to, on one machine, without planning a cluster. This is not a separate procedure. It is the same install as the rest of this documentation, with one extra values file per chart. ## What you need | | | |---|---| | One node | 12 vCPU, 32 GB RAM, 200 GB disk | | Kubernetes | k3s | | StorageClass | Any default block StorageClass. k3s installs `local-path` as the default, which is enough | | Tools | The same list as [Requirements](./02-requirements.mdx#tools-on-your-workstation) | | Credentials | The same two as [Requirements](./02-requirements.mdx#credentials-from-gen0sec). The relay is required here too | ## How it differs from production The kit ships an overlay per chart that reduces everything to one replica. | | Production | This quickstart | |---|---|---| | Object store | 3 replicas, erasure coded, distributed | 1 replica, **standalone mode**, and a Deployment rather than a StatefulSet | | Postgres | 3 instances | 1 instance, no replica pooler | | Kafka | 3 brokers, replication factor 3 | 1 broker, every factor 1 | | Application services | 2 or 3 replicas each | 1 each | | Cache | Chart default | 256 MB, one thread | | Storage | Your block StorageClass, 500 GiB per node minimum | The cluster's default StorageClass, about 65 GiB of volumes | | Survives losing a node | Yes | No | :::note[The connection pooler stays on] Only the **replica** pooler is disabled, because there is no replica to pool. Every service still connects through `core-pooler`, exactly as in production, so the secrets that `make-db-secrets.sh` composes are the same shape here. ::: :::note[Two things that surprise people] The object store stops being a StatefulSet and becomes a Deployment of the same name. That is why the readiness checks in [Install](./05-install.mdx) resolve the kind instead of naming it. Volume claims from a previous multi-replica install are not reused. If you are converting an existing install, delete them first. ::: ## Run it Follow the spine exactly as written, starting at [Prepare the artifacts](./04-prepare-artifacts.mdx). Three commands change. Nothing else does. | Spine step | On this install | |---|---| | [Prepare the artifacts](./04-prepare-artifacts.mdx) | Unchanged. Either method works, and the registry method is fewer steps | | [Install](./05-install.mdx) steps 1 and 2 | Unchanged. Same namespaces, same three secrets | | [Install](./05-install.mdx) steps 3, 4 and 7 | Add the overlay. See below | | [Install](./05-install.mdx) steps 5 and 6 | Unchanged. You still create the bucket, and you still run the secret script | | [Publish the endpoints](./06-publish.mdx) | Unchanged. See the note on the address below | | [First sign-in](./07-first-signin.mdx) and [Verify the install](./08-verify.mdx) | Unchanged | Pass the overlay **after** the production values file, so it wins. ```bash helm install g0s-infra $B/charts/gen0sec-infra-*.tgz -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml \ -f $KIT/examples/single-node/infra-values.yaml ``` ```bash helm install g0s-data $B/charts/gen0sec-data-*.tgz -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml \ -f $KIT/examples/single-node/data-values.yaml ``` ```bash helm install g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-node/platform-values.yaml \ --set global.imageTag=$VERSION \ --timeout 20m ``` The wait gates after each one are the same as on [Install](./05-install.mdx). Read the files before you use them: each overlay carries comments explaining what it trades away. :::caution[Your cluster needs a default StorageClass] The overlays leave the storage class empty in all three places — `rustfs.storageclass.name` in `infra-values.yaml`, `postgres.volume.storageClass` and `kafka.storage.class` in `data-values.yaml` — which means "use the cluster's default", the same as the production values file. On k3s that resolves to `local-path` and there is nothing to do. If your cluster has **no** default, every PVC stays `Pending`. `helm install` still reports success, and the only symptom is the wait gate above timing out without naming a cause. Check first: ```bash kubectl get storageclass # exactly one row must be marked (default) ``` To pin a class explicitly instead of relying on the default, set those three keys, or layer a fourth `-f` file last. ::: ## The address [Publish the endpoints](./06-publish.mdx) is unchanged, including the requirement for a host. On k3s the bundled LoadBalancer implementation gives the ingress data plane the node's own address, so there is nothing further to provision. For an evaluation, a name in `/etc/hosts` on the machine you browse from is enough. It has to be the name you actually type in the browser, because the dashboard builds its sign-in URL from it. :::note[About TLS terminatotion] In case your TLS termination does not happen on a higher level, configure Synapse Ingress to work with cert-manager (see the [Kubernetes ingress reference](https://docs.gen0sec.com/docs/client/kubernetes-ingress)). The shipped ingress overlay sets `externalScheme: https` and configures no certificate at the ingress. A single-node evaluation usually has none. Either provision a certificate, or serve plain HTTP: set `externalScheme` to `http` in the ingress overlay and use an `http://` base URL in the sign-in and verify steps below. ::: ## First sign-in Nothing is different here, and it is worth knowing before you get there: the dashboard makes you enroll multi-factor authentication and replace the seeded password before anything opens. See [First sign-in](./07-first-signin.mdx). ## Verify Run the same checks as a production install, from [Verify the install](./08-verify.mdx). The smoke test is the one that matters, because it proves the platform serves an agent rather than merely starting: ```bash # Save the agent key you minted at first sign-in, then read it back below. mkdir -p ~/.gen0sec printf '%s' '' > ~/.gen0sec/agent-key export GEN0SEC_BASE_URL=https://$ENTRYPOINT_HOST export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) $KIT/scripts/verify-deployment.sh ``` A single-node install that passes it is a good rehearsal for a real one. That is most of the value of building this. ## Tearing it down ```bash $KIT/scripts/uninstall.sh --yes ``` It asks you to type the cluster context name back before deleting anything, so it will not run unattended. See [Uninstall](./operate/uninstall.mdx). ## When you are ready for production [Sizing a production install](./02-requirements.mdx#sizing-a-production-install) has the node counts, per-node capacity and storage figures. There is no in-place conversion from this install to a production one: build the production cluster separately. --- # Bundler CLI Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/reference/bundler # Bundler CLI `bundler.sh` ships in the install kit. It builds an offline bundle on a connected host and loads it into your registry on a host that has no internet access. Its only input is the `release-manifest.yaml` inside the kit. No access to our source is needed, so you can build your own bundle from any published release. For the procedure rather than the reference, see [Prepare the artifacts](./../04-prepare-artifacts.mdx). ## Which commands need internet access This is the table an isolated site actually needs. The pipeline is split on internet access, not on environment. | Command | Internet | Talks to | |---|---|---| | `validate` | **Required** | Parses the manifest, resolves tooling, then fetches one image manifest to confirm registry access | | `assemble` | **Required** | Source registry: images and charts | | `assemble-charts` | **Required** | Source registry: charts only | | `assemble-images` | **Required** | Source registry: images only | | `verify-assembly` | **Required** | Source registry for signature artifacts, plus Sigstore trust material | | `create-package` | **Required** | Inherits `assemble` and `verify-assembly` | | `package` | None | Local `tar` | | `verify-package` | None | Local extract and checksum comparison | | `mirror` | None | Destination registry, over your local network | | `retarget` | None | Local files | | `apply` | None | Destination registry. This is `mirror` plus `retarget` | | `cleanup`, `version`, `help` | None | Local | **Split the pipeline at the tarball, not earlier.** `create-package` cannot run on an isolated host: it fails in preflight with no route to the source registry. ## Commands | Command | What it does | |---|---| | `validate` | Preflight. The manifest exists and its schema is sane, every image and chart is digest-pinned, the tooling resolves, and the source registry is readable. Fetches one image manifest, so a bad host, missing login, TLS mismatch or unpublished digest fails here rather than partway through a multi-gigabyte download | | `assemble` | Creates the bundle directory, pulls every image and chart, extracts each chart's production values file, copies in the manifest and itself. Fails if the directory exists | | `assemble-charts` | The charts half. **What the registry install method uses**, since the cluster pulls its own images | | `assemble-images` | The images half. Separable so a failed multi-hour pull can be retried without re-fetching charts | | `verify-assembly` | Keyless signature verification of every pinned image and chart, then writes `SHA256SUMS` and prints the bundle **fingerprint**. Reports every failure and refuses to continue on any | | `package` | Tars the bundle directory into a tarball **beside** it, never inside | | `verify-package` | Unpacks to a temporary directory and re-checks `SHA256SUMS` | | `create-package` | `assemble`, `verify-assembly`, `package`, `verify-package`, in order | | `mirror` | Verifies the bundle against `SHA256SUMS`, then pushes every image to your registry | | `retarget` | Rewrites the registry host in the bundle's own values files, in place, and updates the affected `SHA256SUMS` entries so the bundle stays self-consistent | | `apply` | `mirror` then `retarget`. The usual customer step | | `cleanup` | Removes every directory and archive the bundler created | | `version` | The bundler's version, stamped at release | | `help` | Usage | Every command runs `validate` first, so any of them is safe to invoke on its own. ## Options | Option | Effect | |---|---| | `--arch amd64` or `--arch arm64` | Single-architecture bundle, roughly half the size. **Loses SBOM attestations.** See below | | `-o ` | A different tarball name | | `-r ` | Destination registry for `mirror`, `retarget` and `apply` | | `--dangerously-skip-cosign-verify` | Drops signature verification and its tooling requirement. The bundle is annotated as unverified | ### The --arch trade-off A single-architecture bundle extracts one manifest out of each multi-architecture index, which changes every image digest by definition. | | Default | With `--arch` | |---|---|---| | Image signatures | Verify | **Verify.** Each per-architecture manifest is signed individually | | SBOM attestations | Verify | **Lost.** They hang off the index, which is what is left behind | | Digests match the manifest | Yes | No | | Tarball and directory name | `multiarch` | Named after the architecture, so substitute it everywhere | If your acceptance process checks attestations, or your nodes are mixed, or you are not certain what they run: take the default. A missing architecture fails when the kubelet tries to pull, on an isolated cluster, after the transfer. ## Environment variables | Variable | Purpose | |---|---| | `MIKEFARAH_YQ_PATH` | Path to the Go `yq`, when the one on `PATH` is the Python one | | `BUNDLE_TLS_VERIFY=false` | Disables TLS verification on the **pull** side | | `MIRROR_TLS_VERIFY=false` | Disables TLS verification on the **push** side, for a plain-HTTP registry | The two TLS variables are the only places in the whole install where verification can be disabled. Prefer a certificate from your internal CA. See [Hardening](./../security/hardening.mdx). ## Credentials, and the three stores Three tools, three credential stores. You log in to two of them. | Tool | Store | Login needed | |---|---|---| | `skopeo` | The containers store | Yes, both source and destination | | `helm` | `HELM_REGISTRY_CONFIG` | Yes, source only | | `cosign` | The Docker config | **No.** The bundler lends it the one `skopeo` holds | If you see the bundler say it is reusing skopeo's credential for cosign, that is the expected path and not a warning. ## Behavior worth knowing | | | |---|---| | `assemble` refuses to write into an existing directory | So a stale bundle cannot be silently mixed with a new one | | `retarget` takes no file arguments | Rewriting arbitrary paths handed in on a command line is not something an isolated install should be able to ask for | | `retarget` verifies integrity **before** editing, never after | That order is what stops a checksum update from re-blessing a bundle that arrived altered | | `retarget` is idempotent | Running `apply` twice is safe | | `mirror` runs before `retarget` inside `apply` | So a registry failure leaves your values files untouched | | Mirroring is additive | New tags land beside old ones, so a running install keeps working and a rollback still has its images | | An egress proxy is fine for the bundler | Provided the host can reach the registry and Sigstore. Unlike the relay, these are ordinary CLI tools that honour proxy variables | ## Bundle contents ``` gen0sec-cerebellum--/ ├── oci/ every image, as an OCI layout ├── charts/ installable chart archives ├── values/ each chart's production values, retargeted by `apply` ├── release-manifest.yaml the digests this bundle was built from ├── bundler.sh a copy └── SHA256SUMS over every file above ``` `assemble` extracts each chart's values file as `values/-values-onprem.yaml`. There is no bare `values-onprem.yaml` in the bundle root, which is why the install commands name the file in full. The bundle does **not** carry the install kit's `scripts/` or `examples/`. Those cross the air gap inside the kit directory. See [Install kit contents](./kit.mdx). --- # Release notes Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/reference/changelog # Release notes ## v0.1.1 A security patch for the dashboard. It fixes [CVE-2026-94545](https://nvd.nist.gov/vuln/detail/CVE-2026-94545) (medium, CVSS 4.0 score 5.3), in which Satori, the HTML-to-SVG library Next.js bundles for `next/og`, did not escape certain values in the SVG it generates. The dashboard does not render images through `next/og`, but the vulnerable copy shipped in its image and scanners flag it. The fix updates Next.js from 15.5.24 to 16.3.7, and React from 19.2.3 to 19.3.0. It upgrades in place from v0.1.0, with no migration and no values changes. Nothing else changed: the install kit's overlays are identical to v0.1.0's, and the [known limitations](./../troubleshoot/known-limitations.mdx) of v0.1.0 still apply. ## v0.1.0 First on-premises release. ### Installing Two methods, both documented on one procedure that branches once: [install from our registry](./../04-prepare-artifacts.mdx), or [create an offline bundle](./../04-prepare-artifacts.mdx) and mirror it into your own. ### Requires | | | |---|---| | Kubernetes | 1.23 or later | | Nodes | 3 for production, 16 vCPU and 32 GiB each recommended | | Storage | [sized to your retention](./../02-requirements.mdx#sizing-storage-for-your-retention) | | Cluster egress | `api.gen0sec.com:443`, required | ### Credentials you need **Two**, and the second is easy to miss. Both are self-service from the dashboard: 1. A registry token, for pulling charts and images. Created under **Repository Access**. The install kit itself is public and needs no credentials. 2. **Your Gen0Sec API key**, which the relay uses to fetch data from us. Without it the whole fleet receives empty artifacts with no error. See [Credentials from gen0sec](./../02-requirements.mdx#credentials-from-gen0sec). ### Not in this release | | | |---|---| | Zero-egress deployment | [Known limitations](./../troubleshoot/known-limitations.mdx#no-zero-egress-deployment) | | Explicit egress proxy for the relay | [Known limitations](./../troubleshoot/known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy) | | Single sign-on of any kind | Email and password only. Microsoft Entra ID is in development | | Shipped dashboards and alert rules | [Monitor](./../operate/monitor.mdx) lists what to watch meanwhile | | The threat service | Not shipped at this release | --- ## How we version | | | |---|---| | Scheme | Semantic versioning. Each minor is a release train | | Supported | The latest two minor trains | | Upgrades | Sequential minors, patches in place. **Skipping a minor is not supported** | | Release candidates | Not published to `releases.gen0sec.com`. Only stable releases are mirrored | Full policy: [Component versions](./../configure/components.mdx#support-policy). ## Finding a release ```bash curl -fsSL https://releases.gen0sec.com/api/repos/cerebellum/latest | jq -r .tag_name ``` Returns the latest full release. **Pin the tag explicitly for an install you want to reproduce**, so a rebuild in six months fetches the same bytes. ## What every release publishes | Artifact | Where | |---|---| | Install kit and its signature | `https://releases.gen0sec.com/cerebellum//` | | Images and charts | The registry, each pinned to a digest in the kit's release manifest | | Signatures and SBOM attestations | Alongside each image and chart, as OCI referrers | See [Supply chain](./../security/supply-chain.mdx). --- # Install kit contents Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/reference/kit # Install kit contents ## What you download Two files, published per release at `https://releases.gen0sec.com/cerebellum//`. | File | What | |---|---| | `cerebellum-install-kit.tar.gz` | The kit | | `cerebellum-install-kit.tar.gz.cosign.bundle` | Its detached signature | Verify before unpacking. See [Prepare the artifacts](./../04-prepare-artifacts.mdx#3-verify-the-signature). ## Layout ``` cerebellum-install-kit-/ ├── bundler.sh builds and loads the offline bundle ├── release-manifest.yaml every image and chart, pinned to a digest ├── scripts/ │ ├── preflight.sh cluster checks the charts assume but do not create │ ├── make-db-secrets.sh composes the ~20 application secrets │ ├── uninstall.sh the three releases, finalizers, namespaces │ └── verify-deployment.sh external smoke test, read-only ├── examples/ │ ├── single-node/ three overlays, one replica each │ └── single-entrypoint/ one address for the dashboard and every API ├── offline-data/ │ └── README.txt └── SHA256SUMS over every file above ``` **No images and no charts.** Those come from the registry, or from the offline bundle. ## The scripts | Script | Run it | Documented in | |---|---|---| | `preflight.sh` | Before installing | [Install step 1](./../05-install.mdx#1-check-the-cluster) | | `make-db-secrets.sh` | After the databases are ready, before the platform | [Install step 6](./../05-install.mdx#6-create-the-application-secrets) | | `verify-deployment.sh` | After every install and every upgrade | [Verify the install](./../08-verify.mdx#4-the-platform-serves-an-agent) | | `uninstall.sh` | When removing the installation | [Uninstall](./../operate/uninstall.mdx) | `verify-deployment.sh` reads two environment variables: ```bash export GEN0SEC_BASE_URL=https://cerebellum.example.internal export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) ``` ## The examples Everything under `examples/` is a starting point you are meant to read and edit. Each file carries comments marking which values are yours. | Directory | Purpose | |---|---| | `single-node/` | Three overlays that reduce every component to one node. [Quickstart](./../quickstart.mdx) | | `single-entrypoint/` | One address for the dashboard and every agent-facing API. [Publish the endpoints](./../06-publish.mdx) | ## Verifying the kit's own integrity ```bash cd $KIT && sha256sum -c SHA256SUMS ``` `SHA256SUMS` covers the kit **as delivered**. Adding files makes it no longer match, which is expected once you start using it. ## What the offline bundle method has to transfer Two things, and the order matters. | Transfer | Contains | |---|---| | The bundle tarball | Images, charts, values, the manifest, a copy of the bundler | | The kit directory | `scripts/`, `examples/`, and the manifest | The bundle carries its own copy of `bundler.sh`, so loading it needs no tooling from the kit. **The kit still has to cross**, because the install uses its scripts and its examples. Copy the bundle first, then clean up, then the kit. `create-package` runs inside the kit directory, so the bundle is sitting in it, and copying the kit first transfers tens of gigabytes twice. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). Both archives unpack side by side: ``` ~/gen0sec-cerebellum-multiarch-0.1.1/ the bundle ~/cerebellum-install-kit-0.1.1/ the kit ``` ## The offline-data directory The kit creates `offline-data/` containing a single `README.txt`, and nothing else. It is a placeholder for a data-delivery path that is not part of this release: the four datasets the platform needs are fetched at runtime through the relay, not delivered as files. See [Data relay](./../configure/relay.mdx). If you were told to populate it, check with us first. At this release you should not need to. --- # Requirements Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/requirements # Requirements ## Cluster | | Requirement | |---|---| | Kubernetes | 1.23 or later | | Access | `cluster-admin`, for the install only. See [Privileges and RBAC](./security/privileges.mdx) for what it is used for and what it leaves behind | | StorageClass | A default StorageClass on block storage. **Volume expansion recommended** | | Supported architecture | `linux/amd64` or `linux/arm64`. Mixed-architecture clusters are supported | | LoadBalancer | The ingress data plane is exposed as a `LoadBalancer` type Service by default. See the note below | :::note[The ingress needs to be reachable from outside the cluster] The ingress data plane is a `LoadBalancer` type Service, because nothing it programs can serve traffic if it stays cluster-internal. Your cluster needs a LoadBalancer implementation. If your cluster fronts ingress another way, a NodePort, a hardware balancer or an external proxy, change the Service type to match. ::: Cerebellum brings its own data plane - which contains Postgres, Kafka, object store and cache. You do not connect it to existing instances of any of them. :::info[Maintenance] We've deliberately chosen to use well-known solutions from the software industry in our data layer to ease the burden of maintenance. While Cerebellum ships this data layer by default, to make the setup easier, the data in it is the highest importance to you. So make sure you monitor the dataplane and do backups and maintenances regularly. ::: :::note[Container runtime, for the registry install method only] Our registry challenges pulls that carry no `User-Agent`, and on some distributions the kubelet's pull path sends none. containerd's own user agent passes; the problem is its absence. Either configure one on every node, or use the offline bundle method, which pulls into your own registry instead. Settle this before you choose a method. See [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: ### Kubernetes versions 1.23 is the floor because `autoscaling/v2` is the newest API the charts use, and it reached GA in that release. While anything from 1.23 and up should work, we recommend you use the latest stable version. ## Sizing a production install Production runs on **3 nodes**. Several data plane members each form a three-member quorum, so three is the floor for redundancy, not a suggestion. ### CPU & memory per node | | Resources | |---|---| | **Minimum** | 8 vCPU, 16 GiB | | **Recommended** | 16 vCPU, 32 GiB | ### Storage per node We recommend to start your nodes with `500 GiB` **minimum**. Several of the data plane members use PVCs. While the defaults are kept low - as your platform matures the data will grow and you'll need to expand the storage volumes. ### Sizing storage for your retention The shipped sizes are a starting point, not a capacity plan. They are deliberately modest, so for most production workloads you will be growing them rather than shrinking them. Four keys control everything: | Key | Chart | Default | |---|---|---| | `postgres.volume.size` | `gen0sec-data` | 50 GiB | | `kafka.storage.size` | `gen0sec-data` | 50 GiB | | `rustfs.storageclass.dataStorageSize` | `gen0sec-infra` | 50 GiB | | `rustfs.storageclass.logStorageSize` | `gen0sec-infra` | 10 GiB | Which one to grow depends on what you are retaining: | If you need | Grow | |---|---| | Longer event and telemetry history | `postgres.volume.size`. This is the system of record | | A longer Kafka retention window, or a bigger ingest buffer | `kafka.storage.size` | | More room for datasets and locally-produced artifacts | `rustfs.storageclass.dataStorageSize` | Example for longer retention: ```yaml # gen0sec-data postgres: volume: size: 200Gi kafka: storage: size: 200Gi ``` ```yaml # gen0sec-infra rustfs: storageclass: dataStorageSize: 100Gi ``` :::note[Plan for growth] Volume expansion grows a volume if your StorageClass supports it, and it is straightforward. Nothing shrinks one: reducing a size means recreating the volumes and losing what is on them. Starting at the shipped defaults and growing is therefore the recommended direction. It is the reason for low initial limits. Check that your StorageClass has `allowVolumeExpansion: true` before you install. ::: ### What is not on this page **Throughput.** How many agents, events per second or lookups a given topology sustains is not directly derivable from the charts' parameters. We suggest you scale the platform components and storage sizes according to your topology's specific needs. {/* Four workloads declare no CPU or memory requests: the object store, the Kafka brokers, valkey, and the Postgres operator. They are therefore BestEffort or Burstable and are evicted first under node pressure, which is why the per-node recommendation is driven by limits rather than by the requests total. Raised separately as a chart issue. */} ### Evaluating on one node Use the [Quickstart](./quickstart.mdx). It runs everything at a single replica on one node, for evaluation, training and upgrade rehearsals. ## Tools on your workstation Install these on the machine you run the commands from, not on the cluster. | Tool | Version | Used for | Install | |---|---|---|---| | `helm` | 3.16 or later | Installing the charts, and pulling them from the registry | [helm.sh](https://helm.sh/docs/intro/install/) | | `kubectl` | matching your cluster | Everything cluster-side | [kubernetes.io](https://kubernetes.io/docs/tasks/tools/#kubectl) | | `cosign` | 2 or later | Verifying signatures on the install kit, images and charts | [docs.sigstore.dev](https://docs.sigstore.dev/cosign/system_config/installation/) | | `skopeo` | any | Copying images between registries | [github.com/containers/skopeo](https://github.com/containers/skopeo/blob/main/install.md) | | `yq` | 4 or later, **mikefarah/yq only** | Reading the release manifest | [github.com/mikefarah/yq](https://github.com/mikefarah/yq#install) | | `jq` | any | Reading the release API | [jqlang.github.io](https://jqlang.github.io/jq/download/) | | `aws` or `mc` | any | Creating the object store bucket, one step during install | [aws-cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) or [MinIO client](https://min.io/docs/minio/linux/reference/minio-mc.html) | | `curl`, `tar`, `openssl`, `sha256sum` or `shasum` | any | Downloading, unpacking, generating secrets, checking checksums | Your OS packages | ### Check your yq There are two separate tools that are called `yq`. The Python one (`kislyuk/yq`) cannot parse the expressions the bundler uses, and it fails with a syntax error partway through. ```bash yq --version ``` The output must contain `mikefarah`. If it does not, install the Go version and point the bundler at it: ```bash export MIKEFARAH_YQ_PATH=/path/to/mikefarah-yq ``` ## Credentials from Gen0Sec You need **two**, and they are different things. Missing the second one produces a working-looking install whose entire fleet silently receives nothing. | Credential | Used by | Used for | |---|---|---| | **Registry token** | The bundler, on your workstation | Pulling charts and images from our registry | | **Your Gen0Sec API key** | The relay pod, inside your cluster | Fetching threat intelligence, GeoIP, models and IDS rules from Gen0Sec | Both are self-service from the dashboard. Neither requires contacting us. :::note[The install kit itself needs no credentials] The kit is published openly. Downloading and verifying it, in [Prepare the artifacts](./04-prepare-artifacts.mdx), happens before you use either credential — so if that download fails, the token is not what to check. The usual cause is a tag that is not published; the [release index](https://releases.gen0sec.com/cerebellum/) lists the ones that are. ::: ### The registry token Create it on the dashboard, under **Repository Access**: ``` https://app.gen0sec.com//repositoryAccess ``` Your workspace id is in the address bar once you are signed in, and the same page is linked from **Clients**. The token is shown **once**, at creation — store it before leaving the page, because it cannot be retrieved afterwards and a lost one has to be replaced. **The username is literally `any`.** Wherever a command asks for one, type `any`. Only the token is checked, so the username can be anything at all — `any` is simply what these docs use so the commands are copy-pasteable. ### Your Gen0Sec API key You can create your API key on the [SaaS Dashboard](https://app.gen0sec.com) Scopes required: - `ids_rules:read`, - `indicators:read`, - `ml_models:read` :::note[Why this one is separate from the registry token] Your agents get their API keys from your own dashboard, and those keys exist only in your cluster's database, so they cannot authenticate to Gen0Sec. The API key created in this step is used by your in-cluster relay to authenticate against the Gen0Sec SaaS, so it can fetch data related to threat intelligence, GeoIP, ML models and IDS rules. The chart expects it as a secret named `gen0sec-download-proxy`, and the relay pod will not start without it, so a missing key is a failed rollout rather than a silently degraded fleet. You will create this secret in the step 2 of [Install](./05-install.mdx). ::: --- # Architecture and data flows Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/security/architecture # Architecture and data flows Written for a security reviewer. Everything here is observable in a running install, and every claim has a page you can check it against. ## What runs where Two namespaces, three charts, one ingress point. ``` internet │ │ api.gen0sec.com:443 (the only outbound path) │ ┌───────────────────────────┼──────────────────────────────────────────────┐ │ your cluster │ │ │ ▲ │ │ ┌─────────────────────── │ ─────────────────────────────────────────┐ │ │ │ namespace: gen0sec │ │ │ │ │ │ │ │ │ │ dashboard relay(download-api) 13 application services │ │ │ │ │ │ │ └───────────────────────────────┬───────────────────────────────────┘ │ │ │ in-cluster only │ │ ┌───────────────────────────────┼───────────────────────────────────┐ │ │ │ namespace: gen0sec-system │ │ │ │ │ ▼ │ │ │ │ Postgres (3) Kafka (3) object store (4) caches (2) │ │ │ │ Postgres operator Kafka operator ingress operator │ │ │ └───────────────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────┘ ▲ │ one address, TLS, path-prefix routed │ your agents and your users ``` | Namespace | Contains | Reachable from outside | |---|---|---| | `gen0sec` | The application services and the dashboard | Yes, through the ingress you publish | | `gen0sec-system` | Postgres, Kafka, object store, two caches, and the three operators | No | Nothing in `gen0sec-system` is published. The application reaches it by cluster DNS. ## Trust boundaries Four, and they are worth naming separately because they fail differently. | Boundary | Crossed by | Controlled by | |---|---|---| | **Internet to cluster** | The relay, outbound only, one destination | Your firewall. See [Ports and egress](./egress.mdx) | | **Your network to cluster** | Agents and users, inbound, one address | Your ingress and its TLS | | **Namespace to namespace** | The application reading its databases | Cluster DNS. No NetworkPolicy ships. See [Hardening](./hardening.mdx) | | **Cluster to registry** | Image pulls | The image pull secret, one per namespace | ## Data flows ### Inbound: agents and users | Flow | Path | Authentication | |---|---|---| | Agent telemetry and events | Your address, then the ingest services, then Kafka, then Postgres | Agent API key | | Agent configuration pull and push | Your address, then the configuration services | Agent API key | | Agent artifact download | Your address, then the relay, then Gen0Sec | Agent API key, validated locally. The relay uses your installation's own key upstream, so agent keys never leave. See [Data handling](./data-handling.mdx) | | Dashboard sign-in | Your address, then the dashboard, then Postgres | Email and password, held in your Postgres. No external identity provider at this release | ### Outbound: the relay, and nothing else One destination, one purpose: fetching four licensed datasets that are in no artifact we ship. | Dataset | Why it cannot be shipped | |---|---| | CTI indicators | Licensed by upstream providers | | GeoIP databases | Licensed by MaxMind and IPInfo | | ML models | Published by gen0sec, refreshed independently of releases | | IDS rules | Licensed as part of the Emerging Threats ruleset | The relay is a **relay, not a cache**. Nothing is stored on the way through, so there is no local artifact store to size and no staleness window. ### Never outbound | Dataset | What it is | Where it stays | |---|---|---| | `identity` | Workload identity produced in your cluster | Your object store | | `policy-edges` | Pod IPs, namespaces, NetworkPolicy edges | Your object store | Relaying either would push a map of your workload topology to a service you do not run. It is not configurable and there is no flag that enables it. Uploads are not relayed either: the relay is read-only. ## Where data is stored | Store | Contains | Backed up by you | |---|---|---| | Postgres | The system of record: configuration, agents, policy, history | Yes, always | | Object store | Datasets, and the two locally-produced sets above | Yes | | Kafka | In-flight events between ingest and persistence | No, it is transport | | Cache | Nothing durable | No | See [Back up and restore](./../operate/backup.mdx). ## What the platform requires of your cluster | Requirement | Detail | |---|---| | `cluster-admin` during install | For CRDs and cluster-scoped RBAC. See [Privileges and RBAC](./privileges.mdx) | | Custom resource definitions | Postgres and Kafka operator CRDs, cluster-scoped by nature | | Its own ingress controller | Installed by the infra chart, `ingressClassName: synapse` | | One outbound destination | `api.gen0sec.com:443` | | No node access, no host mounts, no privileged containers | See [Hardening](./hardening.mdx) | ## Questions a reviewer usually asks next | Question | Page | |---|---| | What exactly can leave the cluster? | [Data handling](./data-handling.mdx) | | What ports and destinations do I open? | [Ports and egress](./egress.mdx) | | Why does it need `cluster-admin`? | [Privileges and RBAC](./privileges.mdx) | | How do I verify what you shipped me? | [Supply chain](./supply-chain.mdx) | | What is the container security posture? | [Hardening](./hardening.mdx) | | What does not work yet? | [Known limitations](./../troubleshoot/known-limitations.mdx) | --- # Data handling Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/security/data-handling # Data handling ## What leaves your cluster One connection, outbound, to one destination. | | | |---|---| | Destination | `api.gen0sec.com:443` | | Made by | The relay pod only | | Direction | Outbound, request and response | | Purpose | Fetching four licensed datasets | **What is in the request.** A path naming the artifact, and your own Gen0Sec API key. Nothing else. The relay does not add a customer identifier, a cluster identifier, an agent identifier, telemetry or usage data. **What is in the response.** The artifact: a CTI indicator database, a GeoIP database, an ML model, or an IDS rule bundle. ## What never leaves your cluster | Data | What it is | |---|---| | `identity` | Workload identity produced inside your cluster | | `policy-edges` | Pod IP addresses, namespaces, NetworkPolicy edges | | Agent telemetry and events | Everything your agents report | | Configuration and policy | Everything you configure | | Logs | All of it, to container stdout in your cluster | | Database contents | Postgres never leaves | | Dashboard user accounts and passwords | Held in your Postgres | `identity` and `policy-edges` deserve the explicit statement, because they are the two that are technically served by the same service that does the relaying. **They are never relayed.** Doing so would push a map of your workload topology to a service you do not run. It is not configurable, and no flag enables it. The upload surface is not relayed either, because the relay is read-only. ## Who validates agent credentials **Your agents' keys are validated in your cluster, and they never leave it.** Every relayed route sits behind the local authentication middleware, and each one carries the same permission it carries in direct mode. So a key that would be refused when served locally is refused it when relayed. ``` agent --[agent key, authorized locally, permission checked per route]--> relay --[your Gen0Sec API key]--> api.gen0sec.com agent key stops here ``` Two separate concerns, deliberately separated: | Hop | Credential | Checked by | |---|---|---| | Agent to relay | The agent's own key | Your cluster, against your installation, with the artifact's permission | | Relay to Gen0Sec | Your Gen0Sec API key | Gen0Sec | | | | |---|---| | Credentials Gen0Sec receives | One, belonging to your installation | | Credentials Gen0Sec never receives | Your individual agent key | | What Gen0Sec can infer about your fleet | Nothing from the credential. Requests carry no agent or cluster identifier | | Revoking one agent | Local, in your dashboard, immediate | | Revoking your installation's upstream access | Revoke or rotate your Gen0Sec API key | This is the answer to "does our agent inventory leave our perimeter". It does not. :::note[Agent keys are never forwarded upstream, even by accident] On a self-hosted deployment the relay never puts a caller's key in an outbound request, in any configuration. A locally-issued key could not authenticate to Gen0Sec anyway, so forwarding it could only leak a local credential to a third party without making the request succeed. With no Gen0Sec API key configured the relay sends no credential at all rather than sending yours. ::: ## Credentials at rest | Credential | Where it lives | Generated by | |---|---|---| | Database role passwords | Kubernetes secrets | The Postgres operator | | Object store keys | A Kubernetes secret | You, during install | | Registry pull credentials | A Kubernetes secret per namespace | You, during install | | Dashboard session signing key | A Kubernetes secret | Generated once, preserved across regeneration | **No credential appears in a chart, a values file or a rendered manifest.** Every one is referenced by name. See [Secrets](./../configure/secrets.mdx). The install procedure reads every credential from a file or from standard input rather than passing it as a command-line argument, so nothing lands in shell history or in the process list. ## Credentials in transit | Hop | Encrypted | |---|---| | Agents and users to your ingress | Yes, TLS you terminate | | Relay to `api.gen0sec.com` | Yes, TLS | | Services to Postgres | Yes, TLS to the pooler | | Services to the object store | Plain HTTP, in-cluster only | | Services to the cache | Plain, in-cluster only, no credential in the URL | The last two are in-cluster traffic on the pod network. If your threat model includes an attacker on the pod network, add NetworkPolicies and a service mesh. See [Hardening](./hardening.mdx). :::note[One deliberate exception, and why] The dashboard's database connection verifies encryption without verifying the certificate chain, because its client library rejects the pooler's self-signed certificate outright while every other service's library accepts encrypt-without-verify. The traffic is encrypted and in-cluster. Do not copy that setting onto the other services: their driver rejects it. ::: ## Data retention Retention is yours. Nothing expires data on your behalf, and nothing is sent anywhere before it is deleted. | Store | Grows with | Sized by | |---|---|---| | Postgres | Agents, events, history | `postgres.volume.size` | | Kafka | In-flight events, and its retention window | `kafka.storage.size` | | Object store | Datasets and locally-produced artifacts | `rustfs.storageclass.dataStorageSize` | See [Sizing](./../02-requirements.mdx#sizing-a-production-install). ## Deleting everything [Uninstall](./../operate/uninstall.mdx) without `--keep-data` deletes the namespaces and every persistent volume in them: the database, the message log and the object store. There is no copy anywhere else. --- # Ports and egress Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/security/egress # Ports and egress Same as [Network and connectivity](./../03-network.mdx), reproduced here for better visibility. **One canonical source.** If this table and the install page ever disagree, that is a bug in our publishing, not a judgement call for you. Report it. ## Outbound All TCP, all TLS, all outbound. Nothing needs an inbound rule from the internet. | Destination | Port | Needed by | Purpose | Required | |---|---|---|---|---| | `api.gen0sec.com` | 443 | Cluster pods, the relay only | Threat intelligence, GeoIP, ML models, IDS rules | **Always** | | `releases.gen0sec.com` | 443 | Workstation | Download the install kit and its signature | **Always** | | `registry.gen0sec.com` | 443 | Workstation | Pull charts and images | **Always** | | `registry.gen0sec.com` | 443 | Cluster nodes | Image pulls at install time | Registry install only | | `fulcio.sigstore.dev` | 443 | Workstation | Signature certificate verification | **Always** | | `rekor.sigstore.dev` | 443 | Workstation | Transparency log verification | **Always** | | `tuf-repo-cdn.sigstore.dev` | 443 | Workstation | Signature trust root refresh | **Always** | | Your own registry | 443 or 80 | Cluster nodes, and the host that loads the bundle | Image pulls at install time | Offline bundle only | **Only one row applies to the running cluster in steady state:** `api.gen0sec.com:443`, and only the relay pod uses it. Everything else is install-time, and most of it is your workstation rather than your cluster. ## Inbound | Source | Destination | Port | Purpose | |---|---|---|---| | Your agents | Your published address | 443 | Telemetry, events, configuration, artifact download | | Your users | Your published address | 443 | Dashboard | One address serves both, routed by path prefix. See [Publish the endpoints](./../06-publish.mdx#the-route-table). ## In-cluster No NetworkPolicy is required, and none ships. If you enforce default-deny, these are the flows to allow. | From | To | Port | |---|---|---| | `gen0sec` services | `core-pooler.gen0sec-system` (Postgres pooler) | 5432 | | Migration job | `core.gen0sec-system` (Postgres primary) | 5432 | | `gen0sec` services | `core-kafka-bootstrap.gen0sec-system` | 9092 | | `gen0sec` services | `rustfs-svc.gen0sec-system` (object store) | 9000 | | `gen0sec` services | `dragonfly.gen0sec-system` (cache) | 6379 | | Ingress controller | `gen0sec` services | Service ports | See [Hardening](./hardening.mdx) for what we recommend beyond that. ## Notes that save time **The Sigstore hosts are the ones people forget.** Signature verification is the first command of the install, and it fails without them. They are needed on the workstation, not on the cluster. **A proxy does not help for the cluster's one destination.** Explicit forward proxies are not honoured in-cluster. See [Known limitations](./../troubleshoot/known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy). **The offline bundle removes registry egress, not all egress.** The runtime connection stays. See [What this rules out](./../03-network.mdx#what-this-rules-out). --- # Hardening Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/security/hardening # Hardening Two lists: what the charts already do, and what is left to you. The second list is short but it is not empty, and we would rather name it than let you assume otherwise. ## Hardened by default Every application service runs with this security context, and it is not opt-in. ```yaml podSecurityContext: runAsNonRoot: true runAsUser: 65532 runAsGroup: 65532 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: [ALL] ``` | Control | Status | |---|---| | Runs as non-root | Yes, uid 65532 | | Read-only root filesystem | Yes | | Privilege escalation | Disabled | | Linux capabilities | All dropped | | Privileged containers | None | | Host network, PID or IPC | None | | Host path mounts | None | | Node access | None | :::note[Read-only root filesystems were not free] Several services defaulted to writing into a relative path under their working directory, which fails outright on a read-only root. Each one now gets an explicit writable volume mounted where it expects to write. That is why the values file carries per-service volume mounts: they are the cost of this control, not decoration. If you add a service-level override that removes one, that service will fail to start. ::: ## Pod Security Admission The application pods satisfy the **restricted** profile. Label the namespace to enforce it: ```bash kubectl label namespace gen0sec \ pod-security.kubernetes.io/enforce=restricted \ pod-security.kubernetes.io/enforce-version=latest ``` **Do not label `gen0sec-system` restricted yet.** The Postgres and Kafka pods there are created by upstream operators and have not been assessed against that profile. Use `baseline` if you need something, and tell us if you require restricted there. ## What you should add None of these ship, and all of them are worth doing. ### NetworkPolicies **No NetworkPolicy ships with the charts.** On a cluster without default-deny, that means the platform namespaces are as open as everything else. Add default-deny in both namespaces, then allow exactly these: | From | To | Port | |---|---|---| | `gen0sec` services | `core-pooler.gen0sec-system` | 5432 | | Migration job | `core.gen0sec-system` | 5432 | | `gen0sec` services | `core-kafka-bootstrap.gen0sec-system` | 9092 | | `gen0sec` services | `rustfs-svc.gen0sec-system` | 9000 | | `gen0sec` services | `dragonfly.gen0sec-system` | 6379 | | Ingress controller | `gen0sec` services | Service ports | | Relay pod | `api.gen0sec.com:443` | 443, egress | **Egress default-deny is the high-value one here.** The relay is meant to be the only pod with a route out. A NetworkPolicy is what turns that from a design intention into an enforced property, and it is the single strongest control you can add to this deployment. ### TLS everywhere it is not already | Hop | Today | Worth adding | |---|---|---| | Ingress to users and agents | TLS you terminate | Already yours | | Relay to Gen0Sec | TLS | Nothing to do | | Services to Postgres | TLS to the pooler | Nothing to do | | Services to the object store | Plain HTTP, in-cluster | A service mesh, if your threat model includes the pod network | | Services to the cache | Plain, in-cluster, no credential in the URL | Enable cache authentication, then move the connection string into each service's secret | ### Resource limits on the four workloads that have none The object store, the Kafka brokers, the ingress cache and the Postgres operator declare no CPU or memory requests. That makes them BestEffort or Burstable, so the kubelet evicts them first under node pressure. For the object store, holding your data, that is the wrong priority order. Set requests on them, or set a namespace `LimitRange` so nothing lands unbounded. See [Sizing](./../02-requirements.mdx#sizing-a-production-install). ### Secrets from an external manager `make-db-secrets.sh` is fine for evaluation. In production, have your secret manager produce secrets of the same shape so credentials never exist as plain Kubernetes secrets you created by hand. Every name and key is listed in [Secrets](./../configure/secrets.mdx). ## Settings you should never enable Two environment variables exist and both disable a security control. Neither is set by the charts, and neither appears anywhere in the install procedure. | Setting | What it does | When it is acceptable | |---|---|---| | `TLS_INSECURE_SKIP_VERIFY` | Stops the relay verifying the upstream certificate | Never in production. It defeats the protection on your only outbound connection | | `MIRROR_TLS_VERIFY=false` | Stops the bundler verifying your registry's certificate when pushing | Only for a plain-HTTP registry on a trusted network, and preferably not then | If you find either one set in a running deployment, treat it as a finding. ## Verifying the posture Confirm no pod runs as root or privileged: ```bash kubectl -n gen0sec get pods -o json | jq -r ' .items[] | .metadata.name + " runAsNonRoot=" + ((.spec.securityContext.runAsNonRoot // false) | tostring)' ``` Confirm nothing holds cluster-admin: ```bash kubectl get clusterrolebinding -o json \ | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | .metadata.name' ``` Confirm only the relay can reach the internet, once you have added egress policy: ```bash kubectl -n gen0sec run netcheck --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/busybox:1.36.1-glibc -- wget -T5 -qO- https://example.com ; echo "exit=$?" ``` A non-zero exit is the result you want. --- # Privileges and RBAC Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/security/privileges # Privileges and RBAC ## The short answer `cluster-admin` is required **to install**, not to run. It is needed for two things that are cluster-scoped by nature: 1. Installing custom resource definitions. 2. Creating the cluster roles the three operators need. Once installed, no component in the platform holds cluster-admin. ## What is cluster-scoped ### Custom resource definitions From the bundled operators, not written by us. | CRD group | Installed by | Kinds | |---|---|---| | `acid.zalan.do` | Postgres operator | `postgresqls`, `operatorconfigurations`, `postgresteams` | | `kafka.strimzi.io` | Kafka operator | `kafkas`, `kafkatopics`, `kafkanodepools`, `strimzipodsets`, and the Connect and Bridge kinds we do not use | CRDs are cluster-scoped in Kubernetes. There is no namespaced alternative. :::note[Deleting a CRD deletes every resource of that kind, cluster-wide] Including any outside these two namespaces. If anything else in your cluster runs a Postgres or Kafka operator, removing these CRDs would take its databases and clusters with them. [Uninstall](./../operate/uninstall.mdx) removes them by default, but checks first: if any of those kinds is in use outside the two namespaces it is removing, it skips the CRDs and says so. `--keep-crds` skips them unconditionally. ::: ### Cluster roles | Cluster role | Belongs to | What it is for | |---|---|---| | `g0s-postgres-operator` | Postgres operator | Managing Postgres clusters and their pods | | `g0s-postgres-operator-endpoints` | Postgres operator | The `endpoints` permission its own upstream role omits, while still creating a master Endpoints object per cluster | | `postgres-pod` | Postgres pods | What a database pod itself needs | | `strimzi-cluster-operator-global` | Kafka operator | Cluster-scoped reconciliation | | `strimzi-cluster-operator-namespaced` | Kafka operator | Namespaced resources, bound per namespace | | `strimzi-cluster-operator-leader-election` | Kafka operator | Leader election | | `strimzi-cluster-operator-watched` | Kafka operator | Watching the namespaces it manages | | `strimzi-kafka-broker` | Kafka brokers | What a broker pod needs | | `strimzi-entity-operator` | Topic and user operator | Managing topics and users | | `strimzi-kafka-client` | Kafka clients | Client-side permissions | | `g0s-synapse-operator` | Ingress operator | Watching Ingress objects and services | Eleven, and nine of them come straight from the two upstream operators. :::note[The one we added, and why] `g0s-postgres-operator-endpoints` exists because the Postgres operator ships a cluster role without `endpoints`, yet creates a master Endpoints object for every cluster it manages. Without that rule, `postgresql/core` fails with `endpoints is forbidden` and the database never starts. See [Postgres will not start](./../troubleshoot/symptoms.mdx#postgres-will-not-start). ::: ## What the application itself holds The application services in `gen0sec` are not operators and do not watch the cluster. | | | |---|---| | Cluster-scoped permissions | None | | Namespaced permissions | Reading their own configuration and secrets | | Service accounts | One per service | | Node access | None | | Host mounts | None | | Privileged containers | None | See [Hardening](./hardening.mdx) for the container security context. ## Can I install without cluster-admin? Partly, if you split the install between two roles. This is a supported pattern but it is not the documented golden path, so plan for one extra iteration. | Step | Needs | Who | |---|---|---| | Apply CRDs | Cluster-scoped write on `customresourcedefinitions` | Your platform team, once per release | | Create cluster roles and bindings | Cluster-scoped RBAC write | Your platform team, once | | Install the three charts | Namespace-admin on both namespaces | The installer | Render what a chart would create, and hand that to whoever holds the privilege: ```bash helm template g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system -f $B/values/gen0sec-infra-values-onprem.yaml \ | yq eval-all 'select(.kind == "ClusterRole" or .kind == "ClusterRoleBinding")' ``` ## Verifying after install Confirm nothing holds more than it should: ```bash kubectl get clusterrolebinding -o json \ | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | .metadata.name + " -> " + ([.subjects[]?.name] | join(","))' ``` No entry in that output should reference a `gen0sec` service account. If one does, tell us: that is a defect, not a configuration choice. ## What an uninstall leaves behind | | Removed | Reason | |---|---|---| | Namespaced resources | Yes | Helm owns them | | Objects the operators created outside the releases | Yes | The script removes them; `helm uninstall` alone does not | | CRDs | Yes, unless in use elsewhere or `--keep-crds` | Removal is cluster-wide destructive, so the script checks before it acts | | Cluster roles and bindings | Depends on the chart that created them | Check what remains and remove by hand if nothing else uses the operators | [Uninstall](./../operate/uninstall.mdx) reports what it removed and what it left. --- # Provenance, signatures and SBOM Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/security/supply-chain # Provenance, signatures and SBOM Everything we publish is signed, and every signature is verifiable by you without asking us for a key. This page also includes the one case where verification cannot follow the artifacts. ## What is signed, and how | Artifact | Signed | Attested | |---|---|---| | Install kit tarball | Detached cosign bundle, published beside it | | | Container images | `cosign sign --recursive`, so the index **and** each per-architecture manifest | SPDX JSON SBOM, one per architecture | | Helm charts | `cosign sign` on the OCI chart artifact | SPDX JSON SBOM | **Keyless, with GitHub OIDC.** There is no long-lived private key to steal, and no public key for you to fetch and trust. Each signature is bound to a workflow identity recorded in a public transparency log, so verification asserts *who built it*, not *who holds a key*. SBOMs are generated with syft and attached as SPDX JSON attestations. :::note[SLSA provenance is deliberately off] BuildKit's provenance attestations add manifests to the multi-architecture index carrying a placeholder platform. That breaks tools which walk an index by platform, including our own mirroring. We ship the SPDX SBOM attestation instead, which is the thing customers actually verify. If you require SLSA provenance, tell us: it is a build-flag change plus mirroring work, not a redesign. ::: ## Verify the install kit Do this before you unpack it. It is step 3 of [Prepare the artifacts](./../04-prepare-artifacts.mdx). ```bash cosign verify-blob \ --bundle cerebellum-install-kit.tar.gz.cosign.bundle \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/release\.yaml@refs/tags/' \ cerebellum-install-kit.tar.gz ``` **Both certificate flags are required.** `cosign` refuses to verify without an expected issuer and an expected identity, so a shortened command fails rather than passing weakly. The identity is pinned to **one workflow, on a tag reference**. A looser pattern would accept a signature produced by any workflow, on any branch, in our organization. That is the difference between "built by our release pipeline from a tag" and "built by somebody at Gen0Sec". ## Verify an image or a chart Against our registry, or against yours after mirroring: ```bash cosign verify \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/' \ registry.gen0sec.com/gen0sec/auth-api:0.1.1 ``` ## Read the SBOM ```bash cosign download attestation \ --predicate-type https://spdx.dev/Document \ registry.gen0sec.com/gen0sec/auth-api:0.1.1 \ | jq -r '.payload | @base64d | fromjson | .predicate' ``` The predicate is an SPDX JSON document. Feed it to whatever consumes SBOMs in your pipeline. ## Verification across the air gap This is the part nobody else documents, so it gets its own section. ### What crosses, and what does not Signature verification needs the public transparency log, and an isolated network does not have one. So the bundler verifies **once, on the connected staging host, before packaging**, and protects everything downstream with checksums. ``` connected staging host isolated network ────────────────────── ──────────────── pull every artifact by digest cosign verify each one (cannot re-verify: no transparency log) write SHA256SUMS -> verify-package re-checks SHA256SUMS print the bundle fingerprint -> mirror re-checks it before pushing package ``` | Stage | Guarantee | |---|---| | Before packaging | Full cryptographic signature verification, every image and every chart | | The tarball | `SHA256SUMS` over every file, plus a **fingerprint**, which is a sha256 of that list | | On the isolated side | `verify-package` and `mirror` both re-check `SHA256SUMS` and refuse to proceed on a mismatch | **Record the fingerprint** that verification prints on the connected side, out of band. Comparing it on the isolated side is what closes the loop between the two networks. :::note[Signature re-verification inside the isolated network is not supported] It would need the transparency log, or a pinned trust bundle carried across and kept current. We do not pretend otherwise. If your policy requires cryptographic verification at the destination rather than checksum verification, tell us. That is a different design, and we would rather discuss it now than have you find the gap during an audit. ::: ### The one case where attestations are lost `--arch amd64` or `--arch arm64` halves the transfer. It also extracts **one manifest out of each multi-architecture index**, which changes every image digest by definition. | | Multi-architecture bundle (default) | Single-architecture bundle | |---|---|---| | Image signatures | Verify | **Verify.** Each per-architecture manifest is signed individually | | SBOM attestations | Verify | **Lost.** They hang off the index, and the index is what is left behind | | Digests match the release manifest | Yes | No, by definition | The bundler warns at the time and annotates the bundle rather than letting this be a silent difference. **If your acceptance process checks attestations, stay on the default.** ## Digest pinning `release-manifest.yaml` in the install kit pins **every image and chart to a digest**, not to a tag. Tags move. Digests do not. That is what makes a release reproducible: the bundler resolves nothing at build time, it fetches exactly the digests the manifest names, and it fails if one is missing. ```bash yq '.spec.images[] | [.image, .imageTag, .digest] | @tsv' $KIT/release-manifest.yaml ``` ## If verification fails | Symptom | Cause | |---|---| | `no matching signatures` | Wrong identity pattern, or the artifact is not ours | | Certificate fetch errors | The Sigstore hosts are blocked. See [Network and connectivity](./../03-network.mdx#egress-list) | | The image verifies, the SBOM does not | Single-architecture bundle. Expected, see above | | `SHA256SUMS` mismatch on the isolated side | The transfer altered the bundle. Re-transfer, do not proceed | **A failed verification is not a step to skip.** The bundler has a flag that drops the check, and a bundle built with it is annotated as unverified. Use it only if cosign genuinely cannot be installed on your staging host, and record that you did. --- # Collect diagnostics Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/troubleshoot/diagnostics # Collect diagnostics Collect this before you contact us. Everything here is something we would otherwise ask for, so sending it means the first reply is an answer rather than a request for more output. There is no single collection command at this release. This page is the list. ## Set up Run the commands from a directory you can archive afterwards. ```bash NS_SYS=gen0sec-system NS_APP=gen0sec OUT=gen0sec-diagnostics mkdir -p $OUT && cd $OUT ``` Cluster and storage: ```bash kubectl version -o yaml > cluster-version.yaml kubectl get nodes -o wide > nodes.txt kubectl describe nodes > nodes-describe.txt kubectl get storageclass -o yaml > storageclasses.yaml kubectl get pvc -A -o wide > pvc.txt ``` Releases and workloads: ```bash helm list -A > helm-releases.txt for ns in $NS_SYS $NS_APP; do kubectl -n $ns get all -o wide > workloads-$ns.txt kubectl -n $ns get events --sort-by=.lastTimestamp > events-$ns.txt kubectl -n $ns get pods -o yaml > pods-$ns.yaml done ``` Logs from anything not ready: ```bash for ns in $NS_SYS $NS_APP; do for p in $(kubectl -n $ns get pods -o json \ | jq -r '.items[] | select(any(.status.containerStatuses[]?; .ready == false)) | .metadata.name'); do kubectl -n $ns logs $p --all-containers --tail=2000 > logs-$ns-$p.txt 2>&1 kubectl -n $ns logs $p --all-containers --previous --tail=2000 > logs-$ns-$p-previous.txt 2>&1 done done ``` Operators and custom resources: ```bash kubectl -n $NS_SYS logs deploy/g0s-infra-postgres-operator --tail=2000 > logs-postgres-operator.txt kubectl -n $NS_SYS logs deploy/strimzi-cluster-operator --tail=2000 > logs-kafka-operator.txt kubectl -n $NS_SYS logs deploy/g0s-infra-synapse-operator --tail=2000 > logs-ingress-operator.txt kubectl -n $NS_SYS get postgresql,kafka,kafkanodepool,kafkatopic -o yaml > custom-resources.yaml ``` The migration job, which is the most common single cause: ```bash kubectl -n $NS_APP logs job/g0s-db-migrate --tail=2000 > logs-db-migrate.txt 2>&1 ``` Images actually in use, including init containers: ```bash for ns in $NS_SYS $NS_APP; do kubectl -n $ns get pods -o yaml | grep -E '^\s+image:' | sort -u > images-$ns.txt done ``` Routing and the relay: ```bash kubectl -n $NS_APP get ingress -o yaml > ingress.yaml kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' > relay-healthz.json 2>&1 ``` Then pack it: ```bash cd .. && tar -czf gen0sec-diagnostics-$(date +%Y%m%d-%H%M%S).tar.gz gen0sec-diagnostics ``` :::danger[Read the archive before you send it] It is plain text and it is yours. Nothing here redacts anything, so check it, and remove any sensitive data it might have, particularly if you added `helm get values`, which prints whatever you set. None of the commands above read secret contents, database rows or object store objects. Logs can contain request paths and key prefixes. ::: :::note[There should be one command for this] A single `collect-diagnostics.sh` in the install kit would replace this page with two lines, and the list above is the specification for it. A single diagnostics tool can accidentally collect data that's sensitive and may fail to redact that information, so that's why we ask you to assemble your own report. Once we're 100% sure we can provide a diagnostics tool that doesn't leak any information of your sensitive data, we'll do so. ::: ## What to tell us alongside it Four lines, and they save a round trip: 1. Which install method you used. 2. The release version. 3. The last command you ran, verbatim. 4. What you expected to happen. --- # Known limitations in this release Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/troubleshoot/known-limitations # Known limitations If you hit something that is not on this page, we'd like to hear about it! ## No zero-egress deployment **What it means.** Your cluster must reach `api.gen0sec.com` on port 443. That connection carries threat intelligence, GeoIP, ML models and IDS rules, which are licensed upstream data that is in no artifact we ship. **What you can remove.** All dependency on Gen0Sec-hosted registries. The [offline bundle method](./../04-prepare-artifacts.mdx) mirrors every image and chart into your own registry, so your cluster pulls nothing from us. **What you cannot remove.** The runtime connection. There is no supported configuration with zero cluster egress. **If your policy forbids all egress**, please contact us. See [Network and connectivity](./../03-network.mdx). ## Our registry challenges pulls that send no user agent **By design on our side, and it can decide which install method you use.** **Symptom.** Every image pull from `registry.gen0sec.com` returns **403 with a CAPTCHA challenge**, and pods sit in `ImagePullBackOff`. A manual `docker pull` or `skopeo copy` from the same node **succeeds**. That discrepancy is the whole diagnosis. If the CLI works and the kubelet does not, this is it. **Cause.** On some Kubernetes distributions, the component that services the kubelet's pull sets no `User-Agent` header. Go's `net/http` then sends its own default, `Go-http-client/1.1`, and our edge treats that as a scanner signature and issues a challenge. A kubelet cannot solve a CAPTCHA, so the pull never succeeds and never will. It is not your credentials, not the pull secret, not TLS, and not an allowlist. ### What passes and what does not Measured against our registry: | User-Agent | Result | |---|---| | `containerd/1.7.0` | 200 | | `containerd/v2.1.4` | 200 | | `docker/29.4.0` | 200 | | `curl/8.7.1` | 200 | | `skopeo/1.24.0` | 200 | | `Go-http-client/1.1` | **403, challenged** | | `Go-http-client/2.0` | **403, challenged** | **containerd's own user agent is fine.** Any real one is. The failure is specifically the absence of a user agent, which Go turns into `Go-http-client`. ### Confirming it The blocked response identifies what it matched: ``` "user_agent": "Go-http-client/1.1", "ja4_fingerprint": null, "captcha_validated": false ``` Compare a manual pull against what the kubelet gets: ```bash # on the node: succeeds, because the CLI sends its own user agent skopeo inspect docker://registry.gen0sec.com/gen0sec/auth-api:0.1.1 ``` ```bash # what the kubelet sees kubectl -n gen0sec describe pod | grep -A3 Failed ``` Success from the CLI plus 403 from the kubelet confirms it. ### Two ways forward **Option 1: use the offline bundle method.** Nothing to configure on your nodes, and your own registry does not inspect user agents. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). **Option 2: make the pull path send a user agent**, on every node. :::note[Settle this before you choose an install method] If you cannot change container runtime configuration on your nodes, whether by policy or because your distribution manages it, use the offline bundle method. This constraint forces that choice, and it is better made before you start than at the first failed pull. ::: ## `--set entrypointHost` has no effect Every route publishes on the `gen0sec.internal` placeholder whatever address you pass, and nothing reports it. Set `entrypointHost` inside `$KIT/examples/single-entrypoint/ingress-values.yaml` instead. ## `--set externalScheme` has no effect Set `services.ui.ingress.externalScheme` instead. ## The relay cannot use an explicit egress proxy **Symptom.** You set `HTTPS_PROXY` on the relay. Nothing changes, and nothing errors. **Cause.** Every HTTP client in the platform builds its own transport without proxy support, so `HTTP_PROXY` and `HTTPS_PROXY` are ignored inside the cluster. This is by design. **What works instead.** | Option | Notes | |---|---| | Allow `api.gen0sec.com:443` directly | One firewall exception that bypasses the proxy | | Redirect transparently at the network layer | The client does not need to be proxy-aware | A TLS-intercepting proxy also cannot work: the charts expose no way to mount a CA bundle into the platform containers, and the relay rejects the intercepted certificate. ## The object store bucket is not created for you **Symptom.** Services start and then fail to read or write the object store. `download-api` answers `500 NoSuchBucket`. **Cause.** Nothing in the charts creates the bucket. **What to do.** [Install](./../05-install.mdx#5-create-the-object-store-bucket) step 5 creates it. It is a first-class step, not an optional one. ## Three services report readiness from a weaker probe `download-api`, `ids-rules-api` and `service-graph-api` serve a richer `/status/healthz` and, in doing so, never register `/status/readyz`. Requesting it returns 404, so a readiness probe pointed at it fails forever while liveness stays healthy, and the pod sits at `0/1` with zero restarts. **What that means for you.** Do not repoint those services at `/status/readyz`. It will fail. ## Helm never upgrades CRDs Not our defect, but it will bite you on the first upgrade that contains CRDs, so it is here. Helm installs a chart's CRDs on first install and ignores them from then on. An upgrade that ships a new CRD schema fails with a server-side-apply error, or silently rejects new fields. [Upgrade](./../operate/upgrade.mdx) applies them by hand as step 1. Do not skip it. ## Kafka volumes survive an uninstall `deleteClaim: false` is deliberate, so `helm uninstall` leaves the Kafka volumes behind. Reinstalling against them produces `Invalid cluster.id ... Expected X, but read Y`, because the new cluster generates a new identity. Delete the volume claims first, or restore the original cluster. See [Kafka will not start](./symptoms.mdx#kafka-will-not-start). ## Migration `force` means the opposite of what people expect `migrate force ` marks that version **applied** and continues with the next one. To re-run a migration that failed, force the version **before** it. Migrations are not idempotent. Forcing to the wrong version re-runs DDL that already exists and fails with `already exists`. --- # Symptom index Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/troubleshoot/symptoms # Symptom index Everything here was reproduced on a real cluster. ## Images ### ImagePullBackOff Three different causes, and the first thing to establish is which. ```bash kubectl -n get pod -o jsonpath='{.spec.containers[*].image}' kubectl -n describe pod | tail -20 ``` | The image reference is | Cause | Fix | |---|---|---| | Correct, and the error mentions authorization | The `gen0sec-registry` secret is missing from that namespace | It is needed in **both** namespaces. See [Install step 2](./../05-install.mdx#the-image-pull-secret) | | Pointing at the wrong registry | A values file was not passed with `-f` | Re-run with `-f $B/values/-values-onprem.yaml` | | A path that does not exist in your registry | Mirror path mismatch, on the offline bundle method | See the next entry | ### 403 with a CAPTCHA challenge from our registry All images, not one, and only on the registry install method. **The tell:** a manual `docker pull` or `skopeo copy` from the same node succeeds while the kubelet's pull returns 403. ```bash # succeeds: the CLI sends its own user agent skopeo inspect docker://registry.gen0sec.com/gen0sec/auth-api:0.1.1 ``` ```bash # 403: what the kubelet gets kubectl -n gen0sec describe pod | grep -A3 Failed ``` On some distributions the component servicing the kubelet's pull sets no `User-Agent`, so Go sends `Go-http-client/1.1`, which our edge challenges as a scanner signature. A kubelet cannot solve a CAPTCHA, so it never recovers. Not credentials, not TLS, not the pull secret. Fix: set a user agent in the container runtime on every node, or use the offline bundle method. Both in [Known limitations](./known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ### ImagePullBackOff on a pod you did not configure Operator-launched pods (Postgres, the connection pooler, Kafka brokers, the entity operator) get their image reference from the **operator**, not from the chart. If one of those is stuck, the mirror path is wrong, not the pull secret. Confirm the path exists in your registry: ```bash curl -su : https://$REGISTRY/v2/_catalog?n=200 | tr ',' '\n' ``` Two paths look surprising and are not negotiable: | Image | Must be mirrored to | |---|---| | Kafka broker | `/gen0sec/strimzi/kafka:1.0.0-kafka-4.1.1` | | Entity operator | `/gen0sec/strimzi/operator:1.0.0` | Everything is under `gen0sec/`. Kafka additionally keeps the upstream `strimzi/` path inside it, because the operator's `defaultImageRegistry` replaces only the registry host and keeps the repository path. A Kafka pod asking for `/strimzi/kafka` with no `gen0sec/` means that setting lost the namespace. ### 401 Unauthorized pulling into a Postgres or pooler pod The Postgres operator does not copy `imagePullSecrets` onto the database pods it creates. They inherit them from their ServiceAccount, which `values-onprem.yaml` configures through `configKubernetes.pod_service_account_definition`. If you overrode `configKubernetes` wholesale, you dropped it. Merge instead of replacing. ### manifest unknown A chart is asking for a tag that was never published. Check what it rendered: ```bash helm get manifest -n | grep -E '^\s+image:' | sort -u ``` Then compare against [Component versions](./../configure/components.mdx). ## Postgres ### Postgres will not start `postgresql/core` sits in `CreateFailed`. ```bash kubectl -n gen0sec-system logs deploy/g0s-infra-postgres-operator | grep -i 'could not create' ``` | Log line contains | Cause | Fix | |---|---|---| | `endpoints is forbidden` | The operator's own ClusterRole omits `endpoints`, but it creates a master Endpoints object per cluster | The infra chart adds the missing rule. Confirm: `kubectl get clusterrole g0s-infra-postgres-operator-endpoints` | | `Unsupported value: "18"` | The CRDs in the cluster are older than the operator chart | Helm never upgrades CRDs. Apply them first. See [Upgrade](./../operate/upgrade.mdx) | | Nothing, and no volume is bound | No default StorageClass, or not enough capacity | See [Sizing](./../02-requirements.mdx#sizing-a-production-install) | After fixing the cause the operator does not retry promptly. Force it: ```bash kubectl -n gen0sec-system rollout restart deploy/g0s-infra-postgres-operator ``` ### Migration failures The migration job runs as a Helm hook. On failure Helm aborts and deploys nothing, so the cluster is in a safe state. ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` | Log line contains | Cause | Fix | |---|---|---| | `relation "..." does not exist` | The chain is running against a database that is not at the version it expects. Usually a forced version, or a `db-migrate` image older than the chart | Read the version table in the next entry before you force anything, then [collect diagnostics](./diagnostics.mdx) | | `role "..." does not exist` | A database role is missing. The migrations `GRANT` to roles by name and roll back if one is absent | The data chart creates them. Confirm: `kubectl -n gen0sec-system get secret \| grep credentials` | | `already exists` | A migration was forced to the wrong version and re-ran DDL | See the next entry | ### The schema is marked dirty The migration tool marks the schema dirty when a migration fails halfway, and refuses to continue until you say what happened. ```bash kubectl -n gen0sec-system exec -it core-0 -c postgres -- \ psql -U postgres -d ax_prod -c 'select * from schema_migrations' ``` `force ` means "**this version is applied**, continue with the next one". So to re-run a failed migration, force the version **before** it. Migrations are not idempotent. ## Kafka ### Kafka will not start Broker crash loops with `Invalid cluster.id ... Expected X, but read Y`. The volume survived a previous `helm uninstall`, which is deliberate, and the recreated cluster generated a new identity. Either restore the original cluster or wipe the volumes: ```bash kubectl -n gen0sec-system delete kafkanodepool kafka kubectl -n gen0sec-system delete pvc -l strimzi.io/cluster=core helm upgrade g0s-data ... ``` **This deletes the message data on those volumes.** ### No KafkaTopic resources exist although the chart rendered them Helm can skip re-applying objects that were deleted out of band. Re-apply the stored manifest: ```bash helm get manifest g0s-data -n gen0sec-system | kubectl apply -f - ``` ## Object store and cache ### Services cannot reach the object store The object store Service is `rustfs-svc`, with the suffix. `S3_ENDPOINT` and `ARXIGNIS_DATA_URL` must match it. ```bash kubectl -n gen0sec-system get svc | grep rustfs kubectl -n gen0sec get deploy g0s-download-api \ -o jsonpath='{.spec.template.spec.containers[0].env}' | tr ',' '\n' | grep -E 'S3_ENDPOINT|DATA_URL' ``` See [Install step 7](./../05-install.mdx#7-install-the-platform). ### 500 NoSuchBucket The bucket does not exist. [Install step 5](./../05-install.mdx#5-create-the-object-store-bucket) creates it. Nothing in the charts does. ### Everything works but reads are slow The shared cache is never fatal. Missing or unreachable, each service logs a warning once and runs on its in-process cache only. Confirm from the service side, not the pod list: ```bash kubectl -n gen0sec logs deploy/g0s-auth-api | grep -i "L2 cache" ``` `L2 cache (Redis) connected to server` means the cache is in use. `falling back to L1-only cache` means `REDIS_URL` is wrong or unreachable. ```bash kubectl -n gen0sec-system get svc dragonfly ``` The Service is deliberately not prefixed with the release name, because `REDIS_URL` names it directly. ### The cache pod crash loops ```bash kubectl -n gen0sec-system logs deploy/dragonfly --previous ``` | Log shows | Cause | Fix | |---|---|---| | `io_uring` errors | Blocked by the host kernel or a seccomp profile | Add `--force_epoll` to `dragonfly.extraArgs` in the infra values | | Read-only filesystem errors | The pod runs with a read-only root, which is why the chart disables snapshotting | Remove any flag you added that re-enables it | ### The cache is being evicted or OOM-killed `maxMemory` is the eviction threshold and the memory limit is the kill threshold. Keep the first comfortably below the second, or the kubelet kills the pod before it gets a chance to evict. ## Install and workstation ### The bundler fails with a yq syntax error You have the wrong `yq`. Two unrelated tools share the name. ```bash yq --version ``` The output must contain `mikefarah`. If not, install the Go version and point the bundler at it with `MIKEFARAH_YQ_PATH`. See [Requirements](./../02-requirements.mdx#check-your-yq). ### zsh: no matches found, before helm runs `zsh` expands `[0]` in a `--set` argument as a glob. Single-quote the whole argument: ```bash --set 'services.ui.ingress.hosts[0].host=example.internal' ``` ### The namespace hangs in Terminating Strimzi puts a finalizer on all 23 KafkaTopics, and once the operator is gone nothing clears it. ```bash kubectl get kafkatopics.kafka.strimzi.io -n gen0sec-system -o name \ | xargs -r -n1 kubectl patch -n gen0sec-system --type=merge \ -p '{"metadata":{"finalizers":null}}' ``` `uninstall.sh` does this for you, in the right order. See [Uninstall](./../operate/uninstall.mdx). ### A pod reports Running but nothing works A pod in `CrashLoopBackOff` still reports `status.phase: Running`. Read the `READY` column, never a field selector on phase. ```bash kubectl -n gen0sec get pods ``` ### Agents get 401 on every relayed artifact Your Gen0Sec API key is wrong, or the secret holds a trailing newline. ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` ``` artifact relayed from upstream path=/v1/ids-rules/version status=401 ``` Recreate the secret with `printf`, not `echo`. `--from-file` stores the bytes exactly, so a newline becomes part of the key. See [Data relay](./../configure/relay.mdx#getting-the-key-right). A missing secret does not produce this: the relay pod fails to start instead. See the next entry. ### CreateContainerConfigError on the relay pod The pod references a secret that does not exist. Almost always `gen0sec-download-proxy`: `downloadProxy.apiKeySecret.name` is set but the secret was never created. ```bash kubectl -n gen0sec describe pod -l app.kubernetes.io/name=download-api | tail -20 kubectl -n gen0sec get secret gen0sec-download-proxy ``` Create it. See [Install step 2](./../05-install.mdx#your-gen0sec-api-key). This failure is the visible one. The dangerous case is the opposite: leaving `apiKeySecret.name` unset renders no secret reference at all, the pod starts, and the failure is the silent 401 above. ### The relay reports 502 on every relayed route The upstream is unreachable. A 502 is deliberate rather than a 404 or 500: a polling agent reads it as "ask again later" and keeps the artifact it already holds. ```bash kubectl -n gen0sec exec deploy/g0s-download-api -- curl -fsS localhost:9999/status/healthz ``` Check that `api.gen0sec.com:443` is allowed. See [Ports and egress](./../security/egress.mdx). If you route egress through an explicit proxy, that is the cause: proxies are not honoured in-cluster. See [Known limitations](./known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy). ## Routing ### An agent gets HTML, or a redirect, from an API The request reached the dashboard's catch-all instead of the intended service. Routing is by longest matching prefix and the dashboard owns `/`, so any path missing from the route table lands there. ```bash kubectl -n gen0sec get ingress -o wide ``` Compare against [Publish the endpoints](./../06-publish.mdx#the-route-table). The failures this causes are quiet by design of HTTP, not by ours: | Missing route | What the agent sees | |---|---| | `/v1/authcheck` | A redirect and HTML. The agent cannot distinguish a bad key from a routing mistake and continues unverified | | `/v1/agents/events` | `SSE connect returned HTTP 404`, then silent fallback to polling, which looks like a slow platform | | `/v1/ban-ledger` | A redirect the agent reads as success. The contribution is lost silently | ### kubectl get ingress shows an empty HOSTS column The ingress controller ignores host-less rules: the object is created, nothing is logged, and no traffic is served. Set `entrypointHost`. See [Publish the endpoints](./../06-publish.mdx). ## Air gap and mirroring ### Confirming the cluster really is isolated From a pod, not from a node. ```bash kubectl -n gen0sec run netcheck --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/busybox:1.36.1-glibc -- wget -T5 -qO- https://ghcr.io/ ; echo "exit=$?" ``` A non-zero exit is the expected result. If it succeeds, the cluster still has egress and an install that appears to work may be pulling from the internet. ### An image is missing for one architecture You built the bundle with `--arch`. A single-architecture bundle fails when the kubelet tries to pull on a node of the other architecture, which on an isolated cluster is after the transfer. Rebuild without `--arch`, or with the architecture your nodes actually run. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). ## Still stuck [Collect diagnostics](./diagnostics.mdx), then send us the archive. --- # Verify the install Source: https://docs.gen0sec.com/docs/cerebellum/0.1.1/verify # Verify the install Running pods does not mean a working platform. A worker with no Service, a misconfigured credential, or a missing route all leave every pod `Ready` while no data moves. This page defines what "installed successfully" means. ## Before you start - [ ] [First sign-in](./07-first-signin.mdx) complete - [ ] An agent API key was created ## 1. Every pod is ready ```bash kubectl -n gen0sec-system get pods kubectl -n gen0sec get pods ``` Every pod reports `Running` and `n/n` in the `READY` column. On a correct install there are no exceptions: nothing is expected to sit unready. :::note[Check readiness, not phase] A pod in `CrashLoopBackOff` still reports `status.phase: Running`. A `--field-selector` on phase reports a broken install as clean, which is why the command above reads the `READY` column instead. ::: ## 2. The migration job is gone ```bash kubectl -n gen0sec get job g0s-db-migrate ``` The output is similar to this: ``` Error from server (NotFound): jobs.batch "g0s-db-migrate" not found ``` `NotFound` is the pass. Helm deletes the job the moment it succeeds. If it is still there, it failed: `kubectl -n gen0sec logs job/g0s-db-migrate`. ## 3. The relay is connected ```bash kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' ``` The output is similar to this: ```json { "status": "ready", "mode": "proxy", "upstream": "https://api.gen0sec.com", "external_relays": ["geoip", "ids-rules"] } ``` `"mode": "proxy"` confirms the relay came up. If the upstream is unreachable, relayed routes answer 502, which a polling agent reads as "ask again later" rather than "the artifact is gone". Then confirm it is actually authorized upstream, which the mode alone does not tell you: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means your Gen0Sec API key is wrong. ## 4. The platform serves an agent This is the check that matters. It makes the same requests an agent makes, from outside the cluster, over your published address. ```bash export GEN0SEC_BASE_URL=https://$ENTRYPOINT_HOST export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) $KIT/scripts/verify-deployment.sh ``` It checks four things: 1. The API refuses unauthenticated callers. 2. Configuration is served. 3. Threat intelligence and IDS rule bundles are published. 4. A downloaded bundle matches the SHA-256 its manifest advertises. It is read-only and safe to run against production. It exits non-zero and names the failing check. **Run it after every install and every upgrade.** It is the cheapest way to catch a deployment that looks healthy and serves nothing. ## Offline bundle only: confirm nothing pulls from outside ```bash kubectl -n gen0sec-system get pods -o yaml | grep -E '^\s+image:' | sort -u kubectl -n gen0sec get pods -o yaml | grep -E '^\s+image:' | sort -u ``` Every image listed is under your registry. Anything else means a values file was not passed with `-f`. :::note[Why both namespaces, and why read the YAML] The application runs in `gen0sec`; only the operators, object store and cache run in `gen0sec-system`. Checking one namespace covers well under half the images. Reading the YAML rather than a container template also catches init containers, which is where the object store's init image is. ::: ## Definition of installed successfully All five are true: 1. Every pod in both namespaces reports `Running` and `n/n` ready. 2. `kubectl -n gen0sec get job g0s-db-migrate` returns `NotFound` or appears as `Completed`. 3. `download-api` reports `"mode": "proxy"` with your upstream, and its log shows no `status=401` or `status=402` on relayed paths. 4. `verify-deployment.sh` exits zero. 5. On the offline bundle method, every image reference is under your registry. If all five hold, the install is done. If any fail, start at the [symptom index](./troubleshoot/symptoms.mdx). ## You may be interested in... - [Upgrade](./operate/upgrade.mdx), so you know how to move before you need to - [Roll back](./operate/rollback.mdx), and where the data boundary is - [Known limitations in this release](./troubleshoot/known-limitations.mdx) --- # Overview Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/ # Overview ## What is Cerebellum? Cerebellum is the enterprise version of the Gen0Sec platform that you can host in your own Kubernetes cluster. ## Charts | Installation order | Chart | Namespace | What it installs | |---|---|---|---| | 1. | `gen0sec-infra` | `gen0sec-system` | Three operators (Postgres, Kafka, ingress), the object store, and two caches: one shared by the application, one for the ingress data plane | | 2. | `gen0sec-data` | `gen0sec-system` | The Postgres cluster, the Kafka cluster, topics and database roles | | 3. | `gen0sec-platform` | `gen0sec` | Schema migrations, then the application services and the dashboard | The installation order is mandatory, as each chart creates resources the next one depends on. ## Installation methods ### Installation from registry (Recommended) Choose it when, - your cluster can access our registry, - you don't mind depending on an external registry ### Install via creating an offline bundle Choose it when, - your cluster don't have access to our registry, - you'd rather host mirrored images yourself, - you're facing issues pulling images from our registry :::note[Same content] Worry not, the offline bundle builds the same release and installs the same charts. ::: If you are unsure, go with **Install from registry**. You can always switch later. :::info[One thing that can decide this for you] Our registry challenges pulls that arrive with no `User-Agent`, and on some Kubernetes distributions the kubelet's pull path sends none. That is deliberate policy on our side, and the result is a 403 the kubelet cannot recover from. If you cannot change container runtime configuration on your nodes, use **Create an offline bundle**. Your own registry does not inspect user agents. See [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: You will need to decide the installation method in the [Prepare the artifacts](./04-prepare-artifacts.mdx) step. ## Connectivity :::warning[What 'offline' means] The 'offline' part in the offline bundler is used to describe the installation method, not the resulting product. ::: It's important to note that even by creating an offline bundle, doesn't mean you can operate the platform in an airgapped cluster that is entirely isolated from reaching the public internet. Cerebellum needs exactly one outbound destination from your cluster: - `api.gen0sec.com` on port `443`. That single connection carries **threat intelligence, GeoIP databases, ML models and IDS rules**. All four are licensed by their upstream providers and are not included in any of the artifacts we ship. Every agent in your fleet reaches it through one in-cluster service, so it is one destination to allow and one to audit. Nothing else in either namespace needs a route out. The dashboard can optionally check for new versions and send email; see [Network and connectivity](./03-network.mdx). Read [Network and connectivity](./03-network.mdx) before you start. It is the page to send to whoever runs your firewall. ## How long it takes Set aside **half a day for a first install**, and about an hour once you have done one. The install is 26 numbered steps across five pages, and most of the elapsed time is spent waiting on image pulls and on two clusters becoming ready, not on typing. The waits are gated explicitly, so the ceiling is knowable. These are the timeouts the commands themselves carry — the upper bound before a step is treated as failed, not a prediction: | Stage | You are waiting on | Gate | |---|---|---| | [Prepare the artifacts](./04-prepare-artifacts.mdx) | Downloads and signature verification. From the registry this is minutes; building an offline bundle copies every image and is the longest single stage | None | | [Install](./05-install.mdx) infra | Three operators and the object store | None | | [Install](./05-install.mdx) data | Kafka ready, then Postgres `Running` | `15m` each | | [Install](./05-install.mdx) platform | Schema migrations, then every pod ready | `20m` + `10m` | | [Publish](./06-publish.mdx) | The ingress rollout | `20m` | | [First sign-in](./07-first-signin.mdx) | You: MFA enrolment, a forced password change, the onboarding wizard | None | | [Verify](./08-verify.mdx) | Four checks you run | None | What actually decides your number is how fast your nodes pull images. Everything else is bounded by the table above. :::note[The Quickstart is not a shortcut] It is the same procedure with one extra values file per chart, on one node. It is quicker because a single-node cluster pulls and schedules less — not because it skips steps. ::: ## Before you install 1. [Requirements](./02-requirements.mdx). Cluster, tools, and your access token. 2. [Network and connectivity](./03-network.mdx). What to allow, and why. 3. [Prepare the artifacts](./04-prepare-artifacts.mdx). Prepare for the installation. If you want a working instance to look at before you plan a real install, use the [Quickstart](./quickstart.mdx). It builds a throwaway single-node install and tears it down again. --- # Component versions Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/configure/components # Component versions :::note[Read this from your release, not from this page] Every release pins its own versions, and the authoritative list is in the install kit you downloaded. ```bash yq '.spec.images[] | [.image, .imageTag, .digest] | @tsv' $KIT/release-manifest.yaml yq '.spec.charts[] | [.chart, .version, .digest] | @tsv' $KIT/release-manifest.yaml ``` This page describes what those versions mean and which of them have surprising requirements. ::: {/* This page should be generated from release/components.yaml and release-manifest.yaml in CI, per the structure proposal section 7. Hand-maintained, it drifts within one release and is then worse than nothing, because operators trust it. */} ## Third-party components | Component | Version | Notes | |---|---|---| | Postgres operator | 2.0.1 | Zalando. **1.15.x cannot be used**: its schema rejects PostgreSQL 18 | | PostgreSQL | 18, with extensions | Includes AGE, TimescaleDB, `pg_cron`, `pg_net`, `pglogical`, `pgsodium` | | Connection pooler | 2.0.1 | Operator-launched. Services connect through it, never directly | | Logical backup | 2.0.1 | Only used when logical backups are enabled | | Kafka operator | 1.0.0 | Strimzi. v1 CRD API only | | Apache Kafka | 4.1.1 | **KRaft, no ZooKeeper.** Operator-launched | | Object store | Chart 0.10.0 | S3-compatible. Its Service has a `-svc` suffix | | Shared cache | 1.38.1 | Redis protocol. Application services degrade to in-process caching without it | | Ingress cache | Valkey | Deployed by the ingress chart and used only by it. Kept separate on purpose, so a shared-cache outage cannot take the ingress data plane with it. One replica: a restart costs a cold start, not data | | Ingress operator | Commit-tagged | `ingressClassName: synapse`. Built in a sibling repository, so the tag is a commit hash rather than a version | | Platform services | The release version | Resolved from the registry host, namespace and image tag | ## Mirror paths, for the offline bundle method Every mirrored image lands under `gen0sec/`. Two paths look wrong and are not negotiable. | Image | Must be at | |---|---| | Kafka broker | `/gen0sec/strimzi/kafka:1.0.0-kafka-4.1.1` | | Kafka entity operator | `/gen0sec/strimzi/operator:1.0.0` | **Why Kafka keeps an extra path segment.** The Kafka operator's image-registry setting replaces only the registry *host* and preserves the repository path. So the chart points it at `/gen0sec` rather than ``, and the upstream `strimzi/` segment survives inside our namespace. :::note[This exact thing shipped wrong once] With the namespace omitted, the chart asked for `/strimzi/operator` while the release mirrored `/gen0sec/strimzi/operator`. One path segment apart, and the entire Kafka stack failed to pull on a cluster with no fallback. Every chart is now rendered in CI and the build fails on any image reference not backed by the release manifest, so the two cannot drift again. That gate is also why the table above is trustworthy: it catches a chart asking for a tag nobody publishes, which is how the ingress operator once requested a version that was never released. ::: `bundler.sh mirror` reproduces these paths exactly. You do not compute them. ## Not everything a chart renders is mirrored The vendored Kafka chart hardcodes an image map covering four Kafka versions plus Connect and Bridge build tooling. This edition ships one Kafka version and no Connect clusters, so the rest are deliberately absent from the bundle. That is roughly 1.5 GB of images nothing ever pulls. **If you enable Kafka Exporter, Cruise Control, Kafka Bridge or Connect builds, mirror their images first.** The release does not carry them. ## Kubernetes | | | |---|---| | Supported | 1.23 or later | See [Requirements](./../02-requirements.mdx#kubernetes-versions). ## Support policy | | | |---|---| | Supported releases | The latest two minor trains | | Upgrade path | Sequential minors. **Skipping a minor is not supported.** Patches in place | | Upgrade order | Always infra, then data, then platform | | CRDs | Helm never upgrades them. Apply them before the chart. See [Upgrade](./../operate/upgrade.mdx) | | Downgrade | Not supported across a minor that ran a schema migration. Within a train, use rollback. See [Roll back](./../operate/rollback.mdx) | | Kubernetes upgrades | Independent of ours. Stay within the supported range | ## Verifying what you are actually running The dashboard shows it under **Organization settings**, **Components**: what each component is running, and the latest version known to this release. Each chart records its versions in a ConfigMap, which you can also read directly: ```bash kubectl -n gen0sec get cm g0s-installed-versions -o jsonpath='{.data.versions\.json}' kubectl -n gen0sec-system get cm g0s-infra-installed-versions -o jsonpath='{.data.versions\.json}' kubectl -n gen0sec-system get cm g0s-data-installed-versions -o jsonpath='{.data.versions\.json}' ``` Two things about that page: - **Check for updates online** is off by default and needs `org:write`. When on, the dashboard fetches the latest release's version list from `releases.gen0sec.com` at most once an hour. Nothing about your install is sent. Air-gapped installs leave it off. - Opening it creates an API key named **Components tab (internal)**, which it uses to read data versions from the relay. Deleting it is safe; it is recreated on the next visit. If the page says it could not read infrastructure versions, the infra chart's `components.ui.namespace` and `components.ui.serviceAccountName` do not match the platform release. See [Values reference](./values.mdx#gen0sec-infra). The manifest says what a release pins. This says what your cluster pulled: ```bash kubectl -n gen0sec-system get pods -o yaml | grep -E '^\s+image:' | sort -u kubectl -n gen0sec get pods -o yaml | grep -E '^\s+image:' | sort -u ``` Reading the YAML rather than a container template also catches init containers, which is where the object store's init image is. On the offline bundle method, every reference should be under your own registry. Anything else means a values file was not passed. --- # Sign-in and identity Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/configure/identity # Sign-in and identity ## What is available | Method | Status | Where credentials live | |---|---|---| | Email and password | **Available** | Your Postgres | | Passkeys | **Available** | Your Postgres holds the public key. The private key never leaves the user's device | | Multi-factor authentication | **Available, and required during onboarding** | Your Postgres | | Password reset by email | **Available when [SMTP is configured](#invitations-and-email)** | Your mail server sends the link | | Microsoft Entra ID | **In development, not supported at the moment ** | | **Nothing outside your cluster is involved in sign-in.** Accounts, password hashes, passkeys and sessions are all in your own database. If you configure SMTP, invitation and password-reset emails go out through your mail server. :::note[Single sign-on is not available yet] Microsoft Entra ID is in development. No generic OIDC provider, no SAML and no LDAP integration exists either. If single sign-on is a requirement for you, tell us which provider, because that shapes what ships first. ::: ## The first account Registration is invite-only and invites need an existing member, so a fresh install seeds one account with a **published password**. Signing in, completing onboarding and changing that password is a required install step, not an optional one. See [First sign-in](./../07-first-signin.mdx). ## The published address decides whether sign-in works The dashboard derives its sign-in base URL from the host you publish it on. If that does not match the address people actually type in the browser, every sign-in is rejected as an invalid origin. Two rules: 1. **Use the address people type.** Not an internal service name, not a load balancer's own hostname. 2. **If TLS terminates upstream of the ingress**, set the scheme explicitly, because the chart cannot infer it and guesses `http` while the browser is on `https`. ```bash --set externalScheme=https ``` Leave that unset when TLS terminates at the ingress. See [Publish the endpoints](./../06-publish.mdx#if-tls-terminates-upstream). ## Passkeys Users can register with a passkey instead of a password, and add, rename or remove passkeys under **Profile**. Signing in with a passkey does not ask for the authenticator-app code: the passkey's own PIN or biometric check stands in for it. Enrolling multi-factor is still required during onboarding. A passkey is bound to the host name the dashboard is published on, taken from its sign-in URL. So: - **They need HTTPS and a DNS name.** A plain-HTTP address or a bare IP cannot register one. - **Changing `entrypointHost` makes every registered passkey stop working.** Users then sign in with a password, or register a new passkey on the new address. - **A user who registered with a passkey only has no password to fall back on.** Ask them to set one under **Profile**, especially when SMTP is not configured and nobody can reset it by email. ## Invitations and email Invitations grant one or more workspaces, each with a role no higher than the inviter's own. Pending invitations are listed on the **Team** tab, where they can be revoked. A link expires after seven days, and inviting the same address again replaces the previous link. With SMTP configured, the invitation is emailed and users can reset a forgotten password. Without it, the dashboard shows the invitation link for you to copy and send yourself, and the sign-in page offers no password reset. SMTP is configured through the dashboard's environment. The chart has no dedicated keys for it, so set them under `services.ui` in your overrides file, with the credentials in a Secret you create: ```bash kubectl -n gen0sec create secret generic g0s-ui-smtp \ --from-literal=user='' --from-literal=password='' ``` ```yaml title="my-overrides.yaml" services: ui: env: HOST_EMAIL: smtp.example.com HOST_PORT: "587" HOST_ALIAS_EMAIL: no-reply@example.com HOST_APP_URL: https://cerebellum.example.com envSecrets: HOST_AUTH_USER: { name: g0s-ui-smtp, key: user } HOST_AUTH_PASS: { name: g0s-ui-smtp, key: password } ``` | Variable | Notes | |---|---| | `HOST_EMAIL` | The SMTP server's host name | | `HOST_PORT` | `587` by default, with STARTTLS. `465` switches to implicit TLS | | `HOST_AUTH_USER`, `HOST_AUTH_PASS` | **Required.** SMTP is only used when the host and both credentials are set. A relay without authentication is not supported | | `HOST_ALIAS_EMAIL` | **Required.** The sender address | | `HOST_APP_URL` | **Required.** The address people type, with its scheme. Links in emails are built from it, and it is not derived from `entrypointHost` | Pass the file to every later `helm upgrade` as well. The dashboard's outbound connection to your mail server is listed on [Ports and egress](./../security/egress.mdx). ## Sessions The signing secret lives in `g0s-db-ui`, created with all the other database secrets in [Install step 6](./../05-install.mdx#6-create-the-application-secrets). | Action | Effect on sessions | |---|---| | Re-running `make-db-secrets.sh` | None. The existing signing secret is preserved deliberately | | Rotating the signing secret | Every session ends and every outstanding invite link stops working. Passwords and passkeys are unaffected | | Changing `entrypointHost` | Registered passkeys stop working | | Upgrading | None. Upgrading from v0.1.x to v0.2.0 invalidates invite links issued before the upgrade | ## Turning the dashboard off entirely If your operators use the API only: ```bash --set services.ui.enabled=false ``` The agent-facing APIs are unaffected. Note that agent API keys are issued through the dashboard, so turn it off only once you have the keys you need. ## If it fails | Symptom | Cause | |---|---| | Sign-in rejected as an invalid origin | The published host does not match the address typed, or the scheme is wrong. See above | | Sessions drop after a re-run of the secret script | The signing secret changed. It should not; check whether the secret was deleted first | | Every sign-in fails after an upgrade | Check that `externalScheme` and the host survived the upgrade. `helm get values g0s -n gen0sec` | | The passkey prompt fails or offers no passkey | The address typed differs from `entrypointHost`, the page is plain HTTP, or the host changed after the passkey was registered | | Emailed links start with `undefined/` | `HOST_APP_URL` is not set | | Invitations show a link instead of being emailed | SMTP is not configured, or sending failed. Check `kubectl -n gen0sec logs deploy/g0s-ui` | --- # Reference configurations Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/configure/reference-configs # Reference configurations Two named configurations rather than a blank values file. Each ships in the install kit, and each is reproduced in full on this page, so you can read it before you download anything. | Configuration | For | Redundancy | Storage | |---|---|---|---| | [Single node](#single-node) | Evaluation, training, upgrade rehearsal | **None** | Smallest | | [Production](#production) | Anything you depend on | Full | Smallest production grade | ## Single node **Not for production.** Everything at one replica on one node. Losing the node loses the data. Ships as three overlays in the kit: ``` $KIT/examples/single-node/infra-values.yaml $KIT/examples/single-node/data-values.yaml $KIT/examples/single-node/platform-values.yaml ``` Layer each one **after** the production values file, so it wins. Full procedure: [Quickstart](./../quickstart.mdx). | | Production | Single node | |---|---|---| | Object store | 3 replicas, erasure coded, StatefulSet | 1 replica, standalone, **Deployment** | | Postgres | 3 instances, both poolers on | 1 instance, replica pooler off | | Kafka | 3 brokers, replication factor 3 | 1 broker, every factor 1 | | Services | 2 or 3 replicas | 1 each | | StorageClass | Yours, set explicitly | The cluster's default (empty in the overlay) | ### What the overlays set Verbatim from the v0.2.0 kit, so you can read them without unpacking it first.
infra-values.yaml — object store and cache ```yaml title="$KIT/examples/single-node/infra-values.yaml" # ============================================================================ # gen0sec-infra — SINGLE-NODE EVALUATION overlay # ============================================================================ # Layered ON TOP of values-onprem.yaml (pass both, in this order): # helm install g0s-infra charts/gen0sec-infra -n gen0sec-system \ # -f values/gen0sec-infra-values-onprem.yaml \ # -f examples/single-node/infra-values.yaml # # Shrinks the footprint to fit one worker node. NOT for production: the object # store loses erasure coding and every component becomes a single point of # failure. Use it for evaluation, training, and dry-running an upgrade. # ============================================================================ rustfs: # Distributed mode needs >= 2 nodes for erasure coding; on one node the pods # would schedule but the cluster is not fault-tolerant anyway, so run the # single-process server and be explicit about it. mode: standalone: enabled: true distributed: enabled: false replicaCount: 1 storageclass: # Empty = the cluster's default StorageClass, matching values-onprem.yaml. On # k3s that resolves to local-path, which this overlay used to hardcode — but a # literal `local-path` exists on no other distribution, so the PVCs bound # nowhere and only surfaced as pods Pending past the rollout timeout. name: "" dataStorageSize: 20Gi logStorageSize: 5Gi dragonfly: # Cache only — shrink it so the node's RAM goes to Postgres and Kafka. Keep # maxMemory under resources.limits.memory so it evicts instead of being killed. maxMemory: 256mb proactorThreads: 1 resources: requests: { cpu: 100m, memory: 128Mi } limits: { cpu: 500m, memory: 512Mi } ```
data-values.yaml — Postgres and Kafka ```yaml title="$KIT/examples/single-node/data-values.yaml" # ============================================================================ # gen0sec-data — SINGLE-NODE EVALUATION overlay # ============================================================================ # helm install g0s-data charts/gen0sec-data -n gen0sec-system \ # -f values/gen0sec-data-values-onprem.yaml \ # -f examples/single-node/data-values.yaml # # One Postgres instance, one Kafka node, replication factor 1. Any node loss is # data loss — evaluation only. # ============================================================================ postgres: numberOfInstances: 1 # With a single instance there is no replica to pool. enableReplicaConnectionPooler: false volume: size: 20Gi # Empty = the cluster's default StorageClass (see infra-values.yaml). storageClass: "" resources: requests: { cpu: 500m, memory: 1Gi } limits: { cpu: "4", memory: 6Gi } parameters: shared_buffers: 512MB effective_cache_size: 1536MB max_connections: "400" max_worker_processes: "16" max_parallel_workers: "8" timescaledb.max_background_workers: "8" kafka: replicas: 1 storage: size: 20Gi # Empty = the cluster's default StorageClass (see infra-values.yaml). class: "" # min.insync.replicas must stay < replicas, so every factor drops to 1 # together with the node count. config: offsets.topic.replication.factor: 1 transaction.state.log.replication.factor: 1 transaction.state.log.min.isr: 1 default.replication.factor: 1 min.insync.replicas: 1 log.compression.type: zstd topicDefaults: partitions: 1 replicas: 1 ```
platform-values.yaml — one replica per service ```yaml title="$KIT/examples/single-node/platform-values.yaml" # ============================================================================ # gen0sec-platform — SINGLE-NODE EVALUATION overlay # ============================================================================ # helm install g0s charts/gen0sec-platform -n gen0sec \ # -f values/gen0sec-platform-values-onprem.yaml \ # -f examples/single-node/platform-values.yaml \ # --set global.imageTag=$VERSION --timeout 20m # # One replica of everything: no HA, and a rolling update is a short outage per # service. Resource requests are unchanged — they are already small. # ============================================================================ services: auth-api: { replicaCount: 1 } ban-ledger-api: { replicaCount: 1 } config-api: { replicaCount: 1 } data-api: { replicaCount: 1 } download-api: { replicaCount: 1 } feeds-api: { replicaCount: 1 } ids-rules-api: { replicaCount: 1 } rules-validator: { replicaCount: 1 } service-graph-api: { replicaCount: 1 } signal-api: { replicaCount: 1 } telemetry-api: { replicaCount: 1 } threat-api: { replicaCount: 1 } agent-status: { replicaCount: 1 } ui: { replicaCount: 1 } ```
## Production The default. `values-onprem.yaml` in the bundle **is** the production configuration, so there is no overlay to apply: what you install in [Install](./../05-install.mdx) is this. What you set yourself: ```yaml # gen0sec-infra rustfs: storageclass: name: your-block-storage-class # gen0sec-data postgres: volume: storageClass: your-block-storage-class kafka: storage: class: your-block-storage-class ``` For requirements see [Sizing](./../02-requirements.mdx#sizing-a-production-install). ## Extending retention The shipped sizes are deliberately modest, so growing is the normal direction. Which key you grow depends on what you are retaining. See [Sizing storage for your retention](./../02-requirements.mdx#sizing-storage-for-your-retention). eg. ```yaml # gen0sec-data postgres: volume: size: 200Gi storageClass: your-block-storage-class kafka: storage: size: 200Gi class: your-block-storage-class ``` ```yaml # gen0sec-infra rustfs: storageclass: name: your-block-storage-class dataStorageSize: 100Gi ``` :::note[You can also grow later] Volume expansion grows a volume if your StorageClass has `allowVolumeExpansion: true`. Starting at the defaults and expanding when you need to is a reasonable plan, and it is the direction that works: nothing shrinks a volume. ::: ## Publishing endpoints Independent of the three above, and needed by all of them: ``` $KIT/examples/single-entrypoint/ingress-values.yaml ``` One address for the dashboard and every agent-facing API, routed by path prefix. See [Publish the endpoints](./../06-publish.mdx) for the route table and the two values you set.
The header of that file, with both customer inputs and why they exist The rest of the file is the per-service route list. It is reproduced as a table on [Publish the endpoints](./../06-publish.mdx#the-route-table) rather than twice here. ```yaml title="$KIT/examples/single-entrypoint/ingress-values.yaml (head)" # ============================================================================ # gen0sec-platform — SINGLE-ENTRYPOINT INGRESS overlay # ============================================================================ # Publishes the dashboard and the agent-facing APIs through ONE address, served # by the Synapse proxy that gen0sec-infra deploys for the `synapse` IngressClass. # # helm upgrade --install g0s charts/gen0sec-platform -n gen0sec \ # -f values/gen0sec-platform-values-onprem.yaml \ # -f examples/single-entrypoint/ingress-values.yaml # # Layered ON TOP of values-onprem.yaml, which leaves every ingress disabled. # That default is deliberate — turning this on makes the APIs reachable from # wherever the proxy Service is reachable — so this is opt-in, not the default. # # A HOST IS REQUIRED — set `entrypointHost`, either below or with # `--set entrypointHost=...`. The chart reads it and publishes every rule under # it, so it is named once and nowhere else. # # It used to be a YAML anchor aliased into each rule. That could only ever be # changed by editing this file: anchors are expanded by the YAML parser before # Helm merges `--set`, so `--set entrypointHost=...` rewrote the top-level key # while every aliased copy kept `gen0sec.internal`, and the install published # on a host nobody could reach. The chart consumes the value now, so the # documented `--set` works. # # Leaving it empty FAILS THE RENDER rather than publishing anything. Host-less # rules do not work with this controller: synapse's upstreams.yaml is keyed by # host, and the operator skips any rule with an empty host # (controllers/ingress_controller.go, "if host == \"\" || rule.HTTP == nil # { continue }"). Such a rule produces an Ingress object that its own controller # silently ignores, with no event and no log line, so the chart refuses to # render one at all. # # The dashboard needs this host to be the address people actually type: # BETTER_AUTH_URL is derived from it (services.ui.urlFromIngress), and sign-in # breaks otherwise. # # ROUTING IS BY PATH PREFIX, because one address serves every backend. The # prefixes below are the real route roots, read off each service's router -- # they are not a naming convention and do not survive being guessed at. # # One overlap to keep in mind, resolved by longest-prefix matching: # /v1/agents/events config-api serves this; data-api owns /v1/agents # If a request lands on the wrong backend, that is the first thing to check. # # /v1/ids-rules goes to download-api, NOT the in-cluster ids-rules-api: in proxy # mode IDS rules are relayed from the platform, and ids-rules-api serves a local # ET ingest that this edition does not run. # ============================================================================ # CUSTOMER-INPUT: the single address every rule below is published under. entrypointHost: "" # CUSTOMER-INPUT: the scheme users actually reach that address on. # # Only needed when TLS terminates UPSTREAM of the Ingress -- a cloud load # balancer, a CDN, a tunnel. The chart otherwise infers the scheme from the # Ingress's own `tls:` block, which is absent in that setup, so it guesses http # while the browser is on https. The dashboard hands this to Better Auth as its # base URL and every sign-in is then rejected with "Invalid origin". # # Leave it unset when TLS terminates at the Ingress (i.e. you set ingress.tls), # and the chart reads the scheme off that TLS block instead. # # A single service can still override this with # `services..ingress.externalScheme`, which wins over this value. externalScheme: https ```
## Keeping your overrides Put them in a file under version control, not in `--set` arguments. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ -f my-site/platform.yaml \ --set global.imageTag=$VERSION \ --timeout 20m ``` `helm upgrade` uses only what you pass it. Anything set with `--set` on a previous install and omitted now reverts to the chart default, silently. A file makes that impossible to forget. Keep `entrypointHost` in that file. The ingress overlay ships it empty, and the render fails without it. Pass your file after the overlay so your value wins. Two things that are awkward to keep in a file, because they are per-release or per-secret: | Keep as `--set` | Why | |---|---| | `global.imageTag` | Changes every release | | `downloadProxy.apiKeySecret.name` | Or put it in your file. Just never omit it. See [Data relay](./relay.mdx) | ## What the kit contains ```bash ls $KIT/examples/*/ ``` Everything under `examples/` is a starting point you are meant to read and edit, not a black box. Each file carries comments explaining which values are yours to set. --- # Threat intelligence and the data relay Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/configure/relay # Threat intelligence and the data relay The relay is one in-cluster service, `download-api`. It is the only component with a route out, and it is how your fleet gets threat intelligence. This is not a choice you make. It is a requirement. See [Known limitations](./../troubleshoot/known-limitations.mdx#no-zero-egress-deployment). ## What it relays Read-only, upstream-produced content only. | Class | Routes | Why it is not shipped | |---|---|---| | CTI indicators | `/v1/indicators/version`, `/v1/indicators/download` | Built by the upstream intelligence pipeline | | ML models | `/v1/models/*` and the per-model aliases | Trained and published by Gen0Sec | | GeoIP | `/v1/geoip/manifest`, `/v1/geoip/{file}/download` | Licensed by MaxMind and IPInfo. The IPInfo Lite bundle is the default source | | IDS rules | `/v1/ids-rules/*` | Licensed as part of the Emerging Threats ruleset | **It is a relay, not a cache.** Nothing is stored on the way through. There is no local artifact store to size, no synchronization job to supervise, and no staleness window. An agent that polls gets whatever upstream has at that moment. **The relayed URLs are identical to the direct ones**, so your agents need no configuration change. ## What it never relays | Class | Routes | Why | |---|---|---| | Workload identity | `/v1/identity/*` | Produced in your cluster, describes your workloads | | Policy edges | `/v1/policy-edges/*` | Pod IPs, namespaces, NetworkPolicy edges | | Fleet bans | `/v1/fleet-bans/*` | Your own agents' ban decisions | These three are tenant-scoped: they describe your infrastructure or your decisions. Relaying them would push that to a service you do not run. They keep the local object store path behind the local authorization check, in relay mode exactly as without it. Uploads are never relayed. The relay is read-only, so `PUT` and `POST` on the download surface are served locally or not at all. ## How authorization works Two hops, two credentials, checked in two places. | Hop | Credential | Checked by | |---|---|---| | Agent to relay | The agent's own key, issued by your dashboard | Your cluster. Relayed routes sit behind the local auth middleware and each carries the same permission it has in direct mode | | Relay to Gen0Sec | Your Gen0Sec API key | Gen0Sec | **A caller's key is never forwarded upstream on a self-hosted deployment.** A locally-issued key cannot authenticate to gen0sec. Forwarding it would leak a local credential to a third party without making the request succeed. With no Gen0Sec API key configured, the relay sends no credential at all. ## The setting that must be right ```yaml downloadProxy: enabled: true upstream: "https://api.gen0sec.com" apiKeySecret: name: gen0sec-download-proxy # named by default key: DOWNLOAD_PROXY_API_KEY ``` ### Why the key is required Your agents' keys are issued by **your** dashboard and exist only in **your** database, so they cannot authenticate to gen0sec. The relay presents your own Gen0Sec API key instead, and that is the only credential the upstream will accept from your installation. Without it, the relay sends no credential, Gen0Sec answers 401, and no relayed artifact reaches your fleet. :::note[The chart names the secret by default, so this fails loudly now] `apiKeySecret.name` defaults to `gen0sec-download-proxy`. If that secret does not exist, the relay pod never starts: `CreateContainerConfigError`, visible immediately. That is deliberate. An empty name renders no environment variable at all, the relay starts, and the only symptom is every relayed class answering 401 while agents report empty rulesets rather than errors. A failed rollout at install time is the cheaper failure. Blank the name only if you relay through a gateway that supplies its own credential. ::: ### Getting the key right ```bash printf '%s' '' > ~/.gen0sec/relay-key kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-file=DOWNLOAD_PROXY_API_KEY=$HOME/.gen0sec/relay-key ``` The value renders as a `secretKeyRef`, so it never appears in a manifest or in a values file. ## All settings | Key | Default | Meaning | |---|---|---| | `downloadProxy.enabled` | `true` | Turning it off is not supported | | `downloadProxy.upstream` | `https://api.gen0sec.com` | The upstream base URL. A malformed value **fails startup** rather than degrading | | `downloadProxy.external` | `geoip,ids-rules` | Which external classes to relay. An unknown name fails startup | | `downloadProxy.timeout` | `60s` | Upstream request timeout, matching the router's own | | `downloadProxy.apiKeySecret.name` | `""` | **Set this.** See above | | `downloadProxy.apiKeySecret.key` | `DOWNLOAD_PROXY_API_KEY` | The key within the secret | :::note[Failing startup is deliberate] A malformed upstream is not treated as "fall back to reading from the local object store". That would leave every agent poll erroring against a store this deployment does not populate. Failing fast is the safer wrong answer. ::: ## Verifying ```bash kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' ``` The output is similar to this: ```json { "status": "ready", "mode": "proxy", "upstream": "https://api.gen0sec.com", "external_relays": ["geoip", "ids-rules"] } ``` **That tells you the relay started, not that it is authorized.** For that, read the log: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means your Gen0Sec API key is missing or wrong. ## Response codes worth knowing | Code | Means | What an agent does | |---|---|---| | 200 | Relayed successfully | Uses the artifact | | **401** | The upstream rejected the credential | **Reports an empty ruleset, not an error.** This is the silent failure | | **502** | The upstream is unreachable | Reads it as "ask again later" and keeps what it holds. Correct degradation | | 404 | The artifact does not exist upstream | Treats it as withdrawn | The 401 and 502 rows are the reason this page exists. ## Preview a change before applying it ```bash helm template g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set downloadProxy.apiKeySecret.name=gen0sec-download-proxy \ | grep -A3 DOWNLOAD_PROXY ``` --- # Secrets Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/configure/secrets # Secrets **No chart contains a credential.** Every one is referenced by name. This page is explains what must exist, with which keys, before the platform starts. ## The secrets you create | Secret | Namespace | Keys | Created in | |---|---|---|---| | `gen0sec-registry` | **Both** | `.dockerconfigjson` | [Install step 2](./../05-install.mdx#the-image-pull-secret) | | `gen0sec-object-store` | `gen0sec-system` | `RUSTFS_ACCESS_KEY`, `RUSTFS_SECRET_KEY` | [Install step 2](./../05-install.mdx#the-object-store-credentials) | | `gen0sec-download-proxy` | `gen0sec` | `DOWNLOAD_PROXY_API_KEY` | [Install step 2](./../05-install.mdx#your-gen0sec-api-key) | The pull secret is needed in **both** namespaces. The application runs in `gen0sec`; the operators, object store and cache run in `gen0sec-system`. ## The twenty the script creates `make-db-secrets.sh` composes one secret per service from the passwords the Postgres operator generated. | Secret | Contains | Role it uses | Endpoint | |---|---|---|---| | `g0s-db-` | `DATABASE_URL`, plus object store keys for services that use it | Its own per-service role, read-only where it only reads | The connection pooler | | `gen0sec-db-migrate` | `DATABASE_URL` | `postgres`, superuser | The primary, unpooled | | `g0s-db-ui` | `DATABASE_URL`, `BETTER_AUTH_SECRET`, object store keys | `frontend` | The connection pooler | One more is optional and not created by the script: SMTP credentials for invitation and password-reset email, referenced from `services.ui.envSecrets`. See [Invitations and email](./identity.mdx#invitations-and-email). A referenced Secret that does not exist leaves the dashboard in `CreateContainerConfigError`. Every service connects as its own role. The migration job needs a superuser on the primary because it runs `CREATE EXTENSION`, which an owner role cannot do. :::note[The session signing key is preserved] `g0s-db-ui` carries the dashboard's session signing secret. Re-running the script keeps the existing value, so sessions and invite links stay valid across a re-run. Nothing else regenerates it. ::: Four services need no database connection and get no `DATABASE_URL`: the GeoIP updater, the GeoIP database updater, the model publisher and the rules validator. ## Where the database passwords come from The Postgres operator creates a credentials secret per role it provisions, named after the role: ``` .core.credentials.postgresql.acid.zalan.do ``` Underscores in the role become dashes in the secret name. Compose each connection string from that role's password: ``` postgres://:@core-pooler.gen0sec-system.svc.cluster.local:5432/ax_prod?sslmode=require ``` | Consumer | Role | Endpoint | |---|---|---| | Each service | Its own per-service role | `core-pooler` | | Migration job | `postgres` | `core`, the primary, with the on-prem marker appended | | Dashboard | `frontend` | `core-pooler` | :::note[One deliberate exception] The dashboard's connection string uses `sslmode=no-verify`. Its client library verifies the certificate chain for `sslmode=require` and rejects the pooler's self-signed certificate; the Go library every other service uses reads the same value as encrypt-without-verify. **Do not copy `no-verify` onto the other services.** Their driver rejects it outright. ::: ## For production: use your secret manager `make-db-secrets.sh` is fine for evaluation. In production, treat its output as a **specification** rather than a tool, and have your secret manager produce secrets of the same shape. Two reasons: 1. The script writes plain Kubernetes secrets from operator-generated passwords. Your manager should be the source of truth, with the cluster holding copies. 2. Rotation is then a manager operation rather than a re-run plus a rollout. An external secret can template the connection string from the operator-generated password, so nothing is copied by hand: ```yaml # shape, not a working manifest: adapt to your operator data: - secretKey: DATABASE_URL remoteRef: key: g0s-auth-api-ro.core.credentials.postgresql.acid.zalan.do property: password # then template: postgres://:{{ .password }}@core-pooler...:5432/ax_prod?sslmode=require ``` To see the exact list your install needs, read it out of the values file rather than from this page: ```bash grep -oE 'secretRef:\s*\{?\s*name:\s*[a-z0-9-]+' \ $B/values/gen0sec-platform-values-onprem.yaml | sort -u ``` Or inspect your live installation inside your cluster. ## Rotation | Credential | How | |---|---| | Database role passwords | Rotate through the Postgres operator, then restart the consumers | | Object store keys | Update **both** `gen0sec-object-store` and every `g0s-db-` that carries them, then restart the consumers | | Registry pull credentials | Recreate `gen0sec-registry` in both namespaces. Existing pods keep running; new pulls use the new value | | Your Gen0Sec API key | Recreate `gen0sec-download-proxy`, then restart the relay | | Dashboard session key | Rotating invalidates every session and every outstanding invite link. Passwords and passkeys are unaffected | | SMTP credentials | Update the Secret, then `kubectl -n gen0sec rollout restart deploy/g0s-ui` | :::note[One credential pair, surfaced two ways] The object store keys appear as `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` to the object store itself, and as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to the services that read it. Same values, two naming conventions. Rotating means updating both places. ::: ## Verifying Nothing should be missing before you install the platform: ```bash $KIT/scripts/preflight.sh ``` After install, confirm the count: ```bash kubectl -n gen0sec get secret | wc -l # ~20 service secrets plus the ones you created kubectl -n gen0sec-system get secret | grep credentials | wc -l # expect 6 ``` ## What is never in a secret | | Where it is instead | |---|---| | The registry host and namespace | The values file | | The relay upstream URL | The values file | | The cache connection string | The values file, deliberately with no credential in it | If you enable authentication on the cache, remove that line from the values file and put the connection string in each service's secret instead. --- # Values reference Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/configure/values # Values reference :::note[This page is the reference] It covers the keys you actually set, which is a much shorter list than every key a chart accepts, and states which of them are required. For anything not listed here, read the chart's own `values.yaml` in the bundle. There is no generated key-by-key reference. The charts carry no JSON schema yet, so nothing can be generated from one; when that changes this page will link to it. ::: ## How values are layered Three files, applied in order, last one wins. ``` chart values.yaml the chart's own defaults, registry deliberately blank values-onprem.yaml shipped in the bundle, points everything at a registry your overrides what you set ``` **Always pass `-f values-onprem.yaml`.** The default `values.yaml` leaves the registry blank on purpose, so installing without it cannot resolve images. Put your overrides in a file rather than in `--set` arguments, because `helm upgrade` uses only what you pass it. Anything set with `--set` last time and omitted this time reverts to the chart default. ```bash helm get values g0s -n gen0sec # what the running release actually has ``` ## gen0sec-infra Top-level keys: `postgres-operator`, `strimzi-kafka-operator`, `synapse-operator`, `synapse`, `rustfs`, `dragonfly`, `components`. | Key | Default | Set it when | |---|---|---| | `rustfs.storageclass.name` | `""`, meaning the cluster default | You want a specific StorageClass | | `rustfs.storageclass.dataStorageSize` | `50Gi` | Sizing the object store. **Four drives per replica** | | `rustfs.storageclass.logStorageSize` | `10Gi` | Rarely | | `rustfs.replicaCount` | `3` | Fewer nodes. Erasure coding assumes they spread | | `rustfs.clusterDomain` | `cluster.local` | Your cluster uses a custom DNS domain | | `dragonfly.extraArgs` | `[]` | Adding `--force_epoll` when `io_uring` is blocked | | `dragonfly.maxMemory` | Chart default | Keep it comfortably under the memory limit, or the kubelet kills the pod before it can evict | | `components.ui.namespace` | `gen0sec` | You install the platform chart in another namespace. Names the dashboard's service account for the Components tab | | `components.ui.serviceAccountName` | `g0s-ui` | You install the platform chart under a release name other than `g0s`. Must be `-ui` | See [Sizing storage for your retention](./../02-requirements.mdx#sizing-storage-for-your-retention). ## gen0sec-data Top-level keys: `postgres`, `kafka`, `kafka2pg`. | Key | Default | Set it when | |---|---|---| | `postgres.numberOfInstances` | `3` | Never below 3 in production | | `postgres.volume.size` | `50Gi` | Sizing the database. This is the system of record, so it is the one to grow first | | `postgres.volume.storageClass` | `""` | You want a specific StorageClass. **Fast block storage** | | `postgres.resources` | 1 vCPU / 2 GiB requested, 8 vCPU / 12 GiB limit | Tuning. The limit sets the per-node minimum | | `kafka.replicas` | `3` | Fewer nodes. **Drop the replication factors together**, or brokers refuse writes | | `kafka.storage.size` | `50Gi` | Sizing the message log and the ingest buffer | | `kafka.storage.class` | `""` | You want a specific StorageClass | :::danger[Kafka replicas and replication factor move together] `min.insync.replicas` must stay below `replicas`. Reducing brokers without reducing the factors gives you a cluster that schedules and then refuses every write. ::: ## gen0sec-platform Top-level keys: `global`, `migration`, `downloadProxy`, `services`. ### global | Key | Default | Notes | |---|---|---| | `global.registry.host` | `registry.gen0sec.com` | Rewritten by the bundler on the offline path | | `global.registry.namespace` | `gen0sec` | Every image lives under this | | `global.imageTag` | Empty | **Required.** Set it to the release version | | `global.imagePullSecrets` | `gen0sec-registry` | Must exist in both namespaces | | `global.commonEnv.S3_ENDPOINT` | The object store service URL | Must match the object store Service, which has a `-svc` suffix | | `global.commonEnv.ARXIGNIS_DATA_URL` | The same URL plus `/platform-data` | **The bucket name must match the bucket you created** | | `global.commonEnv.REDIS_URL` | The cache service URL | No credential in it, deliberately | | `global.podSecurityContext` | Non-root, uid 65532 | Do not weaken. See [Hardening](./../security/hardening.mdx) | | `global.securityContext` | Read-only root, no capabilities | Do not weaken | | `global.ingress.className` | `synapse` | The ingress controller the infra chart installs | | `global.commonEnv.CACHE_L1_TTL` | Unset, so `30s` | Default lifetime of an in-process cache entry, as a duration such as `45s`. An invalid value is ignored | ### Publishing `entrypointHost` is passed from the first install, and `externalScheme` by the [single-entrypoint overlay](./../06-publish.mdx). `values-onprem.yaml` sets neither and leaves every ingress disabled. Both are top level, and the chart applies both to every rule the overlay writes, which is why they are listed here rather than under `services`. | Key | Default | Notes | |---|---|---| | `entrypointHost` | `""` | **Required from the first install**, published or not. The one address that serves the dashboard and every agent-facing API, and the source of the dashboard's sign-in URL even before its Ingress exists. Empty fails the render and names the value to set, unless the dashboard is disabled or reached without an Ingress (see below). It must also be the address people actually type: the dashboard derives its sign-in URL from it | | `externalScheme` | Unset | Set to `https` **only when TLS terminates upstream** of the Ingress — a cloud load balancer, a CDN, a tunnel. With no `tls:` block to read, the chart otherwise infers `http` while the browser is on `https`, and every sign-in is rejected with `Invalid origin`. Leave it unset when TLS terminates at the Ingress | ### downloadProxy | Key | Default | Notes | |---|---|---| | `downloadProxy.enabled` | `true` | Turning it off is not supported | | `downloadProxy.apiKeySecret.name` | `""` | **Required. Empty is a silently broken deployment** | Full detail on [Data relay](./relay.mdx). This is the single most consequential key in the chart. ### services Twenty-one entries. Each one accepts the same shape, and you rarely need to touch any of it. | Key pattern | Notes | |---|---| | `services..enabled` | `threat-api` is not shipped at this release | | `services..replicaCount` | 2 or 3 in production, 1 in the quickstart overlay | | `services..resources` | Requests and limits. See [Sizing](./../02-requirements.mdx#sizing-a-production-install) | | `services..env` | Per-service environment, including `PORT` | | `services..ingress` | Off by default. Set by the entrypoint overlay | | `services..volumes`, `.volumeMounts` | **Do not remove these.** They exist because the read-only root filesystem needs a writable path where each service expects one | | `services.ui.enabled` | `true`. The dashboard | | `services.ui.env.BETTER_AUTH_URL` | Derived. Set it only when the dashboard is reached without an Ingress, such as through `kubectl port-forward` or your own proxy. It then wins over `entrypointHost` | | `services.ui.serviceAccount.automount` | `true`. The dashboard reads the installed-versions ConfigMaps for the Components tab. Every other service keeps `false` | | `services.ui.entraId.enabled` | `false`. Entra ID is in development and not supported at this release. See [Sign-in and identity](./identity.mdx) | :::danger[Removing a service's volume mount breaks that service] Several services default to writing into a path relative to their working directory, which fails on a read-only root filesystem. Each one has an explicit writable volume mounted where it writes. Those mounts are load-bearing, not leftovers. ::: ## Checking what a change will do Always render before you apply. ```bash helm template g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f my-overrides.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ | less ``` To diff against what is running: ```bash helm get manifest g0s -n gen0sec > running.yaml helm template g0s ... > proposed.yaml diff running.yaml proposed.yaml ``` ## Named starting points Two tested configurations rather than a blank page: [Reference configurations](./reference-configs.mdx). --- # First sign-in Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/first-signin ## Why this page exists Registration is invite-only, and invites can only be created by an existing member. A fresh install can be logged in by a pre-seeded admin account. ## The seeded account | | | |---|---| | Email | `admin@gen0sec.local` | | Password | `Gen0Sec!ChangeMe` | :::danger[This password is published, and it is the same on every install] It is a constant in our migration file, so anyone who has read our documentation knows it. The dashboard will not let you past it: the account is created with a must-change flag, so setting a new password is a step you cannot skip or postpone. What that does not do is protect the account between the moment you publish the address and the moment you first sign in. **Whoever signs in first is who sets the new password.** Sign in before you give the address to anyone else. If you have already shared it, sign in now and treat the interval as exposed. ::: ## 1. Sign in Open your published address and sign in with the details above. ```bash echo https://$ENTRYPOINT_HOST ``` If sign-in is rejected as an invalid origin, the published host or scheme does not match what you typed. See [Sign-in and identity](./configure/identity.mdx#the-published-address-decides-whether-sign-in-works). ## 2. Complete first-run setup The dashboard holds you at each of these in turn, and it will not let you skip ahead. A deep link to any other page returns you to the step you are on. 1. **Enroll multi-factor authentication.** Nothing else opens until you confirm a six-digit code. 2. **Set a new password.** Step 3 below. 3. **Create your organization.** 4. **Finish the remaining onboarding steps.** The wizard creates your first API key, then walks the WAF, access rule and monitoring surfaces. Multi-factor comes first, so the published password stops being enough to sign in before you have even reached the password screen. ## 3. Mandatory password change Because the password is preseeded, after setting up MFA, you'll be asked to change your password. :::note[The change signs out this account everywhere else] Every other session on the seeded account ends when the password changes. On a first install there are none, which is the point: if the address was reachable before you signed in, this is what closes that off. ::: The seed is also idempotent on the email address. Re-running the migration or the secret script neither duplicates the account nor re-flags it, and it does not reset a password you have already changed. ## 4. Invite the people who need access Invite your operators rather than sharing the seeded account. Under **Organization settings**, **Team**, pick the workspaces each person gets and a role for each. If [SMTP is configured](./configure/identity.mdx#invitations-and-email) the invitation is emailed. Otherwise the dashboard shows the link: copy it and send it yourself. Links expire after seven days. Invitees register with a passkey or a password, and enroll multi-factor like you did. ## 5. Mint an agent API key Create an agent API key in the dashboard, then save it where the verification step reads it from: ```bash mkdir -p ~/.gen0sec printf '%s' 'yor-agent-key' > ~/.gen0sec/agent-key ``` ## If it fails | Symptom | Cause | |---|---| | The seeded account does not exist | The migration ran without the on-prem marker. Secrets were provisioned without `make-db-secrets.sh` | | Sign-in rejected as an invalid origin | The published host or scheme does not match the address you typed | | The dashboard loads but onboarding will not advance | Check `kubectl -n gen0sec logs deploy/g0s-ui` | | The password is rejected | It is case-sensitive and contains a `!`. Watch your shell if you pasted it | | The new password is rejected as too weak | 12 characters, one symbol, and three of the four character classes. The server applies the same rules as the form | | The password screen appears again on a later sign-in | The change did not complete. The flag clears only on success | | An invitation link says it is no longer valid | It was revoked, replaced by a newer invitation, already used, expired, or issued before an upgrade from v0.1.x | --- # Install Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/install # Install ## Before you start - [ ] [Prepare the artifacts](./04-prepare-artifacts.mdx) complete - [ ] `$KIT`, `$B`, `$REGISTRY`, `$REGISTRY_USER`, `$REGISTRY_TOKEN` are set in your shell - [ ] `$G0S_DOWNLOAD_API_KEY` is set. Step 2 writes it into a secret, and an unset variable produces an empty one that fails silently - [ ] You are logged in to the registry - [ ] The address the dashboard will be reached on, in `$ENTRYPOINT_HOST`. Step 7 needs it, and [Publish the endpoints](./06-publish.mdx) publishes under it ```bash export ENTRYPOINT_HOST=cerebellum.example.internal ``` ## 1. Check the cluster ```bash $KIT/scripts/preflight.sh ``` This checks the prerequisites the charts assume but do not create: a reachable cluster, a StorageClass, and the namespaces and secrets created in step 2. On a fresh cluster it reports the namespaces and secrets as missing, which is expected at this point. Pass a StorageClass name to pin it instead of accepting the default: ```bash $KIT/scripts/preflight.sh gen0sec-system gen0sec my-block-storage ``` ## 2. Create the namespaces and the two bootstrap secrets ```bash for ns in gen0sec-system gen0sec; do kubectl create namespace $ns --dry-run=client -o yaml | kubectl apply -f - done ``` Written to converge, so re-running step 2 after a failure further down is safe rather than a wall of `AlreadyExists`. ### The image pull secret Both namespaces need it. The application runs in `gen0sec`; the operators, object store and cache run in `gen0sec-system`. ```bash for ns in gen0sec-system gen0sec; do kubectl -n $ns create secret docker-registry gen0sec-registry \ --docker-server=$REGISTRY \ --docker-username=$REGISTRY_USER \ --docker-password=$REGISTRY_TOKEN \ --dry-run=client -o yaml | kubectl apply -f - done ``` ### The object store credentials :::danger[Run this once, and only once] Unlike the rest of step 2 this does **not** converge. It mints new random keys every time it runs, and the object store's existing data was written with the old ones — recreating the secret on a second pass strands that data behind credentials nothing holds any more. The guard below stops before that happens. ```bash kubectl -n gen0sec-system get secret gen0sec-object-store >/dev/null 2>&1 \ && echo "already exists — skip to step 3" \ || echo "not present — run the block below" ``` ::: ```bash umask 077 S=$(mktemp -d) openssl rand -hex 16 | tr -d '\n' > "$S/RUSTFS_ACCESS_KEY" openssl rand -hex 32 | tr -d '\n' > "$S/RUSTFS_SECRET_KEY" kubectl -n gen0sec-system create secret generic gen0sec-object-store --from-file="$S" ``` **Save both files somewhere safe before you delete them.** Nothing regenerates these keys, and the data in the object store is written with them. ```bash mkdir -p ~/gen0sec-object-store-keys cp "$S"/RUSTFS_* ~/gen0sec-object-store-keys/ # or load them into your secret manager rm -rf "$S" ``` Step 5 reads them back from that directory. ### Your Gen0Sec API key The relay presents this key to Gen0Sec when it fetches threat intelligence, GeoIP, models and IDS rules. The chart expects this exact secret name, so the relay pod will not start without it. See [Credentials from gen0sec](./02-requirements.mdx#your-gen0sec-api-key). ```bash # Refuses to run rather than creating an empty secret. An empty key is accepted # by Kubernetes and by the chart, and only surfaces later as every relay fetch # returning 401 across the whole fleet — with nothing in the install log. : "${G0S_DOWNLOAD_API_KEY:?not set — see Prepare the artifacts, step 0}" kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-literal=DOWNLOAD_PROXY_API_KEY=$G0S_DOWNLOAD_API_KEY ``` ## 3. Install infra ```bash helm install g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml ``` This installs three operators (Postgres, Kafka, ingress), the object store and the shared cache. ### Wait for all five The next step creates resources these operators have to reconcile. Do not continue until every one is available. ```bash kubectl -n gen0sec-system rollout status deploy/g0s-infra-postgres-operator kubectl -n gen0sec-system rollout status deploy/strimzi-cluster-operator kubectl -n gen0sec-system rollout status deploy/g0s-infra-synapse-operator kubectl -n gen0sec-system rollout status deploy/dragonfly kubectl -n gen0sec-system rollout status \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" ``` The output for each is similar to this: ``` deployment "g0s-infra-postgres-operator" successfully rolled out ``` :::note[Why the last one resolves its own kind] The object store is a StatefulSet in the production configuration and a Deployment in the single-node configuration. Resolving the kind means the same command works in both. ::: ## 4. Install data ```bash helm install g0s-data $B/charts/gen0sec-data-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml ``` This creates the Postgres cluster, the Kafka cluster, its topics, and the database roles. ### Wait for both clusters Both take several minutes. Nothing after this works until Postgres reports `Running`. ```bash kubectl -n gen0sec-system wait --for=condition=Ready kafka/core --timeout=15m kubectl -n gen0sec-system wait \ --for=jsonpath='{.status.PostgresClusterStatus}'=Running postgresql/core --timeout=15m ``` ## 5. Create the object store bucket The object store starts empty and nothing creates the bucket for you. Create it now, before the platform install. Services that read or write the object store fail until it exists. This step is required on every install. The two datasets your cluster produces about itself, `identity` and `policy-edges`, live here and are never sent anywhere. See [Network and connectivity](./03-network.mdx#what-never-leaves-your-cluster). Run it as a throwaway pod in the cluster. The pod reads the keys straight from the `gen0sec-object-store` secret, so they never reach your shell and never touch your local AWS configuration. ```bash kubectl -n gen0sec-system run s3-init --rm -i --restart=Never \ --image=amazon/aws-cli:2.15.0 \ --overrides='{"spec":{"containers":[{"name":"s3-init","image":"amazon/aws-cli:2.15.0", "env":[{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"name":"gen0sec-object-store","key":"RUSTFS_ACCESS_KEY"}}}, {"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"name":"gen0sec-object-store","key":"RUSTFS_SECRET_KEY"}}}, {"name":"AWS_EC2_METADATA_DISABLED","value":"true"}, {"name":"AWS_REGION","value":"us-east-1"}], "args":["--endpoint-url","http://rustfs-svc.gen0sec-system.svc.cluster.local:9000","s3","mb","s3://platform-data"]}]}}' ``` Expect `make_bucket: platform-data`. `BucketAlreadyOwnedByYou` is also a pass, so this step converges: running it twice is harmless. :::note[Offline installs: mirror this one image first] `amazon/aws-cli` is **not** in the offline bundle. The bundle carries only what `release-manifest.yaml` pins, and its `SHA256SUMS` covers the whole tree — the bundler refuses a package containing anything the checksums do not list — so it cannot be added after the fact. Mirror it alongside the bundle, then point the pod at your copy: ```bash skopeo copy docker://amazon/aws-cli:2.15.0 docker://$REGISTRY/amazon/aws-cli:2.15.0 ``` Substitute `$REGISTRY/amazon/aws-cli:2.15.0` for `amazon/aws-cli:2.15.0` in both the `--image` flag and the `image` field of the overrides above. They must match: the `--image` flag sets the pod's image, and the override object sets the container's, and Kubernetes uses the latter. ::: Verify by running the same command with `"s3","ls"` in place of `"s3","mb","s3://platform-data"`. The bucket should be listed: ``` 2026-09-01 14:02:11 platform-data ``` An empty result means the bucket was not created, and the services in step 7 will fail with `500 NoSuchBucket`. Do not continue. :::note[If you changed the bucket name] `platform-data` is the value of `global.commonEnv.S3_BUCKET`. If you overrode it, use your name instead. ::: :::caution[Do not export the object-store keys into your shell] The in-cluster method above is recommended because the credentials never enter your shell. That matters most on managed Kubernetes. **On EKS, exporting them breaks `kubectl`.** The kubeconfig authenticates through an `aws eks get-token` exec plugin that reads `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` — the same two variables. Setting them to the object-store keys makes every later command in that shell fail with: ``` error: You must be logged in to the server (Unauthorized) ``` The error names kubectl, not the variables, which makes it hard to attribute. The in-cluster method avoids it entirely. If you use the workstation method below, scope the credentials to the one command with `env ...` rather than exporting them. **If the object store returns `401`,** the keys are not the ones this install generated. Keys left over from a previous install of the same cluster do this, and so does a stale copy in `~/gen0sec-object-store-keys/`. Credentials are genuinely validated: wrong keys return `InvalidAccessKeyId`, and none at all returns `403`. :::
Workstation method, for reference This needs no S3 client image in the cluster, which is why it was preferred: on the offline bundle method an in-cluster image has to be mirrored first, so the bucket comes to depend on a step that exists only to create the bucket. Use it if that trade-off suits you — but read the caution above first, because exporting the object-store keys will break `kubectl` on EKS. ### 5a. Open a port forward In a second terminal: ```bash kubectl -n gen0sec-system port-forward svc/rustfs-svc 9000:9000 ``` Leave it running. The output is similar to this: ``` Forwarding from 127.0.0.1:9000 -> 9000 ``` ### 5b. Create the bucket Back in your first terminal, use the two keys you generated in step 2. Scope them to the one command with `env` rather than exporting them, so they cannot clobber the variables your kubeconfig's exec plugin reads: ```bash env AWS_ACCESS_KEY_ID="$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ AWS_SECRET_ACCESS_KEY="$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" \ AWS_REGION=us-east-1 AWS_EC2_METADATA_DISABLED=true \ aws --endpoint-url http://localhost:9000 s3 mb s3://platform-data ``` The output is similar to this: ``` make_bucket: platform-data ``` `BucketAlreadyOwnedByYou` is also a pass. With the MinIO client instead: ```bash mc alias set g0s http://localhost:9000 \ "$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ "$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" mc mb g0s/platform-data mc ls g0s ``` ### 5c. Verify it exists ```bash env AWS_ACCESS_KEY_ID="$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ AWS_SECRET_ACCESS_KEY="$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" \ AWS_REGION=us-east-1 AWS_EC2_METADATA_DISABLED=true \ aws --endpoint-url http://localhost:9000 s3 ls ``` The output is similar to this: ``` 2026-09-01 14:02:11 platform-data ``` An empty result means the bucket was not created, and the services in step 7 will fail with `500 NoSuchBucket`. Do not continue. Then stop the port forward with `Ctrl-C` in the second terminal.
## 6. Create the application secrets ```bash $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml ``` This composes around twenty secrets from the passwords the Postgres operator generated in step 4. That is why it cannot run earlier. Each service gets its own database role through the connection pooler. The migration job gets a superuser connection direct to the primary, because it runs `CREATE EXTENSION`. :::note[In production, treat this script's output as a specification] `make-db-secrets.sh` is fine for evaluation. For production, have your secret manager produce secrets of the same shape. Every name and key is listed in [Secrets](./configure/secrets.mdx). ::: ## 7. Install the platform ```bash helm install g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` **`entrypointHost` is required here**, before anything is published. The dashboard builds its sign-in URL from it, and without it the render fails with `entrypointHost is required`. Use the address people will type, the one you publish under in the next step. :::note[Nothing to set for the relay] The chart already points at `gen0sec-download-proxy` by default. You created that secret in step 2, and if it is missing the relay pod fails to start with `CreateContainerConfigError` rather than coming up half-configured. ::: Schema migrations run first, as a pre-install hook. If they fail, Helm aborts and nothing is deployed. That is deliberate: a partially migrated database is worse than no install. ### Check the migration ```bash kubectl -n gen0sec get job g0s-db-migrate ``` The output is similar to this: ``` Error from server (NotFound): jobs.batch "g0s-db-migrate" not found ``` **`NotFound` is the success case.** The job is a Helm hook with `hook-delete-policy: hook-succeeded`, so Helm deletes it the moment it passes. If the job is still there, it may have failed: ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` ### Wait for the pods ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m ``` :::note[Check readiness, not phase] A pod in `CrashLoopBackOff` still reports `status.phase: Running`. A `--field-selector` on phase reports a broken install as clean. ::: :::note[Two values that must match your cluster] Both are set in `values-onprem.yaml` and both point at the in-cluster object store. Check them if anything cannot read its data. | Key | Must be | |---|---| | `global.commonEnv.S3_ENDPOINT` | `http://rustfs-svc.gen0sec-system.svc.cluster.local:9000` | | `global.commonEnv.ARXIGNIS_DATA_URL` | The same endpoint, plus the bucket: `.../platform-data` | `ARXIGNIS_DATA_URL` is where in-cluster services read their datasets from. It points at your object store, not at Gen0Sec. **The bucket name at the end of it must match the bucket you created in step 5.** If you used a different name there, change it here too. The Service is `rustfs-svc`, with the `-svc` suffix. That suffix is the single most common cause of services that come up and then cannot read anything. ::: ## If it fails | Symptom | Start here | |---|---| | `helm install` fails with `entrypointHost is required` | `entrypointHost` was not passed in step 7. Set it and re-run | | `ImagePullBackOff` | [ImagePullBackOff](./troubleshoot/symptoms.mdx#imagepullbackoff) | | Install aborts at `g0s-db-migrate` | [Migration failures](./troubleshoot/symptoms.mdx#migration-failures) | | `postgresql/core` never reaches `Running` | [Postgres will not start](./troubleshoot/symptoms.mdx#postgres-will-not-start) | | Kafka broker crash loops | [Kafka will not start](./troubleshoot/symptoms.mdx#kafka-will-not-start) | --- # Network and connectivity Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/network # Network and connectivity ## Why the platform must reach Gen0Sec No platform is useful without data, neither is Gen0Sec. Because the datasets are licensed per customer, they are not in the install kit, not in the offline bundle, and not in any container image. The platform fetches them at runtime from `api.gen0sec.com`. | Dataset | Licensed by | |---|---| | CTI indicators | Upstream threat intelligence providers | | GeoIP databases | MaxMind, IPInfo | | IDS rules | Emerging Threats ruleset | | ML models | Trained and published by Gen0Sec | One in-cluster service, `download-api`, does the fetching. It relays each request upstream and streams the response straight back. It is a relay, not a cache, so there is no artifact bucket to size, no sync job to supervise and no staleness window. An agent that polls gets whatever upstream has at that moment. This also means you get the most recent data we can provide. ## One hop, not many Your agents never talk to the internet. They talk to `download-api`, and `download-api` is the only service in either namespace that needs a route out. The dashboard makes optional calls of its own, listed in the [egress list](#egress-list); none of them carries agent data. ``` agents -> download-api -> api.gen0sec.com:443 ^ | the only egress point in the deployment ``` That means one destination to allow, one to audit. It also means the relayed URLs are identical to the direct ones, so agents need no configuration change. ## What never leaves your cluster Two datasets describe your own infrastructure, and they are never relayed: | Dataset | What it contains | Where it stays | |---|---|---| | `identity` | Workload identity produced in your cluster | Your object store | | `policy-edges` | Pod IPs, namespaces, NetworkPolicy edges | Your object store | No hidden switches, no configuration options, it is not implemented. Your data is yours and yours only. Uploads are not relayed either. The relay is read-only, so `PUT` and `POST` on the download surface are served locally or not at all. ## How authorization works **Your agents' API keys never leave your cluster.** Agents authenticate to your installation with keys your own dashboard issued, held in your own database. On a correct configuration the relay does not forward those keys upstream. It presents **your own Gen0Sec API key**, the one issued with your subscription, on the second hop only. ``` agent --[your agent's key]--> relay --[your Gen0Sec API key]--> api.gen0sec.com agent key stops here ``` | | | |---|---| | What Gen0Sec sees | One credential, belonging to your installation | | What Gen0Sec never sees | Any individual agent key, or how many you have issued | | Revoking an agent | Entirely local, in your dashboard | | Revoking your installation's access | Revoke or rotate your Gen0Sec API key | :::danger[This key is required, and omitting it fails silently] With no Gen0Sec API key configured, the relay falls back to forwarding the caller's own key. That is correct for the hosted product, where the same platform issued it. Here it is always wrong: your keys exist only in your cluster, so Gen0Sec rejects every one with a 401, and **the agent reports an empty ruleset rather than an error**. The install may look healthy, but the fleet quietly stops receiving threat intelligence, GeoIP, models and IDS rules. Set up in [Install](./05-install.mdx) step 2, enabled in step 7, and checked by [Verify the install](./08-verify.mdx). ::: Routes that are not relayed, including `identity` and `policy-edges`, keep the local authorization check against your own database. ## Egress list All connections are outbound, TCP, TLS. Nothing needs an inbound rule from the internet. | Destination | Port | Needed by | Purpose | Required | |---|---|---|---|---| | `api.gen0sec.com` | 443 | Cluster pods (`download-api` only) | Threat intelligence, GeoIP, ML models, IDS rules | **Always** | | `releases.gen0sec.com` | 443 | Your workstation | Download the install kit and its signature | **Always** | | `registry.gen0sec.com` | 443 | Your workstation | `helm pull` for charts, `skopeo copy` for images | **Always** | | `registry.gen0sec.com` | 443 | Cluster nodes (kubelet) | Image pulls at install time | Registry install only | | `fulcio.sigstore.dev` | 443 | Your workstation | `cosign` certificate verification | **Always** | | `rekor.sigstore.dev` | 443 | Your workstation | `cosign` transparency log verification | **Always** | | `tuf-repo-cdn.sigstore.dev` | 443 | Your workstation | `cosign` trust root refresh | **Always** | | Your own registry | 443 or 80 | Cluster nodes, and the host that runs `bundler.sh apply` | Image pulls at install time | Offline bundle only | | `helm.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest agent version, shown on the Clients and Components pages | No. Always attempted. Blocking it only hides the latest version | | `releases.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest component versions | No. Only if an administrator turns on **Check for updates online** | | Your SMTP server | 587 or 465 | Cluster pods, the dashboard only | Invitation and password-reset email | No. Only if you [configure SMTP](./configure/identity.mdx#invitations-and-email) | {/* ASSUMPTION: this list is derived from the release workflow, the bundler and the chart values. Confirm it is complete, and confirm the Sigstore host set, which changes with cosign versions. */} :::note[The Sigstore hosts are the ones people forget] `cosign verify-blob` checks a signature against a public transparency log. Without those three destinations it fails, and it fails on the first command of the install. Allow them on the workstation, not on the cluster. ::: ### Inside the cluster No NetworkPolicy is required for the install. If you enforce a default-deny policy, the two namespaces need to reach each other: the application in `gen0sec` talks to Postgres, Kafka, the object store and the cache in `gen0sec-system`. ## What this rules out `api.gen0sec.com` on port 443 is required for the platform to serve data. An agent fleet with the platform hosted on a cluster that is permitted no outbound connections at all cannot function correctly. The offline bundle method removes every dependency on gen0sec-hosted registries: your cluster pulls images only from your own. It does not remove the runtime connection, because the mentioned datasets above are not ours to ship. If your policy allows no cluster egress, talk to us. We would rather hear it now than during your evaluation. > **Air-gapped.** Often used to mean "no connection of any kind". Under that definition Cerebellum > is not deployable. If it means "no dependency on vendor-hosted registries and one audited > outbound destination", the offline bundle method fits, and this page is the list your firewall team > needs. --- # Back up and restore Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/operate/backup # Back up and restore ## What to back up Not everything in the deployment is equally valuable. Backing up the wrong things is how backup jobs become expensive and untested. | Component | Back up | If you lose it | |---|---|---| | **Postgres** | **Yes, always** | Everything. Configuration, agents, policy, history, the audit log. This is the system of record | | **Object store** | **Yes** | The datasets and the artifacts your cluster produced about itself. Datasets are re-fetchable, `identity` and `policy-edges` are not | | Kafka | No | In-flight messages only. It is a transport, not a store. Producers re-send | | Shared cache | No | Nothing. It repopulates | | Secrets | **Yes** | Access to your own volumes. See the warning below | :::danger[Back up the volumes along with the secrets, or the volumes are useless] The database passwords are generated by the Postgres operator and stored as Kubernetes secrets. Data is written with those credentials. A volume backup without the matching secret backup restores data you cannot open. ::: ## Postgres The Postgres operator ships a logical backup image, already mirrored as part of the release. Enable it in the infra values rather than scripting `pg_dump` yourself. ```yaml # gen0sec-infra values postgres-operator: configLogicalBackup: logical_backup_schedule: "0 2 * * *" logical_backup_provider: s3 ``` Verify a backup ran: ```bash kubectl -n gen0sec-system get cronjob | grep logical-backup kubectl -n gen0sec-system get job | grep logical-backup ``` :::note[A backup you have not restored is not a backup] Restore into a scratch namespace once, before you need to. The [Quickstart](./../quickstart.mdx) cluster is a reasonable place to rehearse it, and rehearsing a restore is also a rehearsal of an upgrade rollback. ::: ## Object store Two prefixes matter, and they matter differently. | Prefix | Recoverable without a backup? | |---|---| | `identity/`, `policy-edges/` | **No.** Produced by your cluster about your cluster | | Dataset prefixes | Yes. They are fetched from Gen0Sec | Back up the whole bucket with any S3-compatible tool, with an in-cluster job or manually, over a port forward as in [Install step 5](./../05-install.mdx#5-create-the-object-store-bucket): ```bash kubectl -n gen0sec-system port-forward svc/rustfs-svc 9000:9000 ``` ```bash aws --endpoint-url http://localhost:9000 s3 sync s3://platform-data ./platform-data-backup ``` Restore is the same command with the arguments reversed. ## Secrets ```bash kubectl -n gen0sec-system get secret -o yaml > secrets-gen0sec-system.yaml kubectl -n gen0sec get secret -o yaml > secrets-gen0sec.yaml ``` **These files contain live credentials in base64.** Treat them exactly as you treat the database itself: encrypted at rest, access controlled, and out of any shared drive. If you run an external secret manager, you do not need this: your manager is already the source of truth and the cluster holds copies. See [Secrets](./../configure/secrets.mdx). ## Recovery point and recovery time | | Value | Determined by | |---|---|---| | **Recovery point** | Your backup interval | The logical backup schedule. Nightly gives up to 24 hours of loss | | **Recovery time** | Measure it | Database size and your storage throughput. Rehearse it and record your own figure | We do not publish an RTO as it depends entirely on your configuration specifics and the data size. Time your own rehearsal and use that number in your own runbook. ## Restoring Order matters, and it is the install order. 1. Install infra and data as on a [first install](./../05-install.mdx), so the operators and empty clusters exist. 2. Restore the secrets, before anything tries to use the volumes. 3. Restore Postgres from the logical backup. 4. Restore the object store bucket. 5. Install the platform. **The migration hook runs against restored data**, so the restored schema must match the release you are installing. Restore into the version it came from, then upgrade. 6. Run the smoke test: `$KIT/scripts/verify-deployment.sh`. :::note[Do not restore an old database into a new release] The migration chain expects to run forward from the schema version in the dump. Restoring a v0.1 backup and installing v0.3 skips the intermediate migrations. Restore v0.1, then upgrade one minor at a time. See [Upgrade](./upgrade.mdx). ::: --- # Monitor Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/operate/monitor # Monitor ## What the platform exposes today | Signal | Where | Notes | |---|---|---| | Health and readiness | `/status/healthz` and `/status/readyz` on each service | Three services serve only `healthz`. See [Known limitations](./../troubleshoot/known-limitations.mdx#three-services-report-readiness-from-a-weaker-probe) | | Relay mode and upstream | `/status/healthz` on the relay | Returns the mode, the upstream and the active relay classes | | Prometheus metrics | Worker port 8080 | Exposed, but no Service is created for workers, so scraping needs a direct pod target | | Kubernetes state | Pods, Deployments, custom resources | The primary signal | | Block events refused by the database | Kafka topic `block_events_dlq`, 14 days retention | Each message carries the reason in its `dlq_reason` header. Steady growth means events are being lost | | Block events dropped at ingest | telemetry-api log line `OTLP block events rejected, or stored without a field they carried` | At most once a minute per organization, with counts per reason | | Installed and latest versions | Dashboard, **Organization settings**, **Components** | See [Component versions](./../configure/components.mdx#verifying-what-you-are-actually-running) | | Who changed what | Dashboard, **Organization settings**, **Audit Log** | Owner and Admin roles only | **No ServiceMonitor, alert rules or dashboards ship with the charts at the moment.** If you run Prometheus, you build the scrape configuration. ## The four checks worth alerting on If you set up nothing else, set up these. ### 1. Pods not ready ```bash kubectl -n gen0sec get pods --no-headers \ | awk '{split($2,r,"/"); if (r[1] != r[2]) print}' ``` Any output is a problem. **Read the ready column, never a field selector on phase:** a pod in `CrashLoopBackOff` still reports `status.phase: Running`. ### 2. The database cluster is not Running ```bash kubectl -n gen0sec-system get postgresql core \ -o jsonpath='{.status.PostgresClusterStatus}' ``` Expect `Running`. Anything else and nothing downstream works. ### 3. The relay lost its upstream ```bash kubectl -n gen0sec exec deploy/g0s-download-api -- \ curl -fsS localhost:9999/status/healthz ``` Expect `"status": "ready"` and `"mode": "proxy"`. An unreachable upstream answers **502** on relayed routes rather than 404 or 500. That is deliberate: a polling agent reads 502 as "ask again later" and keeps the artifact it already holds, instead of treating it as withdrawn. So a broken upstream degrades your fleet slowly rather than immediately, which is safer and also easier to miss. Alert on it. ### 4. The platform stopped serving agents ```bash $KIT/scripts/verify-deployment.sh ``` Read-only, safe against production, exits non-zero and names the failing check. Running it on a schedule is the single most useful synthetic check available, because it is the only one that proves data actually moves. ## Signals that are noisy on purpose Do not page on these. | Signal | Why it is not an incident | |---|---| | `falling back to L1-only cache` | The shared cache is never fatal. Services run on their in-process cache. Reads get slower, nothing breaks | | A 502 from a relayed route, once | Upstream hiccup. Agents retry and keep what they have | | The migration job absent | That is success. Helm deletes it on completion | | `unknown_source` in a telemetry-api warning | An agent newer than the platform. The event is stored, without its source | ## Logs ```bash kubectl -n gen0sec logs deploy/ --tail=200 kubectl -n gen0sec-system logs deploy/g0s-infra-postgres-operator --tail=200 ``` All services log to standard output as structured JSON, so any collector that reads container stdout works without configuration. Log level is set per service through the values file. See [Values reference](./../configure/values.mdx). ## What a healthy install looks like Five statements, the same five as [Verify the install](./../08-verify.mdx#definition-of-installed-successfully). If you build a dashboard, build it to answer these: 1. Every pod in both namespaces is `Running` and fully ready. 2. `postgresql/core` reports `Running`. 3. `kafka/core` reports `Ready`. 4. The relay reports `"mode": "proxy"` with your upstream. 5. The smoke test exits zero. --- # Roll back Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/operate/rollback # Roll back Read the boundary section first. It decides whether a rollback is safe or whether you need a restore. ## The boundary, stated plainly **`helm rollback` reverts manifests. It does not revert your database.** That single sentence is the whole risk. Helm restores the previous Deployments, images and configuration. Any schema change the upgrade's migration hook applied is still applied. If the old application cannot read the new schema, rolling back the manifests gives you old code against a new database. | Situation | Rollback is | |---|---| | Patch upgrade within a train, no migration ran | Safe. Just do it | | Minor upgrade whose migrations only added things | Safe. Old code ignores what it does not know | | Minor upgrade whose migrations removed or renamed something the old code reads | **Not safe.** Restore from backup instead | | You are not sure which | Check the release notes. If they do not say, ask us before rolling back | :::note[v0.2.0 back to v0.1.x] Rolling back the platform alone is safe: every v0.2.0 migration only adds, so v0.1.x runs against the v0.2.0 schema. **Do not run the down migrations.** There are 18, and they delete the audit log, registered passkeys and pending invitations. While rolled back, do not `helm upgrade` the v0.1.x chart: its migration job does not know the v0.2.0 schema version and fails with `no migration found for version`. ::: :::note[Why we prefer expand and contract] Migrations that add in one train and remove in a later one mean a one-release rollback never needs a schema change. That is the design intent, and it is what makes most rollbacks safe. It is not a guarantee for every release, which is why this page exists rather than a one-line reassurance. ::: ## Before you start - [ ] You know which revision you are going back to - [ ] You have read the release notes for the version you are leaving - [ ] The previous release's images are still in your registry **Offline bundle method:** mirroring is additive, so the old tags are still there unless you deleted them. Do not prune your registry between an upgrade and the point at which you are confident. ## 1. Find the revision ```bash helm history g0s -n gen0sec ``` The output is similar to this: ``` REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Mon Sep 1 10:14:02 2026 superseded gen0sec-platform-0.1.0 0.1.0 Install complete 2 Mon Sep 1 15:41:18 2026 deployed gen0sec-platform-0.2.0 0.2.0 Upgrade complete ``` ## 2. Roll back the platform Start here. In most cases this is the only step you need, because the application is what broke. ```bash helm rollback g0s 1 -n gen0sec ``` ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` If the smoke test passes, stop. Do not roll back infra or data unless they are the problem. ## 3. Roll back the schema, only if you must Needed when the upgrade applied a change the old code cannot work with. Check the release notes first. ```bash kubectl -n gen0sec run db-migrate-down --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/db-migrate: \ --env DATABASE_URL="$OWNER_DSN" -- down ``` Use the `db-migrate` image of the release you are **leaving**, not the one you are returning to. Only it contains the down migrations for its own schema; an older image fails with `no migration found for version `. `down` reverts one migration per run, so run it once per migration the release added. **Never run `down all`**: it reverts the entire schema. `$OWNER_DSN` is the superuser connection string from the `gen0sec-db-migrate` secret. See [Secrets](./../configure/secrets.mdx). :::danger[A down migration can lose data] Reversing a migration that dropped a column recreates the column, not its contents. If the upgrade removed data, only a restore brings it back. See [Back up and restore](./backup.mdx). The v0.2.0 down migrations drop the audit log, passkeys and pending invitations, and clear the `fleet_ban` source from stored block events. One of them also decompresses recent block events and locks that table while it runs. ::: ## 4. Roll back infra and data, only if they are the problem Same order as the upgrade, in reverse: platform, then data, then infra. ```bash helm rollback g0s-data 1 -n gen0sec-system helm rollback g0s-infra 1 -n gen0sec-system ``` :::note[CRDs do not roll back] Helm never manages CRDs after the first install, in either direction. A rolled-back operator runs against the newer CRD schema. That works when the change was additive, which is the usual case, and it is another reason to roll back the platform alone where possible. ::: ## When a restore is the right answer instead Rollback is for "the new version misbehaves". Restore is for "the data is wrong". | You want to | Use | |---|---| | Undo a bad release | Rollback | | Undo a destructive migration | Restore | | Recover from data corruption | Restore | | Return to a point in time | Restore | [Back up and restore](./backup.mdx). ## If it fails | Symptom | Cause | |---|---| | `ImagePullBackOff` after rollback | The previous version's images were pruned from your registry | | Old pods start, then error on database queries | Schema mismatch. Go to step 3, or restore | | `helm rollback` reports no such revision | `helm history` was truncated. `helm get manifest g0s --revision N` to confirm what exists | ## Next [Uninstall](./uninstall.mdx), or [Back up and restore](./backup.mdx). --- # Rotate credentials Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/operate/rotate # Rotate credentials Credentials rotates differently, and some of them have a consequence worth knowing before you start. | Credential | Rotating it interrupts | Reversible | |---|---|---| | [Database role passwords](#database-role-passwords) | The services that use them, briefly | Yes | | [Object store keys](#object-store-keys) | Everything that reads the object store | Yes, **if you kept the old values** | | [Registry pull credentials](#registry-pull-credentials) | Nothing running. New pulls only | Yes | | [Your Gen0Sec API key](#your-gen0sec-api-key) | Artifact delivery, briefly | Yes | | [Dashboard session signing key](#dashboard-session-signing-key) | **Every session and every invite link** | No | If you run an external secret manager, rotation happens there and the cluster picks up the new value. This page is for installs holding plain Kubernetes secrets. See [Secrets](./../configure/secrets.mdx). ## Database role passwords Rotate through the Postgres operator, not by editing the secret. The operator owns these values, and an edited secret is overwritten on its next reconciliation. After the operator has regenerated the credentials, recompose the application secrets and restart the consumers: ```bash $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy ``` The script preserves the dashboard's session signing key, so re-running it does not sign people out. ## Object store keys These are the ones you generated during install, and **nothing regenerates them**. Rotating means updating two places. ```bash umask 077 S=$(mktemp -d) openssl rand -hex 16 | tr -d '\n' > "$S/RUSTFS_ACCESS_KEY" openssl rand -hex 32 | tr -d '\n' > "$S/RUSTFS_SECRET_KEY" kubectl -n gen0sec-system delete secret gen0sec-object-store kubectl -n gen0sec-system create secret generic gen0sec-object-store --from-file="$S" ``` Then restart the object store, and recompose the application secrets, because the services receive the same values under different names: ```bash kubectl -n gen0sec-system rollout restart \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy ``` Save the new files where you kept the old ones, then remove the temporary directory. :::danger[Do not lose these] Data in the object store was written with them. A retained volume without the matching keys is data you cannot open. See [Back up and restore](./backup.mdx). ::: ## Registry pull credentials Recreate the secret in **both** namespaces. Running pods keep running; the new value applies to the next pull. ```bash umask 077 AUTH=$(printf '%s:%s' "$REG_USER" "$(cat $REG_PASS_FILE)" | base64 | tr -d '\n') printf '{"auths":{"%s":{"auth":"%s"}}}' "$REGISTRY" "$AUTH" > dockerconfig.json unset AUTH for ns in gen0sec-system gen0sec; do kubectl -n $ns delete secret gen0sec-registry kubectl -n $ns create secret generic gen0sec-registry \ --type=kubernetes.io/dockerconfigjson \ --from-file=.dockerconfigjson=dockerconfig.json done rm -f dockerconfig.json ``` Rotate before the old credential expires, not after. An expired pull secret is invisible until something reschedules. ## Your Gen0Sec API key ```bash printf '%s' '' > $RELAY_KEY_FILE kubectl -n gen0sec delete secret gen0sec-download-proxy kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-file=DOWNLOAD_PROXY_API_KEY=$RELAY_KEY_FILE kubectl -n gen0sec rollout restart deploy/g0s-download-api ``` Confirm it worked, because a wrong key fails silently: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means the new key is wrong, or has a trailing newline. Use `printf`, not `echo`. See [Data relay](./../configure/relay.mdx). ## Dashboard session signing key :::danger[This signs everyone out] Rotating it invalidates every active session and every outstanding invite link. There is no partial rotation and no grace period. ::: Only do this if you believe the key is compromised. Delete it and let the secret script regenerate it: ```bash kubectl -n gen0sec delete secret g0s-db-ui $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy/g0s-ui ``` The script preserves an existing key, which is why deleting the secret first is the step that actually rotates it. ## Certificates | Certificate | Owned by | |---|---| | Ingress TLS | You. Rotate in your ingress or through cert-manager | | Postgres pooler | The Postgres operator, self-signed, rotated by it | | Relay to Gen0Sec | Public certificate on our side. Nothing to do | The platform presents no certificate of its own outside the ingress you configure. ## After any rotation ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` The smoke test is the only check that proves the platform still serves an agent rather than merely starting. --- # Uninstall Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/operate/uninstall # Uninstall ## What you are about to delete | | Default | `--keep-data` | `--keep-crds` | |---|---|---|---| | The three Helm releases | Removed | Removed | Removed | | Objects the operators created outside the releases | Removed | Removed | Removed | | Kafka topic finalizers | Cleared | Cleared | Cleared | | Namespaces `gen0sec` and `gen0sec-system` | **Deleted** | Kept | Deleted | | Volumes: Postgres, Kafka, object store | **Deleted** | Kept | Deleted | | Secrets, including the database passwords | **Deleted** | Kept | Deleted | | Operator CRDs | **Removed** | Removed | Kept | :::danger[The default deletes your data, and it asks you to prove you mean this cluster] Without `--keep-data` the databases and the object store go, and nothing can undo that. So that run also **asks you to type the current context name back**, and refuses to run at all with no terminal attached. `--keep-crds` does not trigger that prompt, because a schema the next install reinstalls anyway is no reason to skip asking about the databases. ::: ## Remove everything ```bash $KIT/scripts/uninstall.sh --yes ``` It prints the cluster context, the release list and the volume count, then waits for you to type the context name. Read what it prints before you type. ## Keep the data ```bash $KIT/scripts/uninstall.sh --yes --keep-data ``` Stops after clearing the topic finalizers. The namespaces stay, and so do the volumes and the secrets in them. No context confirmation, because nothing irreversible happens. :::note[Why the secrets have to stay with the volumes] A reinstall over retained volumes needs the original database passwords: the data was written with them and a fresh Postgres operator generates different ones. Keeping the volumes without the secrets leaves you with data you cannot open. ::: ## Keep the CRDs ```bash $KIT/scripts/uninstall.sh --yes --keep-crds ``` Use this when something else in the cluster runs the same operators, or when you want the next install to skip reinstalling schemas. ## How CRD removal is made safe CRDs are cluster-scoped, and **deleting one deletes every resource of that kind in every namespace**. Another Postgres or Kafka cluster run by the same operators would go with them. So the script checks first: if any of those kinds is in use outside the two namespaces it is removing, **it skips the CRDs** and tells you. `--keep-crds` skips them unconditionally. That check is why removing CRDs by default is safe. It is not a blanket delete. If you want to see what would be affected before you run anything: ```bash kubectl get postgresqls,kafkas,kafkatopics,kafkanodepools -A 2>/dev/null | grep -v 'gen0sec' ``` Any output there means the script will leave the CRDs alone. ## The Kafka finalizer, and why order matters Strimzi puts a `strimzi.io/topic-operator` finalizer on all 23 KafkaTopics. Once the operator is gone, nothing clears it, and deleting the namespace leaves it in `Terminating` indefinitely. The script uninstalls the releases first, then clears the finalizers, then deletes the namespaces. That order is the whole reason the script exists. If you already deleted the namespace and it is stuck, this is the fix: ```bash NS=gen0sec-system kubectl get kafkatopics.kafka.strimzi.io -n $NS -o name \ | xargs -r -n1 kubectl patch -n $NS --type=merge \ -p '{"metadata":{"finalizers":null}}' ``` ## Reinstalling over retained volumes Two things bite here. **Kafka.** The volumes survive on purpose, and a recreated cluster generates a new identity. Brokers then crash loop with `Invalid cluster.id`. See [Kafka will not start](./../troubleshoot/symptoms.mdx#kafka-will-not-start). **Postgres.** You need the original role passwords. If you kept the secrets, the operator adopts the existing data. If you did not, the volumes are unreadable. ## Uninstalling by hand The script does more than these commands: it also removes objects the operators created outside the releases, and it handles the CRD safety check. Use it if you can. ```bash helm uninstall g0s -n gen0sec helm uninstall g0s-data -n gen0sec-system helm uninstall g0s-infra -n gen0sec-system # clear the finalizers before touching the namespaces kubectl get kafkatopics.kafka.strimzi.io -n gen0sec-system -o name \ | xargs -r -n1 kubectl patch -n gen0sec-system --type=merge \ -p '{"metadata":{"finalizers":null}}' kubectl delete namespace gen0sec kubectl delete namespace gen0sec-system ``` Do not pass `--wait` to `helm uninstall`. The Kafka topics cannot finish deleting until the finalizers above are cleared, so waiting only burns the timeout. --- # Upgrade Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/operate/upgrade # Upgrade When this page is done you are running the new release, with the same data. Read [Roll back](./rollback.mdx) before you start. The rollback boundary changes what a safe upgrade looks like. ## Before you start - [ ] Read the release notes for every version between yours and the target - [ ] A current backup. See [Back up and restore](./backup.mdx) - [ ] The new release's install kit, verified - [ ] A maintenance window. The application is briefly unavailable while pods roll ## Rules that are not negotiable | Rule | Why | |---|---| | One minor at a time | Skipping a minor is not supported. Each train's migrations assume the previous train ran | | Patches in place | Within a train, upgrade directly | | Same order as install: infra, then data, then platform | Each chart depends on the previous one | | Apply CRDs by hand first | Helm never upgrades CRDs. See step 1 | | Back up first | The migration hook changes your schema, and `helm rollback` does not undo that | ## 1. Apply the new CRDs **Do this before anything else.** Helm installs a chart's CRDs on first install and ignores them from then on. An upgrade that ships a new CRD schema fails with a server-side-apply error, or worse, silently rejects fields it does not know. ```bash cd $B/charts tar xzf gen0sec-infra-*.tgz kubectl apply --server-side --force-conflicts \ -f gen0sec-infra/charts/postgres-operator/crds/ kubectl apply --server-side --force-conflicts \ -f gen0sec-infra/charts/strimzi-kafka-operator/crds/ ``` The output is similar to this: ``` customresourcedefinition.apiextensions.k8s.io/postgresqls.acid.zalan.do serverside-applied ``` :::note[This is not hypothetical] The Postgres operator 1.15 to 2.0 change adds PostgreSQL 18 to the `postgresqls` schema. Without the CRD apply, the `postgresql/core` resource is rejected outright and the database never starts. ::: ## 2. Get the new artifacts Follow [Prepare the artifacts](./../04-prepare-artifacts.mdx) for the new release. Both methods work the same as on a first install, with one addition for the offline bundle. **Offline bundle only.** Mirroring is additive: the new tags land beside the old ones. Your running install keeps working until you upgrade, and a rollback still has its images. Do not reuse the previous bundle's directory. The bundler refuses to write into an existing one, and each bundle's values files are targeted at the version they shipped with. ## 3. Upgrade infra Run every command from the **new** bundle directory, so `charts/` and `values/` are the new release's. ```bash helm upgrade g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml ``` If you installed the platform under a release name or namespace other than `g0s` in `gen0sec`, add `--set components.ui.serviceAccountName=-ui,components.ui.namespace=` here. The infra chart binds the dashboard's service account by name, and a wrong name fails silently: the Components tab then cannot show infrastructure versions. Wait for the operators to finish rolling before continuing. ```bash kubectl -n gen0sec-system rollout status deploy/g0s-infra-postgres-operator kubectl -n gen0sec-system rollout status deploy/strimzi-cluster-operator kubectl -n gen0sec-system rollout status deploy/g0s-infra-synapse-operator kubectl -n gen0sec-system rollout status deploy/dragonfly kubectl -n gen0sec-system rollout status \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" ``` ## 4. Upgrade data ```bash helm upgrade g0s-data $B/charts/gen0sec-data-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml ``` ```bash kubectl -n gen0sec-system wait --for=condition=Ready kafka/core --timeout=15m kubectl -n gen0sec-system wait \ --for=jsonpath='{.status.PostgresClusterStatus}'=Running postgresql/core --timeout=15m ``` ## 5. Upgrade the platform The schema migration runs first, as a **pre-upgrade** hook, before any new pod rolls. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` Upgrading from v0.1.x, `entrypointHost` is required even if you never published the endpoints: the dashboard now builds its sign-in URL from it, and the render fails without it. `$KIT` is the **new** release's kit. Pass every file you layered at install time, in the same order: `helm upgrade` applies only what you give it. Leaving out the ingress overlay removes every Ingress, and leaving out `single-node/platform-values.yaml` on a quickstart install scales every service back to its production replica count. See [Publish the endpoints](./../06-publish.mdx). **If the migration fails, Helm aborts and the old application keeps running.** That is the designed behavior and it is why the hook runs before the rollout. ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` Fix the cause, then re-run the same command. See [Migration failures](./../troubleshoot/symptoms.mdx#migration-failures). :::note[v0.2.0: one migration needs a quiet moment] `20261004150000` changes a constraint on the block events table and waits at most 10 seconds for its lock. On a busy install the job fails with `canceling statement due to lock timeout` and leaves the schema dirty. Nothing was changed. Pause the block events pipeline, mark the previous version and re-run this step: ```bash kubectl -n gen0sec-system scale deploy/g0s-data-kafka2pg-block-events --replicas=0 kubectl -n gen0sec run db-migrate-force --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/db-migrate:$VERSION \ --env DATABASE_URL="$OWNER_DSN" -- force 20261004120000 ``` Scale the pipeline back to 1 once the upgrade completes. Events wait in Kafka meanwhile. ::: ## 6. Verify ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` Run the smoke test after every upgrade, not just after installs. It is the only check that proves the platform still serves an agent rather than merely starting. Full checklist: [Verify the install](./../08-verify.mdx). ## If you changed values along the way `helm upgrade` uses the values files you pass, not the ones from last time. Anything you set with `--set` on the previous install must be set again, or it reverts to the chart default. Check what the running release actually has: ```bash helm get values g0s -n gen0sec ``` Keep your overrides in a file rather than in shell history. See [Reference configurations](./../configure/reference-configs.mdx). ## If it fails | Symptom | Start here | |---|---| | CRD schema error on `helm upgrade` | You skipped step 1 | | `Unsupported value` from the Postgres operator | You skipped step 1 | | Migration job fails | [Migration failures](./../troubleshoot/symptoms.mdx#migration-failures) | | Migration job fails with `lock timeout` | A busy block events table. See the note in step 5 | | Pods roll but the smoke test fails | [Roll back](./rollback.mdx) | | `ImagePullBackOff` on new tags | The new images were not mirrored. Offline bundle method, step 2 | ## Next [Roll back](./rollback.mdx). --- # Prepare the artifacts Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/prepare-artifacts import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; # Prepare the artifacts ## Before you start - [ ] [Requirements](./02-requirements.mdx) met, tools installed, `yq` version checked - [ ] [Network and connectivity](./03-network.mdx) configured by your network team - [ ] Your [registry token](./02-requirements.mdx#the-registry-token), created on the dashboard under **Repository Access** - [ ] Your [Gen0Sec API key with the required scopes](./02-requirements.mdx#your-gen0sec-api-key) ## 0. Set your keys Create your token files. ```bash mkdir -p ~/.gen0sec cd ~/.gen0sec/ && touch .registry-token .download-api-key ``` Then open up an editor of your choice to fill each file with their respective content. Then run ```bash export REGISTRY_TOKEN=$(cat ~/.gen0sec/.registry-token) export G0S_DOWNLOAD_API_KEY=$(cat ~/.gen0sec/.download-api-key) ``` :::tip[Manage your keys securely] By storing keys in files, you make sure they don't get etched into your shell history. If you have history turned off or you know what you're doing, you can also go with ```bash export REGISTRY_TOKEN="your-registry-token" export G0S_DOWNLOAD_API_KEY="your-gen0sec-api-key" ``` ::: ## 1. Set the release Pin the release explicitly. A pinned tag installs the same bytes in six months; a resolved one does not. You can find available release versions on the [Releases page](https://releases.gen0sec.com/cerebellum/). ```bash export TAG=v0.2.0 export VERSION=${TAG#v} ``` To always use the latest version instead: ```bash export TAG=$(curl -fsSL https://releases.gen0sec.com/api/repos/cerebellum/latest | jq -r .tag_name) export VERSION=${TAG#v} ``` :::note[Release candidates are not published here] Only stable releases are mirrored, so an `-rc.N` tag returns 404 no matter how it is spelled. The [release index](https://releases.gen0sec.com/cerebellum/) is the list of what exists. ::: ## 2. Download the install kit ```bash curl -fsSLO https://releases.gen0sec.com/cerebellum/$TAG/cerebellum-install-kit.tar.gz curl -fsSLO https://releases.gen0sec.com/cerebellum/$TAG/cerebellum-install-kit.tar.gz.cosign.bundle ``` ## 3. Verify the signature Run this before you unpack anything. ```bash cosign verify-blob \ --bundle cerebellum-install-kit.tar.gz.cosign.bundle \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/release\.yaml@refs/tags/' \ cerebellum-install-kit.tar.gz ``` The output is similar to this: ``` Verified OK ``` Anything else means stop. Do not unpack the archive. :::note[Why both certificate flags are required] `cosign` refuses to verify without an expected issuer and an expected identity, so a shortened command fails rather than passing weakly. The identity is pinned to one workflow on a tag reference. A broader pattern would accept a signature produced by any workflow in our organization. The signature covers the file's contents, so it proves the bytes are ours regardless of how they reached you. ::: ## 4. Unpack it ```bash tar -xzf cerebellum-install-kit.tar.gz export KIT=$PWD/cerebellum-install-kit-$VERSION cd $KIT ``` The kit contains the bundler, the release manifest, the helper scripts used later in this install, and the sizing overlays. It does not contain images or charts. ## 5. Log in to our registry ```bash export REGISTRY=registry.gen0sec.com export REGISTRY_USER=any printf '%s' "$REGISTRY_TOKEN" | helm registry login $REGISTRY -u "$REGISTRY_USER" --password-stdin printf '%s' "$REGISTRY_TOKEN" | skopeo login $REGISTRY -u "$REGISTRY_USER" --password-stdin ``` Both are needed. `helm` and `skopeo` keep separate credential stores, and the next step uses each of them. `cosign` is a third store again, and it needs no login of its own. The bundler lends it the credential `skopeo` holds. ## 6. Get the charts Pull the charts and their values files. Images are not pulled: your cluster pulls each one from `registry.gen0sec.com` when it needs it. :::note[Confirm your nodes can pull from us before you go further] Our registry challenges pulls that send no `User-Agent`, which on some distributions is what the kubelet's pull path does. This step succeeds either way, because it runs from your workstation. The failure appears at step 3 of [Install](./05-install.mdx), as a 403 on every image. Check it now: [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: ```bash ./bundler.sh assemble-charts release-manifest.yaml export B=$PWD/gen0sec-cerebellum-multiarch-$VERSION ``` The output ends with a per-chart digest confirmation. :::note[What "digest-verified" means here] `assemble-charts` checks two digests per chart: the registry's manifest against the digest this release pins, and the chart archive against the layer named in that manifest. What you install provably came out of the chart this release published. ::: Each chart carries its own `values-onprem.yaml`, extracted into `$B/values/`. That file is what points the install at a registry. The default `values.yaml` leaves the registry blank on purpose, so installing without `-f` will not resolve images. ### 6a. Build the bundle on your workstation ```bash ./bundler.sh create-package release-manifest.yaml ``` This pulls every image and chart the release pins by digest, verifies every signature, then packages the result as `gen0sec-cerebellum-multiarch-$VERSION.tar.gz`. It carries both `linux/amd64` and `linux/arm64`, so it installs on any node in your cluster. Signature verification happens here, on the connected side, because it needs the public transparency log. Everything downstream is protected by checksums instead. :::note[Before you use --arch to halve the transfer] `--arch amd64` or `--arch arm64` extracts one manifest out of each multi-architecture index. That changes every image digest by definition. Per-architecture signatures still verify, because each one is signed individually. **SBOM attestations do not**, because they are attached to the index, and the index is what a single-architecture bundle leaves behind. The bundler warns at the time and annotates the bundle. If your acceptance process checks attestations, stay on the default. If your nodes are mixed, or you are unsure what they run, stay on the default: a missing architecture fails when the kubelet tries to pull, on an isolated cluster, after the transfer. A single-architecture bundle also renames the tarball and the directory inside it, so substitute the name in every command that follows. ::: ### 6b. Move it across In this step you take your bundle onto an intermediary machine inside the network where your internal registry and the target cluster is located you'll install the bundle onto. After transferring the bundle, you can clean up all artifacts it has created. eg. ```bash scp gen0sec-cerebellum-multiarch-$VERSION.tar.gz :~/ # `create-package` ran inside the kit directory, so the bundle is sitting in it. Copying the kit first # would transfer tens of gigabytes twice. `cleanup` removes only what the bundler created. ./bundler.sh cleanup cd .. && scp -r cerebellum-install-kit-$VERSION :~/ # You can even remove the installation kit if you don't intend to keep it. # cd $KIT/.. && rm -rf cerebellum-install-kit.tar.gz $KIT ``` ### 6c. Load it into your registry Log in to your intermediary machine and use the bundler to load the images into your internal registry. ```bash export REGISTRY=registry.internal.example export REGISTRY_USER=your-registry-user export REGISTRY_TOKEN=$(cat path/to/your-registry-token) # Carried across because this is a different machine and Install step 2 needs # it. Unset, the secret it creates is empty and every relay fetch 401s. export G0S_DOWNLOAD_API_KEY=$(cat path/to/your-gen0sec-api-key) export KIT=~/cerebellum-install-kit-$VERSION export VERSION=$(cat $KIT/release-manifest.yaml | yq ".spec.version") tar -xzf gen0sec-cerebellum-multiarch-$VERSION.tar.gz export B=~/gen0sec-cerebellum-multiarch-$VERSION cd $B printf '%s' "$REGISTRY_TOKEN" | skopeo login $REGISTRY -u "$REGISTRY_USER" --password-stdin ./bundler.sh apply release-manifest.yaml -r $REGISTRY ``` `apply` pushes every bundled image into your registry, then rewrites the bundled values files to point at it. The charts are not pushed: you install them from `$B/charts/*.tgz`. The bundle is checked against its own `SHA256SUMS` before anything is pushed, so a corrupted transfer fails here rather than at install time. :::note[For a plain-HTTP registry] Set `MIRROR_TLS_VERIFY=false` before `apply`. This affects the push only, and it is the one place in this install where TLS verification can be disabled. Prefer a certificate from your internal CA. ::: ## Checkpoint Whichever method you used, you should now have the following: | Variable | Points at | |---|---| | `$KIT` | The unpacked install kit | | `$B` | A directory containing `charts/` and `values/` | | `$REGISTRY` | The registry your cluster will pull from | | `$REGISTRY_USER` | The username for that registry (`any` for ours) | | `$REGISTRY_TOKEN` | The password or token for that registry | | `$G0S_DOWNLOAD_API_KEY` | Your Gen0Sec API key, set in step 0. [Install](./05-install.mdx#your-gen0sec-api-key) step 2 puts it in a secret | You are logged in to `$REGISTRY`. Every page after this one assumes exactly that and nothing else, which is why nothing after this point branches. --- # Publish the endpoints Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/publish # Publish the endpoints The ingress controller is installed and running. However it's not configured to serve anything by default. That is deliberate, because creating those routes makes the APIs reachable from wherever the controller is reachable, so it is your decision, not ours. The data plane's LoadBalancer Service already has an external address at this point. Until you complete this page it serves nothing, because no rule names a host. :::note[Ingress and TLS are yours to configure] The platform ships the routing rules. Terminating TLS, issuing the certificate and exposing the LoadBalancer are your cluster's responsibility. For the configuration options, see the [Kubernetes ingress reference](https://docs.gen0sec.com/docs/client/kubernetes-ingress). ::: ## Before you start - [ ] [Install](./05-install.mdx) complete, all pods running - [ ] A DNS name you control, resolving to your ingress controller's address - [ ] A LoadBalancer address for the ingress data plane, or a Service type your cluster can expose - [ ] A TLS certificate for that name, or cert-manager configured ## 1. Choose the address One address serves the dashboard and every agent-facing API. Requests are routed by path prefix. Use the address you set as `entrypointHost` in [Install step 7](./05-install.mdx#7-install-the-platform). It is already in `$ENTRYPOINT_HOST` if you followed the install in this shell: ```bash export ENTRYPOINT_HOST=cerebellum.example.internal ``` **The host is required, and the chart enforces it.** A rule with no host is silently ignored by the ingress controller: the Ingress object is created, no event is logged, and nothing serves traffic. Rather than publish one, the chart refuses to render and names the value to set, so this fails at `helm upgrade` instead of in production. **Use the address people will actually type.** The dashboard derives its sign-in URL from this host, and sign-in fails if the two do not match. ## 2. Publish The kit ships the overlay. Layer it on top of the values file you have been using. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` :::warning[Re-pass every values file you installed with] `helm upgrade` applies only the files you pass it. Anything layered at install time and omitted here reverts to the chart default. On the [Quickstart](./quickstart.mdx) you also passed `-f $KIT/examples/single-node/platform-values.yaml`, so include it in this upgrade too (before the ingress overlay), or every service scales back to its production replica count: ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-node/platform-values.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` ::: :::note[Quote any --set argument containing brackets] `zsh` treats `[0]` as a glob and fails before `helm` runs. Single-quote the whole argument: `--set 'services.ui.ingress.hosts[0].host=...'`. ::: ## 3. Verify the routes exist ```bash kubectl -n gen0sec get ingress ``` The output is similar to this: ``` NAME CLASS HOSTS ADDRESS PORTS AGE g0s-platform synapse cerebellum.example.internal 10.0.0.42 80 30s ``` An empty `HOSTS` column means the host was not applied, and nothing will serve traffic. The upgrade fails before it can leave an Ingress in that state, so if you see one it was created by an earlier install. Then check from outside the cluster. The dashboard is published at `/` and returns a redirect to its sign-in page, so follow it with `-L`: ```bash curl -fsSL -o /dev/null -w '%{http_code}\n' https://$ENTRYPOINT_HOST/ ``` The output is similar to this: ``` 200 ``` :::note[There is no health endpoint on the ingress] `/status/healthz` is served per pod inside the cluster and is deliberately **not** published through the ingress. A request to it from outside falls through to the dashboard catch-all and returns `404`, so do not use it as an external check. The dashboard root above confirms reachability; the [smoke test](./08-verify.mdx#4-the-platform-serves-an-agent) confirms the platform serves agents. ::: ## Find the address to point DNS at The ingress data plane is a `LoadBalancer` Service. ```bash kubectl -n gen0sec-system get svc g0s-infra-synapse -o wide ``` Point your DNS name at the external address. If `EXTERNAL-IP` stays ``, your cluster has no LoadBalancer implementation: change the Service type to something it can expose. It cannot stay cluster-internal. ## The route table You do not configure these. The overlay sets them, read off each service's own router. They are here because they are the answer to "a request landed on the wrong backend", and because a missing route fails quietly rather than loudly. | Path prefix | Served by | |---|---| | `/` | Dashboard (catch-all, shortest prefix, so it is last resort) | | `/v1/indicators`, `/v1/models`, `/v1/geoip`, `/v1/ids-rules` | Relay | | `/v1/penalty-model`, `/v1/traffic-model`, `/v1/ja4-classifier`, `/v1/flow-classifier`, `/v1/jepa-embedder` | Relay, per-model aliases | | `/v1/identity`, `/v1/policy-edges`, `/v1/fleet-bans` | Relay, served locally | | `/v1/authcheck`, `/v1/auth` | Authentication | | `/v1/events`, `/v1/agents`, `/v1/log` | Agent ingest | | `/v1/agents/events` | Configuration push (longer prefix than `/v1/agents`, so it wins) | | `/v1/config` | Configuration | | `/v1/telemetry`, `/v1/logs` | Telemetry | | `/v1/signal`, `/v1/rules` | Signal | | `/v1/ban-ledger` | Ban ledger | | `/v1/threat` | Threat feeds | | `/v1/service-graph` | Service graph | Routing is by **longest matching prefix**, because one address serves every backend. The configuration push stream on `/v1/config` is long-lived, so the chart sets read, write and idle timeouts of 3600 seconds on that route. Do not lower them: agents fall back to polling with no push, which looks like a slow platform rather than a timeout. :::note[Why a missing route is worse than a broken one] The dashboard owns `/`, so any path absent from this table falls through to it and gets a redirect or HTML. HTTP has no way for the caller to tell that apart from a real answer, so the failures are silent: | Missing route | What the agent actually does | |---|---| | `/v1/authcheck` | Receives HTML, cannot distinguish a bad key from a routing mistake, and continues unverified | | `/v1/agents/events` | Logs `SSE connect returned HTTP 404`, then falls back to polling. Looks like a slow platform, not a routing bug | | `/v1/ban-ledger` | Reads the redirect as success. The contribution is lost with no error anywhere | This is why the overlay exists rather than a list of prefixes for you to type. ::: :::note[/v1/ids-rules goes to the relay, not to the in-cluster rules service] IDS rules are relayed from gen0sec. The in-cluster rules service builds rules from a local ingest pipeline that this edition does not run. Route this prefix there and you reach a service with no rules, and the relay is never used. ::: ### If TLS terminates upstream If a load balancer, CDN or tunnel terminates TLS in front of the ingress, the chart cannot infer the external scheme and guesses `http` while the browser is on `https`. The dashboard then builds its sign-in URL with the wrong scheme and rejects every sign-in with `Invalid origin`. Set it explicitly: ```bash --set externalScheme=https ``` Leave it unset when TLS terminates at the ingress, because then the chart reads it from that service's ingress TLS block. One service can differ from the rest with `services..ingress.externalScheme`, which wins over the install-wide value above. Only a service that derives its own external URL reads it, and the dashboard is the only one that does. ## The dashboard The dashboard is enabled by default and is published at the root of your address. Sign-in is email and password, held in your own Postgres, so nothing outside the cluster is involved. Its secret, `g0s-db-ui`, was created in [Install](./05-install.mdx) step 6 with all the others. Re-running that script keeps the existing signing secret, so sessions and invite links stay valid. To leave the dashboard out: ```bash --set services.ui.enabled=false ``` ## If it fails | Symptom | Cause | |---|---| | `helm upgrade` fails with `entrypointHost is required` | Expected, and the point: `entrypointHost` was not passed. Set it and re-run | | `kubectl get ingress` shows an empty `HOSTS` column | An Ingress left by an earlier install, created when a host-less rule could still render. Re-run the upgrade with `entrypointHost` set | | Sign-in is rejected with `Invalid origin` | The dashboard's URL does not match the address people type. Check `entrypointHost`, and `externalScheme` if TLS terminates upstream | | `curl` times out | DNS does not resolve to your ingress controller, or a firewall blocks the port | | The dashboard loads but sign-in redirects to the wrong address | The sign-in URL is derived from `entrypointHost`. It must be the address people type | | A request reaches the wrong service | Longest-prefix routing. Check the `/v1/agents` overlap above | --- # Quickstart on one node Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/quickstart # Quickstart on one node :::danger[Not for production] This runs every component at a single replica on a single node. There is no redundancy anywhere: **losing the node loses the data.** Postgres has no standby, Kafka has one broker at replication factor 1, and the object store has no erasure coding. Use it for evaluation, training and upgrade rehearsals. For anything you depend on, follow [Sizing a production install](./02-requirements.mdx#sizing-a-production-install), which runs on three nodes. ::: ## What you get A working instance you can sign in to, on one machine, without planning a cluster. This is not a separate procedure. It is the same install as the rest of this documentation, with one extra values file per chart. ## What you need | | | |---|---| | One node | 12 vCPU, 32 GB RAM, 200 GB disk | | Kubernetes | k3s | | StorageClass | Any default block StorageClass. k3s installs `local-path` as the default, which is enough | | Tools | The same list as [Requirements](./02-requirements.mdx#tools-on-your-workstation) | | Credentials | The same two as [Requirements](./02-requirements.mdx#credentials-from-gen0sec). The relay is required here too | ## How it differs from production The kit ships an overlay per chart that reduces everything to one replica. | | Production | This quickstart | |---|---|---| | Object store | 3 replicas, erasure coded, distributed | 1 replica, **standalone mode**, and a Deployment rather than a StatefulSet | | Postgres | 3 instances | 1 instance, no replica pooler | | Kafka | 3 brokers, replication factor 3 | 1 broker, every factor 1 | | Application services | 2 or 3 replicas each | 1 each | | Cache | Chart default | 256 MB, one thread | | Storage | Your block StorageClass, 500 GiB per node minimum | The cluster's default StorageClass, about 65 GiB of volumes | | Survives losing a node | Yes | No | :::note[The connection pooler stays on] Only the **replica** pooler is disabled, because there is no replica to pool. Every service still connects through `core-pooler`, exactly as in production, so the secrets that `make-db-secrets.sh` composes are the same shape here. ::: :::note[Two things that surprise people] The object store stops being a StatefulSet and becomes a Deployment of the same name. That is why the readiness checks in [Install](./05-install.mdx) resolve the kind instead of naming it. Volume claims from a previous multi-replica install are not reused. If you are converting an existing install, delete them first. ::: ## Run it Follow the spine exactly as written, starting at [Prepare the artifacts](./04-prepare-artifacts.mdx). Three commands change. Nothing else does. | Spine step | On this install | |---|---| | [Prepare the artifacts](./04-prepare-artifacts.mdx) | Unchanged. Either method works, and the registry method is fewer steps | | [Install](./05-install.mdx) steps 1 and 2 | Unchanged. Same namespaces, same three secrets | | [Install](./05-install.mdx) steps 3, 4 and 7 | Add the overlay. See below | | [Install](./05-install.mdx) steps 5 and 6 | Unchanged. You still create the bucket, and you still run the secret script | | [Publish the endpoints](./06-publish.mdx) | Unchanged. See the note on the address below | | [First sign-in](./07-first-signin.mdx) and [Verify the install](./08-verify.mdx) | Unchanged | Pass the overlay **after** the production values file, so it wins. ```bash helm install g0s-infra $B/charts/gen0sec-infra-*.tgz -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml \ -f $KIT/examples/single-node/infra-values.yaml ``` ```bash helm install g0s-data $B/charts/gen0sec-data-*.tgz -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml \ -f $KIT/examples/single-node/data-values.yaml ``` ```bash helm install g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-node/platform-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` The wait gates after each one are the same as on [Install](./05-install.mdx). Read the files before you use them: each overlay carries comments explaining what it trades away. :::caution[Your cluster needs a default StorageClass] The overlays leave the storage class empty in all three places — `rustfs.storageclass.name` in `infra-values.yaml`, `postgres.volume.storageClass` and `kafka.storage.class` in `data-values.yaml` — which means "use the cluster's default", the same as the production values file. On k3s that resolves to `local-path` and there is nothing to do. If your cluster has **no** default, every PVC stays `Pending`. `helm install` still reports success, and the only symptom is the wait gate above timing out without naming a cause. Check first: ```bash kubectl get storageclass # exactly one row must be marked (default) ``` To pin a class explicitly instead of relying on the default, set those three keys, or layer a fourth `-f` file last. ::: ## The address [Publish the endpoints](./06-publish.mdx) is unchanged, including the requirement for a host, which the platform install above already needs in `$ENTRYPOINT_HOST`. On k3s the bundled LoadBalancer implementation gives the ingress data plane the node's own address, so there is nothing further to provision. For an evaluation, a name in `/etc/hosts` on the machine you browse from is enough. It has to be the name you actually type in the browser, because the dashboard builds its sign-in URL from it. :::note[About TLS terminatotion] In case your TLS termination does not happen on a higher level, configure Synapse Ingress to work with cert-manager (see the [Kubernetes ingress reference](https://docs.gen0sec.com/docs/client/kubernetes-ingress)). The shipped ingress overlay sets `externalScheme: https` and configures no certificate at the ingress. A single-node evaluation usually has none. Either provision a certificate, or serve plain HTTP: set `externalScheme` to `http` in the ingress overlay and use an `http://` base URL in the sign-in and verify steps below. ::: ## First sign-in Nothing is different here, and it is worth knowing before you get there: the dashboard makes you enroll multi-factor authentication and replace the seeded password before anything opens. See [First sign-in](./07-first-signin.mdx). ## Verify Run the same checks as a production install, from [Verify the install](./08-verify.mdx). The smoke test is the one that matters, because it proves the platform serves an agent rather than merely starting: ```bash # Save the agent key you minted at first sign-in, then read it back below. mkdir -p ~/.gen0sec printf '%s' '' > ~/.gen0sec/agent-key export GEN0SEC_BASE_URL=https://$ENTRYPOINT_HOST export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) $KIT/scripts/verify-deployment.sh ``` A single-node install that passes it is a good rehearsal for a real one. That is most of the value of building this. ## Tearing it down ```bash $KIT/scripts/uninstall.sh --yes ``` It asks you to type the cluster context name back before deleting anything, so it will not run unattended. See [Uninstall](./operate/uninstall.mdx). ## When you are ready for production [Sizing a production install](./02-requirements.mdx#sizing-a-production-install) has the node counts, per-node capacity and storage figures. There is no in-place conversion from this install to a production one: build the production cluster separately. --- # Bundler CLI Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/reference/bundler # Bundler CLI `bundler.sh` ships in the install kit. It builds an offline bundle on a connected host and loads it into your registry on a host that has no internet access. Its only input is the `release-manifest.yaml` inside the kit. No access to our source is needed, so you can build your own bundle from any published release. For the procedure rather than the reference, see [Prepare the artifacts](./../04-prepare-artifacts.mdx). ## Which commands need internet access This is the table an isolated site actually needs. The pipeline is split on internet access, not on environment. | Command | Internet | Talks to | |---|---|---| | `validate` | **Required** | Parses the manifest, resolves tooling, then fetches one image manifest to confirm registry access | | `assemble` | **Required** | Source registry: images and charts | | `assemble-charts` | **Required** | Source registry: charts only | | `assemble-images` | **Required** | Source registry: images only | | `verify-assembly` | **Required** | Source registry for signature artifacts, plus Sigstore trust material | | `create-package` | **Required** | Inherits `assemble` and `verify-assembly` | | `package` | None | Local `tar` | | `verify-package` | None | Local extract and checksum comparison | | `mirror` | None | Destination registry, over your local network | | `retarget` | None | Local files | | `apply` | None | Destination registry. This is `mirror` plus `retarget` | | `cleanup`, `version`, `help` | None | Local | **Split the pipeline at the tarball, not earlier.** `create-package` cannot run on an isolated host: it fails in preflight with no route to the source registry. ## Commands | Command | What it does | |---|---| | `validate` | Preflight. The manifest exists and its schema is sane, every image and chart is digest-pinned, the tooling resolves, and the source registry is readable. Fetches one image manifest, so a bad host, missing login, TLS mismatch or unpublished digest fails here rather than partway through a multi-gigabyte download | | `assemble` | Creates the bundle directory, pulls every image and chart, extracts each chart's production values file, copies in the manifest and itself. Fails if the directory exists | | `assemble-charts` | The charts half. **What the registry install method uses**, since the cluster pulls its own images | | `assemble-images` | The images half. Separable so a failed multi-hour pull can be retried without re-fetching charts | | `verify-assembly` | Keyless signature verification of every pinned image and chart, then writes `SHA256SUMS` and prints the bundle **fingerprint**. Reports every failure and refuses to continue on any | | `package` | Tars the bundle directory into a tarball **beside** it, never inside | | `verify-package` | Unpacks to a temporary directory and re-checks `SHA256SUMS` | | `create-package` | `assemble`, `verify-assembly`, `package`, `verify-package`, in order | | `mirror` | Verifies the bundle against `SHA256SUMS`, then pushes every image to your registry | | `retarget` | Rewrites the registry host in the bundle's own values files, in place, and updates the affected `SHA256SUMS` entries so the bundle stays self-consistent | | `apply` | `mirror` then `retarget`. The usual customer step | | `cleanup` | Removes every directory and archive the bundler created | | `version` | The bundler's version, stamped at release | | `help` | Usage | Every command runs `validate` first, so any of them is safe to invoke on its own. ## Options | Option | Effect | |---|---| | `--arch amd64` or `--arch arm64` | Single-architecture bundle, roughly half the size. **Loses SBOM attestations.** See below | | `-o ` | A different tarball name | | `-r ` | Destination registry for `mirror`, `retarget` and `apply` | | `--dangerously-skip-cosign-verify` | Drops signature verification and its tooling requirement. The bundle is annotated as unverified | ### The --arch trade-off A single-architecture bundle extracts one manifest out of each multi-architecture index, which changes every image digest by definition. | | Default | With `--arch` | |---|---|---| | Image signatures | Verify | **Verify.** Each per-architecture manifest is signed individually | | SBOM attestations | Verify | **Lost.** They hang off the index, which is what is left behind | | Digests match the manifest | Yes | No | | Tarball and directory name | `multiarch` | Named after the architecture, so substitute it everywhere | If your acceptance process checks attestations, or your nodes are mixed, or you are not certain what they run: take the default. A missing architecture fails when the kubelet tries to pull, on an isolated cluster, after the transfer. ## Environment variables | Variable | Purpose | |---|---| | `MIKEFARAH_YQ_PATH` | Path to the Go `yq`, when the one on `PATH` is the Python one | | `BUNDLE_TLS_VERIFY=false` | Disables TLS verification on the **pull** side | | `MIRROR_TLS_VERIFY=false` | Disables TLS verification on the **push** side, for a plain-HTTP registry | The two TLS variables are the only places in the whole install where verification can be disabled. Prefer a certificate from your internal CA. See [Hardening](./../security/hardening.mdx). ## Credentials, and the three stores Three tools, three credential stores. You log in to two of them. | Tool | Store | Login needed | |---|---|---| | `skopeo` | The containers store | Yes, both source and destination | | `helm` | `HELM_REGISTRY_CONFIG` | Yes, source only | | `cosign` | The Docker config | **No.** The bundler lends it the one `skopeo` holds | If you see the bundler say it is reusing skopeo's credential for cosign, that is the expected path and not a warning. ## Behavior worth knowing | | | |---|---| | `assemble` refuses to write into an existing directory | So a stale bundle cannot be silently mixed with a new one | | `retarget` takes no file arguments | Rewriting arbitrary paths handed in on a command line is not something an isolated install should be able to ask for | | `retarget` verifies integrity **before** editing, never after | That order is what stops a checksum update from re-blessing a bundle that arrived altered | | `retarget` is idempotent | Running `apply` twice is safe | | `mirror` runs before `retarget` inside `apply` | So a registry failure leaves your values files untouched | | Mirroring is additive | New tags land beside old ones, so a running install keeps working and a rollback still has its images | | An egress proxy is fine for the bundler | Provided the host can reach the registry and Sigstore. Unlike the relay, these are ordinary CLI tools that honour proxy variables | ## Bundle contents ``` gen0sec-cerebellum--/ ├── oci/ every image, as an OCI layout ├── charts/ installable chart archives ├── values/ each chart's production values, retargeted by `apply` ├── release-manifest.yaml the digests this bundle was built from ├── bundler.sh a copy └── SHA256SUMS over every file above ``` `assemble` extracts each chart's values file as `values/-values-onprem.yaml`. There is no bare `values-onprem.yaml` in the bundle root, which is why the install commands name the file in full. The bundle does **not** carry the install kit's `scripts/` or `examples/`. Those cross the air gap inside the kit directory. See [Install kit contents](./kit.mdx). --- # Release notes Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/reference/changelog # Release notes ## v0.2.0 A minor release. Upgrade from v0.1.0 or v0.1.1 following [Upgrade](./../operate/upgrade.mdx). It carries 18 schema migrations, which run as a pre-upgrade hook, so take a backup first. Every one only adds, so rolling the platform back to v0.1.x afterwards is safe; see [Roll back](./../operate/rollback.mdx). :::warning[Fleet bans start being enforced] On v0.1.x the fleet-ban artifact was written in a format agents could not read, so every agent loaded zero fleet bans while the publishing job reported success. From v0.2.0 they load. Agents with `platform.fleet_bans.enforce` on will start refusing traffic they did not refuse before. Review your fleet bans, or turn enforcement off, before you upgrade. ::: :::warning[Invitation links issued on v0.1.x stop working] Invitations are now tracked in the database, and links issued before the upgrade are not in it. Re-invite anyone who has not accepted yet. ::: :::warning[Check `externalScheme` in your values before upgrading] `externalScheme` was ignored on v0.1.x and is read from v0.2.0. If your values carry a top-level `externalScheme` that has never had any effect, **it takes effect on upgrade** and changes the scheme in the dashboard's sign-in URL. A scheme that does not match the address people actually type is rejected as `Invalid origin`. ```bash helm get values g0s -n gen0sec | grep externalScheme ``` Remove it or correct it before upgrading. Nothing to do if you pass `examples/single-entrypoint/ingress-values.yaml` unchanged: it ships `externalScheme: https`, which is the scheme v0.1.x effectively used anyway. ::: ### Added - **Passkeys.** Users can register and sign in with a passkey, and manage them under **Profile**. Passkeys are bound to the published host name and need HTTPS. See [Sign-in and identity](./../configure/identity.mdx#passkeys). - **Invitation and password-reset email**, through your SMTP server. Without SMTP, invitations fall back to a link you copy. See [Invitations and email](./../configure/identity.mdx#invitations-and-email). - **Role-scoped invitations.** An invitation grants specific workspaces, each with a role no higher than the inviter's. Pending invitations are listed and can be revoked, and access to one workspace can be revoked without removing the person. - **Audit log**, under **Organization settings** for the Owner and Admin roles. It records changes made in the dashboard and rule changes made through the signal API, with source IP and user agent. It is kept indefinitely; see [Known limitations](./../troubleshoot/known-limitations.mdx). - **Components**, under **Organization settings**: what each component runs, against the latest version known to the release. An online check for newer versions is off by default. See [Component versions](./../configure/components.mdx#verifying-what-you-are-actually-running). - **WAF rules accept `regex_replace()` and the `http.request.raw_path` field.** See [Known limitations](./../troubleshoot/known-limitations.mdx#new-waf-rule-functions-need-current-agents) for agent versions. - `CACHE_L1_TTL` sets the default lifetime of the in-process cache, as a duration. Default `30s`, up from a fixed `15s`. Set it through `global.commonEnv`. ### Fixed **Fleet bans never reached agents.** See the warning above. **Block events were lost in several ways.** Each of these applied to v0.1.x: - A batch of more than 20,000 records was refused whole, and OTLP senders do not retry that. The first 20,000 are now kept. - One row the database refused sent its whole insert batch, other organizations' rows included, to the dead-letter topic. Only the refused row is parked now, with the reason in a `dlq_reason` header. - A NUL byte in any field, a `source_ip` that was not an IP address, or a non-ASCII country code made the database refuse the row. These are now cleaned or dropped, and the event is kept. - A non-integer hit count stalled the dashboards' aggregates for every organization. It is now dropped, and the event kept. - An agent newer than the platform could report a source the platform did not know, and the event was refused. It is now stored without its source. `fleet_ban` is also accepted as a source. - IDS alerts were rejected as malformed block events and warned about once a minute. Records that are not block events are now skipped quietly. **Block events lacked country and ASN** whenever the GeoIP lookup missed, although the agent had sent them. The country is also stored in upper case, so `us` and `US` no longer count twice. **Drops made by the agent's kernel IP filter were not stored.** The agent reports them as `dropped_ip` records rather than block events, and those were skipped, so for an agent enforcing in the IP filter the access-rules views stayed empty. A drop the access rules caused is now stored as an access-rules block event, with its count as the hits and `bpf:ip_filter` as the rule. **Writing a managed WAF rule failed** with `column s.rule_managed_id does not exist`. **`--set entrypointHost` had no effect.** Following the [Publish the endpoints](./../06-publish.mdx) procedure verbatim published every route on the placeholder `gen0sec.internal` whatever address you passed. The overlay carried the host as a YAML anchor, and anchors are expanded when the file is parsed, before Helm merges `--set`, so the flag rewrote the top-level key while all ten rules kept the value the file shipped with. No template read the value either, so removing the anchor alone would not have helped. The failure was silent. The Ingress objects were created and looked correct, but Synapse keys its upstreams by host and serves nothing for an address no rule names. The chart now resolves the entrypoint itself and publishes every rule under it. **The dashboard's sign-in URL could be absent.** `BETTER_AUTH_URL` was derived from the first entry of a service's `ingress.hosts`. With the address named once in `entrypointHost` that list is empty, so the variable was not wrong but missing, and Better Auth rejected every sign-in with `Invalid origin`. It now follows the same entrypoint the rules do. ### Changed **An empty `entrypointHost` now fails the render** instead of publishing host-less rules. Such a rule matches every `Host` header and Synapse's operator skips it, so it could only ever be a misconfiguration that reported nothing. `helm upgrade` now stops and names the value to set. Rules that carry their own `ingress.hosts[].host` are unaffected and still need no `entrypointHost`. **`entrypointHost` is required from the first install**, before anything is published. The dashboard's sign-in URL now always comes from, in order: `services.ui.env.BETTER_AUTH_URL`, the dashboard's own `ingress.hosts[0].host`, then `entrypointHost`, whether or not its Ingress is enabled. With none of them the render fails and names `entrypointHost`. It was previously derived only once the dashboard's Ingress was enabled. [Install](./../05-install.mdx#7-install-the-platform) and [Upgrade](./../operate/upgrade.mdx#5-upgrade-the-platform) pass it. **`--set externalScheme` had no effect either.** Same defect, same shape: nothing read the top-level value, and the single-entrypoint overlay pinned the dashboard's own scheme on top of it, so the flag was inert twice over. It only ever looked correct because what it pinned was `https`, which is what a right answer produces anyway. The wrong scheme in the dashboard's base URL is rejected as `Invalid origin`, so this mattered to anyone terminating TLS upstream who needed anything else. The chart reads the top-level value now, and the overlay stops pinning the dashboard. `services..ingress.externalScheme` still overrides it for one service. A value that was inert becomes live. See the warning at the top before you upgrade. **The dashboard holds one namespaced permission.** The infra chart adds the Role and RoleBinding `g0s-ui-read-installed-versions` in `gen0sec-system`, allowing `get` on two named ConfigMaps, and the dashboard's service account token is now mounted. Nothing to do on the documented release names. Otherwise set `components.ui.*`; see [Upgrade](./../operate/upgrade.mdx#3-upgrade-infra) and [Privileges and RBAC](./../security/privileges.mdx). **The dashboard makes optional outbound calls**, none carrying agent data. See [Ports and egress](./../security/egress.mdx). **telemetry-api logs what it drops**, once a minute per organization, and its existing decode warnings are rate-limited the same way. See [Monitor](./../operate/monitor.mdx). **Rule changes made through the signal API are audited** in the same transaction. If the audit record cannot be written, the rule change is not applied. See [Symptoms](./../troubleshoot/symptoms.mdx#rule-changes-made-through-the-signal-api-never-appear). **The Agents item is hidden** from the dashboard navigation on self-hosted installs. **One migration waits at most 10 seconds for a lock** on the block events table. On a busy install it can fail and is safe to retry. See the note in [Upgrade](./../operate/upgrade.mdx#5-upgrade-the-platform). ### Security - gRPC 1.83.2, fixing an HTTP/2 memory exhaustion reachable in the APIs (GO-2026-6348), and OpenTelemetry 1.47.0. Other Go modules updated for advisories that were not reachable. - Next.js 16.3.7 and React 19.3.0, fixing CVE-2026-94545 as first shipped in v0.1.1. - better-auth 1.7.7, and the dashboard's dependency overrides enforced so `pnpm audit` is clean. ### Documentation - These docs are now versioned, with a version selector. v0.1.0 and v0.1.1 each keep their own version, with both defects above listed in their known limitations. - [Publish the endpoints](./../06-publish.mdx#if-tls-terminates-upstream) documents `--set externalScheme=https` again, now that it works. - [Upgrade](./../operate/upgrade.mdx) now passes the ingress overlay to the platform upgrade. Without it, the upgrade removes every Ingress. - [Roll back](./../operate/rollback.mdx) uses the `db-migrate` image of the release you are leaving. The previous image cannot run the down migrations. ## v0.1.1 A security patch for the dashboard. It fixes [CVE-2026-94545](https://nvd.nist.gov/vuln/detail/CVE-2026-94545) (medium, CVSS 4.0 score 5.3), in which Satori, the HTML-to-SVG library Next.js bundles for `next/og`, did not escape certain values in the SVG it generates. The dashboard does not render images through `next/og`, but the vulnerable copy shipped in its image and scanners flag it. The fix updates Next.js from 15.5.24 to 16.3.7, and React from 19.2.3 to 19.3.0. It upgrades in place from v0.1.0, with no migration and no values changes. Nothing else changed: the procedure is v0.1.0's, and its known limitations still apply. ## v0.1.0 First on-premises release. ### Installing Two methods, both documented on one procedure that branches once: [install from our registry](./../04-prepare-artifacts.mdx), or [create an offline bundle](./../04-prepare-artifacts.mdx) and mirror it into your own. ### Requires | | | |---|---| | Kubernetes | 1.23 or later | | Nodes | 3 for production, 16 vCPU and 32 GiB each recommended | | Storage | [sized to your retention](./../02-requirements.mdx#sizing-storage-for-your-retention) | | Cluster egress | `api.gen0sec.com:443`, required | ### Credentials you need **Two**, and the second is easy to miss. Both are self-service from the dashboard: 1. A registry token, for pulling charts and images. Created under **Repository Access**. The install kit itself is public and needs no credentials. 2. **Your Gen0Sec API key**, which the relay uses to fetch data from us. Without it the whole fleet receives empty artifacts with no error. See [Credentials from gen0sec](./../02-requirements.mdx#credentials-from-gen0sec). ### Not in this release | | | |---|---| | Zero-egress deployment | [Known limitations](./../troubleshoot/known-limitations.mdx#no-zero-egress-deployment) | | Explicit egress proxy for the relay | [Known limitations](./../troubleshoot/known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy) | | Single sign-on of any kind | Email and password only. Microsoft Entra ID is in development | | Shipped dashboards and alert rules | [Monitor](./../operate/monitor.mdx) lists what to watch meanwhile | | The threat service | Not shipped at this release | --- ## How we version | | | |---|---| | Scheme | Semantic versioning. Each minor is a release train | | Supported | The latest two minor trains | | Upgrades | Sequential minors, patches in place. **Skipping a minor is not supported** | | Release candidates | Not published to `releases.gen0sec.com`. Only stable releases are mirrored | Full policy: [Component versions](./../configure/components.mdx#support-policy). ## Finding a release ```bash curl -fsSL https://releases.gen0sec.com/api/repos/cerebellum/latest | jq -r .tag_name ``` Returns the latest full release. **Pin the tag explicitly for an install you want to reproduce**, so a rebuild in six months fetches the same bytes. ## What every release publishes | Artifact | Where | |---|---| | Install kit and its signature | `https://releases.gen0sec.com/cerebellum//` | | Images and charts | The registry, each pinned to a digest in the kit's release manifest | | Signatures and SBOM attestations | Alongside each image and chart, as OCI referrers | See [Supply chain](./../security/supply-chain.mdx). --- # Install kit contents Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/reference/kit # Install kit contents ## What you download Two files, published per release at `https://releases.gen0sec.com/cerebellum//`. | File | What | |---|---| | `cerebellum-install-kit.tar.gz` | The kit | | `cerebellum-install-kit.tar.gz.cosign.bundle` | Its detached signature | The release also publishes `latest-known-versions.json` there. You do not download it: the dashboard reads it only when its online update check is on. It is display data, not signed and not part of the kit. Verify before unpacking. See [Prepare the artifacts](./../04-prepare-artifacts.mdx#3-verify-the-signature). ## Layout ``` cerebellum-install-kit-/ ├── bundler.sh builds and loads the offline bundle ├── release-manifest.yaml every image and chart, pinned to a digest ├── scripts/ │ ├── preflight.sh cluster checks the charts assume but do not create │ ├── make-db-secrets.sh composes the ~20 application secrets │ ├── uninstall.sh the three releases, finalizers, namespaces │ └── verify-deployment.sh external smoke test, read-only ├── examples/ │ ├── single-node/ three overlays, one replica each │ └── single-entrypoint/ one address for the dashboard and every API ├── offline-data/ │ └── README.txt └── SHA256SUMS over every file above ``` **No images and no charts.** Those come from the registry, or from the offline bundle. ## The scripts | Script | Run it | Documented in | |---|---|---| | `preflight.sh` | Before installing | [Install step 1](./../05-install.mdx#1-check-the-cluster) | | `make-db-secrets.sh` | After the databases are ready, before the platform | [Install step 6](./../05-install.mdx#6-create-the-application-secrets) | | `verify-deployment.sh` | After every install and every upgrade | [Verify the install](./../08-verify.mdx#4-the-platform-serves-an-agent) | | `uninstall.sh` | When removing the installation | [Uninstall](./../operate/uninstall.mdx) | `verify-deployment.sh` reads two environment variables: ```bash export GEN0SEC_BASE_URL=https://cerebellum.example.internal export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) ``` ## The examples Everything under `examples/` is a starting point you are meant to read and edit. Each file carries comments marking which values are yours. | Directory | Purpose | |---|---| | `single-node/` | Three overlays that reduce every component to one node. [Quickstart](./../quickstart.mdx) | | `single-entrypoint/` | One address for the dashboard and every agent-facing API. [Publish the endpoints](./../06-publish.mdx) | ## Verifying the kit's own integrity ```bash cd $KIT && sha256sum -c SHA256SUMS ``` `SHA256SUMS` covers the kit **as delivered**. Adding files makes it no longer match, which is expected once you start using it. ## What the offline bundle method has to transfer Two things, and the order matters. | Transfer | Contains | |---|---| | The bundle tarball | Images, charts, values, the manifest, a copy of the bundler | | The kit directory | `scripts/`, `examples/`, and the manifest | The bundle carries its own copy of `bundler.sh`, so loading it needs no tooling from the kit. **The kit still has to cross**, because the install uses its scripts and its examples. Copy the bundle first, then clean up, then the kit. `create-package` runs inside the kit directory, so the bundle is sitting in it, and copying the kit first transfers tens of gigabytes twice. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). Both archives unpack side by side: ``` ~/gen0sec-cerebellum-multiarch-0.2.0/ the bundle ~/cerebellum-install-kit-0.2.0/ the kit ``` ## The offline-data directory The kit creates `offline-data/` containing a single `README.txt`, and nothing else. It is a placeholder for a data-delivery path that is not part of this release: the four datasets the platform needs are fetched at runtime through the relay, not delivered as files. See [Data relay](./../configure/relay.mdx). If you were told to populate it, check with us first. At this release you should not need to. --- # Requirements Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/requirements # Requirements ## Cluster | | Requirement | |---|---| | Kubernetes | 1.23 or later | | Access | `cluster-admin`, for the install only. See [Privileges and RBAC](./security/privileges.mdx) for what it is used for and what it leaves behind | | StorageClass | A default StorageClass on block storage. **Volume expansion recommended** | | Supported architecture | `linux/amd64` or `linux/arm64`. Mixed-architecture clusters are supported | | LoadBalancer | The ingress data plane is exposed as a `LoadBalancer` type Service by default. See the note below | :::note[The ingress needs to be reachable from outside the cluster] The ingress data plane is a `LoadBalancer` type Service, because nothing it programs can serve traffic if it stays cluster-internal. Your cluster needs a LoadBalancer implementation. If your cluster fronts ingress another way, a NodePort, a hardware balancer or an external proxy, change the Service type to match. ::: Cerebellum brings its own data plane - which contains Postgres, Kafka, object store and cache. You do not connect it to existing instances of any of them. :::info[Maintenance] We've deliberately chosen to use well-known solutions from the software industry in our data layer to ease the burden of maintenance. While Cerebellum ships this data layer by default, to make the setup easier, the data in it is the highest importance to you. So make sure you monitor the dataplane and do backups and maintenances regularly. ::: :::note[Container runtime, for the registry install method only] Our registry challenges pulls that carry no `User-Agent`, and on some distributions the kubelet's pull path sends none. containerd's own user agent passes; the problem is its absence. Either configure one on every node, or use the offline bundle method, which pulls into your own registry instead. Settle this before you choose a method. See [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: ### Kubernetes versions 1.23 is the floor because `autoscaling/v2` is the newest API the charts use, and it reached GA in that release. While anything from 1.23 and up should work, we recommend you use the latest stable version. ## Sizing a production install Production runs on **3 nodes**. Several data plane members each form a three-member quorum, so three is the floor for redundancy, not a suggestion. ### CPU & memory per node | | Resources | |---|---| | **Minimum** | 8 vCPU, 16 GiB | | **Recommended** | 16 vCPU, 32 GiB | ### Storage per node We recommend to start your nodes with `500 GiB` **minimum**. Several of the data plane members use PVCs. While the defaults are kept low - as your platform matures the data will grow and you'll need to expand the storage volumes. ### Sizing storage for your retention The shipped sizes are a starting point, not a capacity plan. They are deliberately modest, so for most production workloads you will be growing them rather than shrinking them. Four keys control everything: | Key | Chart | Default | |---|---|---| | `postgres.volume.size` | `gen0sec-data` | 50 GiB | | `kafka.storage.size` | `gen0sec-data` | 50 GiB | | `rustfs.storageclass.dataStorageSize` | `gen0sec-infra` | 50 GiB | | `rustfs.storageclass.logStorageSize` | `gen0sec-infra` | 10 GiB | Which one to grow depends on what you are retaining: | If you need | Grow | |---|---| | Longer event and telemetry history | `postgres.volume.size`. This is the system of record | | A longer Kafka retention window, or a bigger ingest buffer | `kafka.storage.size` | | More room for datasets and locally-produced artifacts | `rustfs.storageclass.dataStorageSize` | Example for longer retention: ```yaml # gen0sec-data postgres: volume: size: 200Gi kafka: storage: size: 200Gi ``` ```yaml # gen0sec-infra rustfs: storageclass: dataStorageSize: 100Gi ``` :::note[Plan for growth] Volume expansion grows a volume if your StorageClass supports it, and it is straightforward. Nothing shrinks one: reducing a size means recreating the volumes and losing what is on them. Starting at the shipped defaults and growing is therefore the recommended direction. It is the reason for low initial limits. Check that your StorageClass has `allowVolumeExpansion: true` before you install. ::: ### What is not on this page **Throughput.** How many agents, events per second or lookups a given topology sustains is not directly derivable from the charts' parameters. We suggest you scale the platform components and storage sizes according to your topology's specific needs. {/* Four workloads declare no CPU or memory requests: the object store, the Kafka brokers, valkey, and the Postgres operator. They are therefore BestEffort or Burstable and are evicted first under node pressure, which is why the per-node recommendation is driven by limits rather than by the requests total. Raised separately as a chart issue. */} ### Evaluating on one node Use the [Quickstart](./quickstart.mdx). It runs everything at a single replica on one node, for evaluation, training and upgrade rehearsals. ## Tools on your workstation Install these on the machine you run the commands from, not on the cluster. | Tool | Version | Used for | Install | |---|---|---|---| | `helm` | 3.16 or later | Installing the charts, and pulling them from the registry | [helm.sh](https://helm.sh/docs/intro/install/) | | `kubectl` | matching your cluster | Everything cluster-side | [kubernetes.io](https://kubernetes.io/docs/tasks/tools/#kubectl) | | `cosign` | 2 or later | Verifying signatures on the install kit, images and charts | [docs.sigstore.dev](https://docs.sigstore.dev/cosign/system_config/installation/) | | `skopeo` | any | Copying images between registries | [github.com/containers/skopeo](https://github.com/containers/skopeo/blob/main/install.md) | | `yq` | 4 or later, **mikefarah/yq only** | Reading the release manifest | [github.com/mikefarah/yq](https://github.com/mikefarah/yq#install) | | `jq` | any | Reading the release API | [jqlang.github.io](https://jqlang.github.io/jq/download/) | | `aws` or `mc` | any | Creating the object store bucket, one step during install | [aws-cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) or [MinIO client](https://min.io/docs/minio/linux/reference/minio-mc.html) | | `curl`, `tar`, `openssl`, `sha256sum` or `shasum` | any | Downloading, unpacking, generating secrets, checking checksums | Your OS packages | ### Check your yq There are two separate tools that are called `yq`. The Python one (`kislyuk/yq`) cannot parse the expressions the bundler uses, and it fails with a syntax error partway through. ```bash yq --version ``` The output must contain `mikefarah`. If it does not, install the Go version and point the bundler at it: ```bash export MIKEFARAH_YQ_PATH=/path/to/mikefarah-yq ``` ## Credentials from Gen0Sec You need **two**, and they are different things. Missing the second one produces a working-looking install whose entire fleet silently receives nothing. | Credential | Used by | Used for | |---|---|---| | **Registry token** | The bundler, on your workstation | Pulling charts and images from our registry | | **Your Gen0Sec API key** | The relay pod, inside your cluster | Fetching threat intelligence, GeoIP, models and IDS rules from Gen0Sec | Both are self-service from the dashboard. Neither requires contacting us. :::note[The install kit itself needs no credentials] The kit is published openly. Downloading and verifying it, in [Prepare the artifacts](./04-prepare-artifacts.mdx), happens before you use either credential — so if that download fails, the token is not what to check. The usual cause is a tag that is not published; the [release index](https://releases.gen0sec.com/cerebellum/) lists the ones that are. ::: ### The registry token Create it on the dashboard, under **Repository Access**: ``` https://app.gen0sec.com//repositoryAccess ``` Your workspace id is in the address bar once you are signed in, and the same page is linked from **Clients**. The token is shown **once**, at creation — store it before leaving the page, because it cannot be retrieved afterwards and a lost one has to be replaced. **The username is literally `any`.** Wherever a command asks for one, type `any`. Only the token is checked, so the username can be anything at all — `any` is simply what these docs use so the commands are copy-pasteable. ### Your Gen0Sec API key You can create your API key on the [SaaS Dashboard](https://app.gen0sec.com) Scopes required: - `ids_rules:read`, - `indicators:read`, - `ml_models:read` :::note[Why this one is separate from the registry token] Your agents get their API keys from your own dashboard, and those keys exist only in your cluster's database, so they cannot authenticate to Gen0Sec. The API key created in this step is used by your in-cluster relay to authenticate against the Gen0Sec SaaS, so it can fetch data related to threat intelligence, GeoIP, ML models and IDS rules. The chart expects it as a secret named `gen0sec-download-proxy`, and the relay pod will not start without it, so a missing key is a failed rollout rather than a silently degraded fleet. You will create this secret in the step 2 of [Install](./05-install.mdx). ::: --- # Architecture and data flows Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/security/architecture # Architecture and data flows Written for a security reviewer. Everything here is observable in a running install, and every claim has a page you can check it against. ## What runs where Two namespaces, three charts, one ingress point. ``` internet │ │ api.gen0sec.com:443 (the only required outbound path) │ ┌───────────────────────────┼──────────────────────────────────────────────┐ │ your cluster │ │ │ ▲ │ │ ┌─────────────────────── │ ─────────────────────────────────────────┐ │ │ │ namespace: gen0sec │ │ │ │ │ │ │ │ │ │ dashboard relay(download-api) 13 application services │ │ │ │ │ │ │ └───────────────────────────────┬───────────────────────────────────┘ │ │ │ in-cluster only │ │ ┌───────────────────────────────┼───────────────────────────────────┐ │ │ │ namespace: gen0sec-system │ │ │ │ │ ▼ │ │ │ │ Postgres (3) Kafka (3) object store (4) caches (2) │ │ │ │ Postgres operator Kafka operator ingress operator │ │ │ └───────────────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────┘ ▲ │ one address, TLS, path-prefix routed │ your agents and your users ``` | Namespace | Contains | Reachable from outside | |---|---|---| | `gen0sec` | The application services and the dashboard | Yes, through the ingress you publish | | `gen0sec-system` | Postgres, Kafka, object store, two caches, and the three operators | No | Nothing in `gen0sec-system` is published. The application reaches it by cluster DNS. ## Trust boundaries Four, and they are worth naming separately because they fail differently. | Boundary | Crossed by | Controlled by | |---|---|---| | **Internet to cluster** | The relay, outbound only, one destination | Your firewall. See [Ports and egress](./egress.mdx) | | **Your network to cluster** | Agents and users, inbound, one address | Your ingress and its TLS | | **Namespace to namespace** | The application reading its databases | Cluster DNS. No NetworkPolicy ships. See [Hardening](./hardening.mdx) | | **Cluster to registry** | Image pulls | The image pull secret, one per namespace | ## Data flows ### Inbound: agents and users | Flow | Path | Authentication | |---|---|---| | Agent telemetry and events | Your address, then the ingest services, then Kafka, then Postgres | Agent API key | | Agent configuration pull and push | Your address, then the configuration services | Agent API key | | Agent artifact download | Your address, then the relay, then Gen0Sec | Agent API key, validated locally. The relay uses your installation's own key upstream, so agent keys never leave. See [Data handling](./data-handling.mdx) | | Dashboard sign-in | Your address, then the dashboard, then Postgres | Email and password, or a passkey, held in your Postgres. No external identity provider at this release | ### Outbound: the relay, plus optional dashboard calls The dashboard can also check for new versions and send email. All of it is optional and carries no agent data. See [Data handling](./data-handling.mdx#the-dashboards-optional-connections). The relay has one destination and one purpose: fetching four licensed datasets that are in no artifact we ship. | Dataset | Why it cannot be shipped | |---|---| | CTI indicators | Licensed by upstream providers | | GeoIP databases | Licensed by MaxMind and IPInfo | | ML models | Published by gen0sec, refreshed independently of releases | | IDS rules | Licensed as part of the Emerging Threats ruleset | The relay is a **relay, not a cache**. Nothing is stored on the way through, so there is no local artifact store to size and no staleness window. ### Never outbound | Dataset | What it is | Where it stays | |---|---|---| | `identity` | Workload identity produced in your cluster | Your object store | | `policy-edges` | Pod IPs, namespaces, NetworkPolicy edges | Your object store | Relaying either would push a map of your workload topology to a service you do not run. It is not configurable and there is no flag that enables it. Uploads are not relayed either: the relay is read-only. ## Where data is stored | Store | Contains | Backed up by you | |---|---|---| | Postgres | The system of record: configuration, agents, policy, history | Yes, always | | Object store | Datasets, and the two locally-produced sets above | Yes | | Kafka | In-flight events between ingest and persistence | No, it is transport | | Cache | Nothing durable | No | See [Back up and restore](./../operate/backup.mdx). ## What the platform requires of your cluster | Requirement | Detail | |---|---| | `cluster-admin` during install | For CRDs and cluster-scoped RBAC. See [Privileges and RBAC](./privileges.mdx) | | Custom resource definitions | Postgres and Kafka operator CRDs, cluster-scoped by nature | | Its own ingress controller | Installed by the infra chart, `ingressClassName: synapse` | | One required outbound destination | `api.gen0sec.com:443` | | No node access, no host mounts, no privileged containers | See [Hardening](./hardening.mdx) | ## Questions a reviewer usually asks next | Question | Page | |---|---| | What exactly can leave the cluster? | [Data handling](./data-handling.mdx) | | What ports and destinations do I open? | [Ports and egress](./egress.mdx) | | Why does it need `cluster-admin`? | [Privileges and RBAC](./privileges.mdx) | | How do I verify what you shipped me? | [Supply chain](./supply-chain.mdx) | | What is the container security posture? | [Hardening](./hardening.mdx) | | What does not work yet? | [Known limitations](./../troubleshoot/known-limitations.mdx) | --- # Data handling Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/security/data-handling # Data handling ## What leaves your cluster One required connection, outbound, to one destination. The dashboard can make three optional ones, listed after it. | | | |---|---| | Destination | `api.gen0sec.com:443` | | Made by | The relay pod only | | Direction | Outbound, request and response | | Purpose | Fetching four licensed datasets | **What is in the request.** A path naming the artifact, and your own Gen0Sec API key. Nothing else. The relay does not add a customer identifier, a cluster identifier, an agent identifier, telemetry or usage data. **What is in the response.** The artifact: a CTI indicator database, a GeoIP database, an ML model, or an IDS rule bundle. ### The dashboard's optional connections | Destination | When | What is sent | |---|---|---| | `helm.gen0sec.com:443` | Opening the Clients or Components page | A request for the public chart index. Nothing about your install | | `releases.gen0sec.com:443` | Only if **Check for updates online** is on | A request for the latest release's public version list. Nothing about your install | | Your SMTP server | Only if you configure SMTP | The recipient's address, your organization's name, and an invitation or password-reset link | All three fail without affecting anything else. See [Ports and egress](./egress.mdx). ## What never leaves your cluster | Data | What it is | |---|---| | `identity` | Workload identity produced inside your cluster | | `policy-edges` | Pod IP addresses, namespaces, NetworkPolicy edges | | Agent telemetry and events | Everything your agents report | | Configuration and policy | Everything you configure | | Logs | All of it, to container stdout in your cluster | | Database contents | Postgres never leaves | | Dashboard user accounts, passwords and passkeys | Held in your Postgres | | Audit log | Who changed what, in the dashboard or through the API: the user or API key, the action, the changed fields with secret-looking fields redacted, source IP and user agent | `identity` and `policy-edges` deserve the explicit statement, because they are the two that are technically served by the same service that does the relaying. **They are never relayed.** Doing so would push a map of your workload topology to a service you do not run. It is not configurable, and no flag enables it. The upload surface is not relayed either, because the relay is read-only. ## Who validates agent credentials **Your agents' keys are validated in your cluster, and they never leave it.** Every relayed route sits behind the local authentication middleware, and each one carries the same permission it carries in direct mode. So a key that would be refused when served locally is refused it when relayed. ``` agent --[agent key, authorized locally, permission checked per route]--> relay --[your Gen0Sec API key]--> api.gen0sec.com agent key stops here ``` Two separate concerns, deliberately separated: | Hop | Credential | Checked by | |---|---|---| | Agent to relay | The agent's own key | Your cluster, against your installation, with the artifact's permission | | Relay to Gen0Sec | Your Gen0Sec API key | Gen0Sec | | | | |---|---| | Credentials Gen0Sec receives | One, belonging to your installation | | Credentials Gen0Sec never receives | Your individual agent key | | What Gen0Sec can infer about your fleet | Nothing from the credential. Requests carry no agent or cluster identifier | | Revoking one agent | Local, in your dashboard, immediate | | Revoking your installation's upstream access | Revoke or rotate your Gen0Sec API key | This is the answer to "does our agent inventory leave our perimeter". It does not. :::note[Agent keys are never forwarded upstream, even by accident] On a self-hosted deployment the relay never puts a caller's key in an outbound request, in any configuration. A locally-issued key could not authenticate to Gen0Sec anyway, so forwarding it could only leak a local credential to a third party without making the request succeed. With no Gen0Sec API key configured the relay sends no credential at all rather than sending yours. ::: ## Credentials at rest | Credential | Where it lives | Generated by | |---|---|---| | Database role passwords | Kubernetes secrets | The Postgres operator | | Object store keys | A Kubernetes secret | You, during install | | Registry pull credentials | A Kubernetes secret per namespace | You, during install | | Dashboard session signing key | A Kubernetes secret | Generated once, preserved across regeneration | | SMTP credentials | A Kubernetes secret | You, optional | **No credential appears in a chart, a values file or a rendered manifest.** Every one is referenced by name. See [Secrets](./../configure/secrets.mdx). The install procedure reads every credential from a file or from standard input rather than passing it as a command-line argument, so nothing lands in shell history or in the process list. ## Credentials in transit | Hop | Encrypted | |---|---| | Agents and users to your ingress | Yes, TLS you terminate | | Relay to `api.gen0sec.com` | Yes, TLS | | Services to Postgres | Yes, TLS to the pooler | | Services to the object store | Plain HTTP, in-cluster only | | Services to the cache | Plain, in-cluster only, no credential in the URL | The last two are in-cluster traffic on the pod network. If your threat model includes an attacker on the pod network, add NetworkPolicies and a service mesh. See [Hardening](./hardening.mdx). :::note[One deliberate exception, and why] The dashboard's database connection verifies encryption without verifying the certificate chain, because its client library rejects the pooler's self-signed certificate outright while every other service's library accepts encrypt-without-verify. The traffic is encrypted and in-cluster. Do not copy that setting onto the other services: their driver rejects it. ::: ## Data retention Retention is yours. Nothing expires data on your behalf, and nothing is sent anywhere before it is deleted. | Store | Grows with | Sized by | |---|---|---| | Postgres | Agents, events, history | `postgres.volume.size` | | Postgres audit log | Administrative changes, kept indefinitely | Small. See [Known limitations](./../troubleshoot/known-limitations.mdx#the-audit-log-is-never-pruned) | | Kafka | In-flight events, and its retention window | `kafka.storage.size` | | Object store | Datasets and locally-produced artifacts | `rustfs.storageclass.dataStorageSize` | See [Sizing](./../02-requirements.mdx#sizing-a-production-install). ## Deleting everything [Uninstall](./../operate/uninstall.mdx) without `--keep-data` deletes the namespaces and every persistent volume in them: the database, the message log and the object store. There is no copy anywhere else. --- # Ports and egress Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/security/egress # Ports and egress Same as [Network and connectivity](./../03-network.mdx), reproduced here for better visibility. **One canonical source.** If this table and the install page ever disagree, that is a bug in our publishing, not a judgement call for you. Report it. ## Outbound All TCP, all TLS, all outbound. Nothing needs an inbound rule from the internet. | Destination | Port | Needed by | Purpose | Required | |---|---|---|---|---| | `api.gen0sec.com` | 443 | Cluster pods, the relay only | Threat intelligence, GeoIP, ML models, IDS rules | **Always** | | `releases.gen0sec.com` | 443 | Workstation | Download the install kit and its signature | **Always** | | `registry.gen0sec.com` | 443 | Workstation | Pull charts and images | **Always** | | `registry.gen0sec.com` | 443 | Cluster nodes | Image pulls at install time | Registry install only | | `fulcio.sigstore.dev` | 443 | Workstation | Signature certificate verification | **Always** | | `rekor.sigstore.dev` | 443 | Workstation | Transparency log verification | **Always** | | `tuf-repo-cdn.sigstore.dev` | 443 | Workstation | Signature trust root refresh | **Always** | | Your own registry | 443 or 80 | Cluster nodes, and the host that loads the bundle | Image pulls at install time | Offline bundle only | | `helm.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest agent version, shown on the Clients and Components pages | No. Always attempted. Blocking it only hides the latest version | | `releases.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest component versions | No. Only if an administrator turns on **Check for updates online** | | Your SMTP server | 587 or 465 | Cluster pods, the dashboard only | Invitation and password-reset email | No. Only if you [configure SMTP](./../configure/identity.mdx#invitations-and-email) | **Only one row is required by the running cluster:** `api.gen0sec.com:443`, and only the relay pod uses it. The dashboard rows are optional: blocking them hides version information, or leaves invitations and password resets without email. Everything else is install-time, and most of it is your workstation rather than your cluster. ## Inbound | Source | Destination | Port | Purpose | |---|---|---|---| | Your agents | Your published address | 443 | Telemetry, events, configuration, artifact download | | Your users | Your published address | 443 | Dashboard | One address serves both, routed by path prefix. See [Publish the endpoints](./../06-publish.mdx#the-route-table). ## In-cluster No NetworkPolicy is required, and none ships. If you enforce default-deny, these are the flows to allow. | From | To | Port | |---|---|---| | `gen0sec` services | `core-pooler.gen0sec-system` (Postgres pooler) | 5432 | | Migration job | `core.gen0sec-system` (Postgres primary) | 5432 | | `gen0sec` services | `core-kafka-bootstrap.gen0sec-system` | 9092 | | `gen0sec` services | `rustfs-svc.gen0sec-system` (object store) | 9000 | | `gen0sec` services | `dragonfly.gen0sec-system` (cache) | 6379 | | Ingress controller | `gen0sec` services | Service ports | | `gen0sec` services | `gen0sec` services | Service ports | | Dashboard (`ui`) pods | The Kubernetes API server | 443 or 6443 | The dashboard reads two ConfigMaps from the API server for the Components tab. Find its address with `kubectl get endpoints kubernetes -n default`. Without that flow only the Components tab degrades. See [Hardening](./hardening.mdx) for what we recommend beyond that. ## Notes that save time **The Sigstore hosts are the ones people forget.** Signature verification is the first command of the install, and it fails without them. They are needed on the workstation, not on the cluster. **A proxy does not help for the cluster's one destination.** Explicit forward proxies are not honoured in-cluster. See [Known limitations](./../troubleshoot/known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy). **The offline bundle removes registry egress, not all egress.** The runtime connection stays. See [What this rules out](./../03-network.mdx#what-this-rules-out). --- # Hardening Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/security/hardening # Hardening Two lists: what the charts already do, and what is left to you. The second list is short but it is not empty, and we would rather name it than let you assume otherwise. ## Hardened by default Every application service runs with this security context, and it is not opt-in. ```yaml podSecurityContext: runAsNonRoot: true runAsUser: 65532 runAsGroup: 65532 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: [ALL] ``` | Control | Status | |---|---| | Runs as non-root | Yes, uid 65532 | | Read-only root filesystem | Yes | | Privilege escalation | Disabled | | Linux capabilities | All dropped | | Privileged containers | None | | Host network, PID or IPC | None | | Host path mounts | None | | Node access | None | :::note[Read-only root filesystems were not free] Several services defaulted to writing into a relative path under their working directory, which fails outright on a read-only root. Each one now gets an explicit writable volume mounted where it expects to write. That is why the values file carries per-service volume mounts: they are the cost of this control, not decoration. If you add a service-level override that removes one, that service will fail to start. ::: ## Pod Security Admission The application pods satisfy the **restricted** profile. Label the namespace to enforce it: ```bash kubectl label namespace gen0sec \ pod-security.kubernetes.io/enforce=restricted \ pod-security.kubernetes.io/enforce-version=latest ``` **Do not label `gen0sec-system` restricted yet.** The Postgres and Kafka pods there are created by upstream operators and have not been assessed against that profile. Use `baseline` if you need something, and tell us if you require restricted there. ## What you should add None of these ship, and all of them are worth doing. ### NetworkPolicies **No NetworkPolicy ships with the charts.** On a cluster without default-deny, that means the platform namespaces are as open as everything else. Add default-deny in both namespaces, then allow exactly these: | From | To | Port | |---|---|---| | `gen0sec` services | `core-pooler.gen0sec-system` | 5432 | | Migration job | `core.gen0sec-system` | 5432 | | `gen0sec` services | `core-kafka-bootstrap.gen0sec-system` | 9092 | | `gen0sec` services | `rustfs-svc.gen0sec-system` | 9000 | | `gen0sec` services | `dragonfly.gen0sec-system` | 6379 | | Ingress controller | `gen0sec` services | Service ports | | `gen0sec` services | `gen0sec` services | Service ports | | Dashboard (`ui`) pods | The Kubernetes API server | 443 or 6443 | | Relay pod | `api.gen0sec.com:443` | 443, egress | **Egress default-deny is the high-value one here.** The relay is meant to be the only pod that needs a route out. The dashboard's optional egress is listed on [Ports and egress](./egress.mdx). A NetworkPolicy is what turns that from a design intention into an enforced property, and it is the single strongest control you can add to this deployment. ### TLS everywhere it is not already | Hop | Today | Worth adding | |---|---|---| | Ingress to users and agents | TLS you terminate | Already yours | | Relay to Gen0Sec | TLS | Nothing to do | | Services to Postgres | TLS to the pooler | Nothing to do | | Services to the object store | Plain HTTP, in-cluster | A service mesh, if your threat model includes the pod network | | Services to the cache | Plain, in-cluster, no credential in the URL | Enable cache authentication, then move the connection string into each service's secret | ### Resource limits on the four workloads that have none The object store, the Kafka brokers, the ingress cache and the Postgres operator declare no CPU or memory requests. That makes them BestEffort or Burstable, so the kubelet evicts them first under node pressure. For the object store, holding your data, that is the wrong priority order. Set requests on them, or set a namespace `LimitRange` so nothing lands unbounded. See [Sizing](./../02-requirements.mdx#sizing-a-production-install). ### Secrets from an external manager `make-db-secrets.sh` is fine for evaluation. In production, have your secret manager produce secrets of the same shape so credentials never exist as plain Kubernetes secrets you created by hand. Every name and key is listed in [Secrets](./../configure/secrets.mdx). ## Settings you should never enable Two environment variables exist and both disable a security control. Neither is set by the charts, and neither appears anywhere in the install procedure. | Setting | What it does | When it is acceptable | |---|---|---| | `TLS_INSECURE_SKIP_VERIFY` | Stops the relay verifying the upstream certificate | Never in production. It defeats the protection on your only outbound connection | | `MIRROR_TLS_VERIFY=false` | Stops the bundler verifying your registry's certificate when pushing | Only for a plain-HTTP registry on a trusted network, and preferably not then | If you find either one set in a running deployment, treat it as a finding. ## Verifying the posture Confirm no pod runs as root or privileged: ```bash kubectl -n gen0sec get pods -o json | jq -r ' .items[] | .metadata.name + " runAsNonRoot=" + ((.spec.securityContext.runAsNonRoot // false) | tostring)' ``` Confirm nothing holds cluster-admin: ```bash kubectl get clusterrolebinding -o json \ | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | .metadata.name' ``` Confirm only the relay can reach the internet, once you have added egress policy: ```bash kubectl -n gen0sec run netcheck --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/busybox:1.36.1-glibc -- wget -T5 -qO- https://example.com ; echo "exit=$?" ``` A non-zero exit is the result you want. --- # Privileges and RBAC Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/security/privileges # Privileges and RBAC ## The short answer `cluster-admin` is required **to install**, not to run. It is needed for two things that are cluster-scoped by nature: 1. Installing custom resource definitions. 2. Creating the cluster roles the three operators need. Once installed, no component in the platform holds cluster-admin. ## What is cluster-scoped ### Custom resource definitions From the bundled operators, not written by us. | CRD group | Installed by | Kinds | |---|---|---| | `acid.zalan.do` | Postgres operator | `postgresqls`, `operatorconfigurations`, `postgresteams` | | `kafka.strimzi.io` | Kafka operator | `kafkas`, `kafkatopics`, `kafkanodepools`, `strimzipodsets`, and the Connect and Bridge kinds we do not use | CRDs are cluster-scoped in Kubernetes. There is no namespaced alternative. :::note[Deleting a CRD deletes every resource of that kind, cluster-wide] Including any outside these two namespaces. If anything else in your cluster runs a Postgres or Kafka operator, removing these CRDs would take its databases and clusters with them. [Uninstall](./../operate/uninstall.mdx) removes them by default, but checks first: if any of those kinds is in use outside the two namespaces it is removing, it skips the CRDs and says so. `--keep-crds` skips them unconditionally. ::: ### Cluster roles | Cluster role | Belongs to | What it is for | |---|---|---| | `g0s-postgres-operator` | Postgres operator | Managing Postgres clusters and their pods | | `g0s-postgres-operator-endpoints` | Postgres operator | The `endpoints` permission its own upstream role omits, while still creating a master Endpoints object per cluster | | `postgres-pod` | Postgres pods | What a database pod itself needs | | `strimzi-cluster-operator-global` | Kafka operator | Cluster-scoped reconciliation | | `strimzi-cluster-operator-namespaced` | Kafka operator | Namespaced resources, bound per namespace | | `strimzi-cluster-operator-leader-election` | Kafka operator | Leader election | | `strimzi-cluster-operator-watched` | Kafka operator | Watching the namespaces it manages | | `strimzi-kafka-broker` | Kafka brokers | What a broker pod needs | | `strimzi-entity-operator` | Topic and user operator | Managing topics and users | | `strimzi-kafka-client` | Kafka clients | Client-side permissions | | `g0s-synapse-operator` | Ingress operator | Watching Ingress objects and services | Eleven, and nine of them come straight from the two upstream operators. :::note[The one we added, and why] `g0s-postgres-operator-endpoints` exists because the Postgres operator ships a cluster role without `endpoints`, yet creates a master Endpoints object for every cluster it manages. Without that rule, `postgresql/core` fails with `endpoints is forbidden` and the database never starts. See [Postgres will not start](./../troubleshoot/symptoms.mdx#postgres-will-not-start). ::: ## What the application itself holds The application services in `gen0sec` are not operators and do not watch the cluster. | | | |---|---| | Cluster-scoped permissions | None | | Namespaced permissions | None, except the dashboard (see below) | | Service accounts | One per service. Tokens are not mounted, except the dashboard's | | Node access | None | | Host mounts | None | | Privileged containers | None | :::note[The one exception: the dashboard reads two ConfigMaps] The `ui` service account (`g0s-ui` in `gen0sec`) is bound to the Role `g0s-ui-read-installed-versions` in `gen0sec-system`, which the infra chart creates. It allows `get` on two ConfigMaps by name, `g0s-infra-installed-versions` and `g0s-data-installed-versions`, and nothing else. There is no `list` or `watch`, so it cannot enumerate other ConfigMaps. The dashboard uses it for the [Components](./../configure/components.mdx) tab, and is the only pod with its service account token mounted. ::: See [Hardening](./hardening.mdx) for the container security context. ## Can I install without cluster-admin? Partly, if you split the install between two roles. This is a supported pattern but it is not the documented golden path, so plan for one extra iteration. | Step | Needs | Who | |---|---|---| | Apply CRDs | Cluster-scoped write on `customresourcedefinitions` | Your platform team, once per release | | Create cluster roles and bindings | Cluster-scoped RBAC write | Your platform team, once | | Install the three charts | Namespace-admin on both namespaces | The installer | Render what a chart would create, and hand that to whoever holds the privilege: ```bash helm template g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system -f $B/values/gen0sec-infra-values-onprem.yaml \ | yq eval-all 'select(.kind == "ClusterRole" or .kind == "ClusterRoleBinding")' ``` ## Verifying after install Confirm nothing holds more than it should: ```bash kubectl get clusterrolebinding -o json \ | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | .metadata.name + " -> " + ([.subjects[]?.name] | join(","))' ``` No entry in that output should reference a `gen0sec` service account. If one does, tell us: that is a defect, not a configuration choice. ## What an uninstall leaves behind | | Removed | Reason | |---|---|---| | Namespaced resources | Yes | Helm owns them | | Objects the operators created outside the releases | Yes | The script removes them; `helm uninstall` alone does not | | CRDs | Yes, unless in use elsewhere or `--keep-crds` | Removal is cluster-wide destructive, so the script checks before it acts | | Cluster roles and bindings | Depends on the chart that created them | Check what remains and remove by hand if nothing else uses the operators | [Uninstall](./../operate/uninstall.mdx) reports what it removed and what it left. --- # Provenance, signatures and SBOM Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/security/supply-chain # Provenance, signatures and SBOM Everything we publish is signed, and every signature is verifiable by you without asking us for a key. This page also includes the one case where verification cannot follow the artifacts. ## What is signed, and how | Artifact | Signed | Attested | |---|---|---| | Install kit tarball | Detached cosign bundle, published beside it | | | Container images | `cosign sign --recursive`, so the index **and** each per-architecture manifest | SPDX JSON SBOM, one per architecture | | Helm charts | `cosign sign` on the OCI chart artifact | SPDX JSON SBOM | **Keyless, with GitHub OIDC.** There is no long-lived private key to steal, and no public key for you to fetch and trust. Each signature is bound to a workflow identity recorded in a public transparency log, so verification asserts *who built it*, not *who holds a key*. SBOMs are generated with syft and attached as SPDX JSON attestations. :::note[SLSA provenance is deliberately off] BuildKit's provenance attestations add manifests to the multi-architecture index carrying a placeholder platform. That breaks tools which walk an index by platform, including our own mirroring. We ship the SPDX SBOM attestation instead, which is the thing customers actually verify. If you require SLSA provenance, tell us: it is a build-flag change plus mirroring work, not a redesign. ::: ## Verify the install kit Do this before you unpack it. It is step 3 of [Prepare the artifacts](./../04-prepare-artifacts.mdx). ```bash cosign verify-blob \ --bundle cerebellum-install-kit.tar.gz.cosign.bundle \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/release\.yaml@refs/tags/' \ cerebellum-install-kit.tar.gz ``` **Both certificate flags are required.** `cosign` refuses to verify without an expected issuer and an expected identity, so a shortened command fails rather than passing weakly. The identity is pinned to **one workflow, on a tag reference**. A looser pattern would accept a signature produced by any workflow, on any branch, in our organization. That is the difference between "built by our release pipeline from a tag" and "built by somebody at Gen0Sec". ## Verify an image or a chart Against our registry, or against yours after mirroring: ```bash cosign verify \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/' \ registry.gen0sec.com/gen0sec/auth-api:0.2.0 ``` ## Read the SBOM ```bash cosign download attestation \ --predicate-type https://spdx.dev/Document \ registry.gen0sec.com/gen0sec/auth-api:0.2.0 \ | jq -r '.payload | @base64d | fromjson | .predicate' ``` The predicate is an SPDX JSON document. Feed it to whatever consumes SBOMs in your pipeline. ## Verification across the air gap This is the part nobody else documents, so it gets its own section. ### What crosses, and what does not Signature verification needs the public transparency log, and an isolated network does not have one. So the bundler verifies **once, on the connected staging host, before packaging**, and protects everything downstream with checksums. ``` connected staging host isolated network ────────────────────── ──────────────── pull every artifact by digest cosign verify each one (cannot re-verify: no transparency log) write SHA256SUMS -> verify-package re-checks SHA256SUMS print the bundle fingerprint -> mirror re-checks it before pushing package ``` | Stage | Guarantee | |---|---| | Before packaging | Full cryptographic signature verification, every image and every chart | | The tarball | `SHA256SUMS` over every file, plus a **fingerprint**, which is a sha256 of that list | | On the isolated side | `verify-package` and `mirror` both re-check `SHA256SUMS` and refuse to proceed on a mismatch | **Record the fingerprint** that verification prints on the connected side, out of band. Comparing it on the isolated side is what closes the loop between the two networks. :::note[Signature re-verification inside the isolated network is not supported] It would need the transparency log, or a pinned trust bundle carried across and kept current. We do not pretend otherwise. If your policy requires cryptographic verification at the destination rather than checksum verification, tell us. That is a different design, and we would rather discuss it now than have you find the gap during an audit. ::: ### The one case where attestations are lost `--arch amd64` or `--arch arm64` halves the transfer. It also extracts **one manifest out of each multi-architecture index**, which changes every image digest by definition. | | Multi-architecture bundle (default) | Single-architecture bundle | |---|---|---| | Image signatures | Verify | **Verify.** Each per-architecture manifest is signed individually | | SBOM attestations | Verify | **Lost.** They hang off the index, and the index is what is left behind | | Digests match the release manifest | Yes | No, by definition | The bundler warns at the time and annotates the bundle rather than letting this be a silent difference. **If your acceptance process checks attestations, stay on the default.** ## Digest pinning `release-manifest.yaml` in the install kit pins **every image and chart to a digest**, not to a tag. Tags move. Digests do not. That is what makes a release reproducible: the bundler resolves nothing at build time, it fetches exactly the digests the manifest names, and it fails if one is missing. ```bash yq '.spec.images[] | [.image, .imageTag, .digest] | @tsv' $KIT/release-manifest.yaml ``` ## If verification fails | Symptom | Cause | |---|---| | `no matching signatures` | Wrong identity pattern, or the artifact is not ours | | Certificate fetch errors | The Sigstore hosts are blocked. See [Network and connectivity](./../03-network.mdx#egress-list) | | The image verifies, the SBOM does not | Single-architecture bundle. Expected, see above | | `SHA256SUMS` mismatch on the isolated side | The transfer altered the bundle. Re-transfer, do not proceed | **A failed verification is not a step to skip.** The bundler has a flag that drops the check, and a bundle built with it is annotated as unverified. Use it only if cosign genuinely cannot be installed on your staging host, and record that you did. --- # Collect diagnostics Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/troubleshoot/diagnostics # Collect diagnostics Collect this before you contact us. Everything here is something we would otherwise ask for, so sending it means the first reply is an answer rather than a request for more output. There is no single collection command at this release. This page is the list. ## Set up Run the commands from a directory you can archive afterwards. ```bash NS_SYS=gen0sec-system NS_APP=gen0sec OUT=gen0sec-diagnostics mkdir -p $OUT && cd $OUT ``` Cluster and storage: ```bash kubectl version -o yaml > cluster-version.yaml kubectl get nodes -o wide > nodes.txt kubectl describe nodes > nodes-describe.txt kubectl get storageclass -o yaml > storageclasses.yaml kubectl get pvc -A -o wide > pvc.txt ``` Releases and workloads: ```bash helm list -A > helm-releases.txt for ns in $NS_SYS $NS_APP; do kubectl -n $ns get all -o wide > workloads-$ns.txt kubectl -n $ns get events --sort-by=.lastTimestamp > events-$ns.txt kubectl -n $ns get pods -o yaml > pods-$ns.yaml done ``` Logs from anything not ready: ```bash for ns in $NS_SYS $NS_APP; do for p in $(kubectl -n $ns get pods -o json \ | jq -r '.items[] | select(any(.status.containerStatuses[]?; .ready == false)) | .metadata.name'); do kubectl -n $ns logs $p --all-containers --tail=2000 > logs-$ns-$p.txt 2>&1 kubectl -n $ns logs $p --all-containers --previous --tail=2000 > logs-$ns-$p-previous.txt 2>&1 done done ``` Operators and custom resources: ```bash kubectl -n $NS_SYS logs deploy/g0s-infra-postgres-operator --tail=2000 > logs-postgres-operator.txt kubectl -n $NS_SYS logs deploy/strimzi-cluster-operator --tail=2000 > logs-kafka-operator.txt kubectl -n $NS_SYS logs deploy/g0s-infra-synapse-operator --tail=2000 > logs-ingress-operator.txt kubectl -n $NS_SYS get postgresql,kafka,kafkanodepool,kafkatopic -o yaml > custom-resources.yaml ``` The migration job, which is the most common single cause: ```bash kubectl -n $NS_APP logs job/g0s-db-migrate --tail=2000 > logs-db-migrate.txt 2>&1 ``` Images actually in use, including init containers: ```bash for ns in $NS_SYS $NS_APP; do kubectl -n $ns get pods -o yaml | grep -E '^\s+image:' | sort -u > images-$ns.txt done ``` Routing and the relay: ```bash kubectl -n $NS_APP get ingress -o yaml > ingress.yaml kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' > relay-healthz.json 2>&1 ``` Then pack it: ```bash cd .. && tar -czf gen0sec-diagnostics-$(date +%Y%m%d-%H%M%S).tar.gz gen0sec-diagnostics ``` :::danger[Read the archive before you send it] It is plain text and it is yours. Nothing here redacts anything, so check it, and remove any sensitive data it might have, particularly if you added `helm get values`, which prints whatever you set. None of the commands above read secret contents, database rows or object store objects. Logs can contain request paths, key prefixes, up to 400 bytes of a telemetry body that failed to decode, and the full content of any block event the pipeline parked. ::: :::note[There should be one command for this] A single `collect-diagnostics.sh` in the install kit would replace this page with two lines, and the list above is the specification for it. A single diagnostics tool can accidentally collect data that's sensitive and may fail to redact that information, so that's why we ask you to assemble your own report. Once we're 100% sure we can provide a diagnostics tool that doesn't leak any information of your sensitive data, we'll do so. ::: ## What to tell us alongside it Four lines, and they save a round trip: 1. Which install method you used. 2. The release version. 3. The last command you ran, verbatim. 4. What you expected to happen. --- # Known limitations in this release Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/troubleshoot/known-limitations # Known limitations If you hit something that is not on this page, we'd like to hear about it! ## No zero-egress deployment **What it means.** Your cluster must reach `api.gen0sec.com` on port 443. That connection carries threat intelligence, GeoIP, ML models and IDS rules, which are licensed upstream data that is in no artifact we ship. **What you can remove.** All dependency on Gen0Sec-hosted registries. The [offline bundle method](./../04-prepare-artifacts.mdx) mirrors every image and chart into your own registry, so your cluster pulls nothing from us. **What you cannot remove.** The runtime connection. There is no supported configuration with zero cluster egress. **If your policy forbids all egress**, please contact us. See [Network and connectivity](./../03-network.mdx). ## Our registry challenges pulls that send no user agent **By design on our side, and it can decide which install method you use.** **Symptom.** Every image pull from `registry.gen0sec.com` returns **403 with a CAPTCHA challenge**, and pods sit in `ImagePullBackOff`. A manual `docker pull` or `skopeo copy` from the same node **succeeds**. That discrepancy is the whole diagnosis. If the CLI works and the kubelet does not, this is it. **Cause.** On some Kubernetes distributions, the component that services the kubelet's pull sets no `User-Agent` header. Go's `net/http` then sends its own default, `Go-http-client/1.1`, and our edge treats that as a scanner signature and issues a challenge. A kubelet cannot solve a CAPTCHA, so the pull never succeeds and never will. It is not your credentials, not the pull secret, not TLS, and not an allowlist. ### What passes and what does not Measured against our registry: | User-Agent | Result | |---|---| | `containerd/1.7.0` | 200 | | `containerd/v2.1.4` | 200 | | `docker/29.4.0` | 200 | | `curl/8.7.1` | 200 | | `skopeo/1.24.0` | 200 | | `Go-http-client/1.1` | **403, challenged** | | `Go-http-client/2.0` | **403, challenged** | **containerd's own user agent is fine.** Any real one is. The failure is specifically the absence of a user agent, which Go turns into `Go-http-client`. ### Confirming it The blocked response identifies what it matched: ``` "user_agent": "Go-http-client/1.1", "ja4_fingerprint": null, "captcha_validated": false ``` Compare a manual pull against what the kubelet gets: ```bash # on the node: succeeds, because the CLI sends its own user agent skopeo inspect docker://registry.gen0sec.com/gen0sec/auth-api:0.2.0 ``` ```bash # what the kubelet sees kubectl -n gen0sec describe pod | grep -A3 Failed ``` Success from the CLI plus 403 from the kubelet confirms it. ### Two ways forward **Option 1: use the offline bundle method.** Nothing to configure on your nodes, and your own registry does not inspect user agents. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). **Option 2: make the pull path send a user agent**, on every node. :::note[Settle this before you choose an install method] If you cannot change container runtime configuration on your nodes, whether by policy or because your distribution manages it, use the offline bundle method. This constraint forces that choice, and it is better made before you start than at the first failed pull. ::: ## The relay cannot use an explicit egress proxy **Symptom.** You set `HTTPS_PROXY` on the relay. Nothing changes, and nothing errors. **Cause.** Every HTTP client in the platform builds its own transport without proxy support, so `HTTP_PROXY` and `HTTPS_PROXY` are ignored inside the cluster. This is by design. **What works instead.** | Option | Notes | |---|---| | Allow `api.gen0sec.com:443` directly | One firewall exception that bypasses the proxy | | Redirect transparently at the network layer | The client does not need to be proxy-aware | A TLS-intercepting proxy also cannot work: the charts expose no way to mount a CA bundle into the platform containers, and the relay rejects the intercepted certificate. ## The object store bucket is not created for you **Symptom.** Services start and then fail to read or write the object store. `download-api` answers `500 NoSuchBucket`. **Cause.** Nothing in the charts creates the bucket. **What to do.** [Install](./../05-install.mdx#5-create-the-object-store-bucket) step 5 creates it. It is a first-class step, not an optional one. ## Three services report readiness from a weaker probe `download-api`, `ids-rules-api` and `service-graph-api` serve a richer `/status/healthz` and, in doing so, never register `/status/readyz`. Requesting it returns 404, so a readiness probe pointed at it fails forever while liveness stays healthy, and the pod sits at `0/1` with zero restarts. **What that means for you.** Do not repoint those services at `/status/readyz`. It will fail. ## Helm never upgrades CRDs Not our defect, but it will bite you on the first upgrade that contains CRDs, so it is here. Helm installs a chart's CRDs on first install and ignores them from then on. An upgrade that ships a new CRD schema fails with a server-side-apply error, or silently rejects new fields. [Upgrade](./../operate/upgrade.mdx) applies them by hand as step 1. Do not skip it. ## Kafka volumes survive an uninstall `deleteClaim: false` is deliberate, so `helm uninstall` leaves the Kafka volumes behind. Reinstalling against them produces `Invalid cluster.id ... Expected X, but read Y`, because the new cluster generates a new identity. Delete the volume claims first, or restore the original cluster. See [Kafka will not start](./symptoms.mdx#kafka-will-not-start). ## The audit log needs new partitions after two years The upgrade to v0.2.0 creates monthly audit log partitions for the previous month and the next 24. Nothing creates later ones yet. Past that, audited actions fail with `no partition of relation "audit_log" found for row`, and rule changes made through the API are lost. Create partitions ahead of time, as the database superuser: ```sql SELECT create_audit_log_partition(date '2028-11-01'); ``` ## The audit log is never pruned It is append-only and kept indefinitely, and the application cannot delete from it. To remove old history, back up and then drop a monthly partition (`audit_log_YYYY_MM`) as the superuser. ## Emails are sent as "My App" Invitation and password-reset emails carry the sender name `My App`, whatever you configure. Only the address, `HOST_ALIAS_EMAIL`, is yours. ## New WAF rule functions need current agents `regex_replace()` and `http.request.raw_path` validate from v0.2.0. The rule language is generated from Synapse 0.8.5, and agents on an older version may not evaluate a rule that uses them. Update agents before you rely on either. ## Rule changes do not reach agents Creating, editing or deleting a rule, or switching a managed rule on or off, saves the change but does not regenerate the configuration agents fetch. The fleet keeps enforcing the previous rules. The rule write succeeds and the dashboard shows nothing; PostgreSQL logs a warning per write: ``` WARNING: notify_config_generator: POST to http://g0s-config-generator... failed: permission denied for function http_post ``` The database trigger that asks for the regeneration lacks the privileges to make the call. Restore them as the database superuser: ```sql ALTER FUNCTION public.notify_config_generator() SECURITY DEFINER; ALTER FUNCTION public.notify_config_generator() SET search_path = public, net, pg_temp; ``` Rules saved before that are picked up by the next rule change in the same workspace. ## Managed WAF rules do not work The dashboard cannot list managed WAF rules or switch them on or off, managed WAF rules are never sent to agents, and changing one fails with `column s.waf_rules_managed_id does not exist`. The database names the column `rule_managed_id`, and everything that reads it expects `waf_rules_managed_id`. ## Creating your organization ends on a page not found Right after first-run setup creates the organization, the dashboard sends you to `//check-workspace`, which does not exist. Nothing is wrong with the organization. Remove `/check-workspace` from the address to reach the workspace overview. ## The overview offers Threats The overview shows a Threats card, but the Threats module is not part of the self-hosted release and opening it fails. Ignore the card. ## Connection-state firewall rules match every packet A firewall rule with a connection-state condition, such as `allow established,related`, reaches agents without that condition. An allow rule then allows every packet in its direction, ahead of any deny it was written to protect, and a drop rule drops every packet. Nothing reports it. Do not use connection-state conditions on this version. Remove any rule that has one. ## arm64 nodes cannot run Postgres or the ingress operator The Postgres image is published for `linux/amd64` only, and the ingress operator's `linux/arm64` image carries an x86-64 binary. On an arm64 node one fails to pull and the other cannot start. Run both on `linux/amd64` nodes. ## Migration `force` means the opposite of what people expect `migrate force ` marks that version **applied** and continues with the next one. To re-run a migration that failed, force the version **before** it. Migrations are not idempotent. Forcing to the wrong version re-runs DDL that already exists and fails with `already exists`. Example: if `20261004150000` fails, run `force 20261004120000`. --- # Symptom index Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/troubleshoot/symptoms # Symptom index Everything here was reproduced on a real cluster. ## Images ### ImagePullBackOff Three different causes, and the first thing to establish is which. ```bash kubectl -n get pod -o jsonpath='{.spec.containers[*].image}' kubectl -n describe pod | tail -20 ``` | The image reference is | Cause | Fix | |---|---|---| | Correct, and the error mentions authorization | The `gen0sec-registry` secret is missing from that namespace | It is needed in **both** namespaces. See [Install step 2](./../05-install.mdx#the-image-pull-secret) | | Pointing at the wrong registry | A values file was not passed with `-f` | Re-run with `-f $B/values/-values-onprem.yaml` | | A path that does not exist in your registry | Mirror path mismatch, on the offline bundle method | See the next entry | ### 403 with a CAPTCHA challenge from our registry All images, not one, and only on the registry install method. **The tell:** a manual `docker pull` or `skopeo copy` from the same node succeeds while the kubelet's pull returns 403. ```bash # succeeds: the CLI sends its own user agent skopeo inspect docker://registry.gen0sec.com/gen0sec/auth-api:0.2.0 ``` ```bash # 403: what the kubelet gets kubectl -n gen0sec describe pod | grep -A3 Failed ``` On some distributions the component servicing the kubelet's pull sets no `User-Agent`, so Go sends `Go-http-client/1.1`, which our edge challenges as a scanner signature. A kubelet cannot solve a CAPTCHA, so it never recovers. Not credentials, not TLS, not the pull secret. Fix: set a user agent in the container runtime on every node, or use the offline bundle method. Both in [Known limitations](./known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ### ImagePullBackOff on a pod you did not configure Operator-launched pods (Postgres, the connection pooler, Kafka brokers, the entity operator) get their image reference from the **operator**, not from the chart. If one of those is stuck, the mirror path is wrong, not the pull secret. Confirm the path exists in your registry: ```bash curl -su : https://$REGISTRY/v2/_catalog?n=200 | tr ',' '\n' ``` Two paths look surprising and are not negotiable: | Image | Must be mirrored to | |---|---| | Kafka broker | `/gen0sec/strimzi/kafka:1.0.0-kafka-4.1.1` | | Entity operator | `/gen0sec/strimzi/operator:1.0.0` | Everything is under `gen0sec/`. Kafka additionally keeps the upstream `strimzi/` path inside it, because the operator's `defaultImageRegistry` replaces only the registry host and keeps the repository path. A Kafka pod asking for `/strimzi/kafka` with no `gen0sec/` means that setting lost the namespace. ### 401 Unauthorized pulling into a Postgres or pooler pod The Postgres operator does not copy `imagePullSecrets` onto the database pods it creates. They inherit them from their ServiceAccount, which `values-onprem.yaml` configures through `configKubernetes.pod_service_account_definition`. If you overrode `configKubernetes` wholesale, you dropped it. Merge instead of replacing. ### manifest unknown A chart is asking for a tag that was never published. Check what it rendered: ```bash helm get manifest -n | grep -E '^\s+image:' | sort -u ``` Then compare against [Component versions](./../configure/components.mdx). ## Postgres ### Postgres will not start `postgresql/core` sits in `CreateFailed`. ```bash kubectl -n gen0sec-system logs deploy/g0s-infra-postgres-operator | grep -i 'could not create' ``` | Log line contains | Cause | Fix | |---|---|---| | `endpoints is forbidden` | The operator's own ClusterRole omits `endpoints`, but it creates a master Endpoints object per cluster | The infra chart adds the missing rule. Confirm: `kubectl get clusterrole g0s-infra-postgres-operator-endpoints` | | `Unsupported value: "18"` | The CRDs in the cluster are older than the operator chart | Helm never upgrades CRDs. Apply them first. See [Upgrade](./../operate/upgrade.mdx) | | Nothing, and no volume is bound | No default StorageClass, or not enough capacity | See [Sizing](./../02-requirements.mdx#sizing-a-production-install) | After fixing the cause the operator does not retry promptly. Force it: ```bash kubectl -n gen0sec-system rollout restart deploy/g0s-infra-postgres-operator ``` ### Migration failures The migration job runs as a Helm hook. On failure Helm aborts and deploys nothing, so the cluster is in a safe state. ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` | Log line contains | Cause | Fix | |---|---|---| | `relation "..." does not exist` | The chain is running against a database that is not at the version it expects. Usually a forced version, or a `db-migrate` image older than the chart | Read the version table in the next entry before you force anything, then [collect diagnostics](./diagnostics.mdx) | | `role "..." does not exist` | A database role is missing. Older migrations `GRANT` to roles by name and roll back if one is absent. From v0.2.0 a missing role skips the grant with a `NOTICE`, and the service fails later with `permission denied` | The data chart creates them. Confirm: `kubectl -n gen0sec-system get secret \| grep credentials` | | `already exists` | A migration was forced to the wrong version and re-ran DDL | See the next entry | | `canceling statement due to lock timeout` | `20261004150000` could not lock the block events table within 10 seconds. Nothing changed | See [Upgrade](./../operate/upgrade.mdx#5-upgrade-the-platform) | | `no migration found for version` | The `db-migrate` image is older than the schema, typically after a rollback | Use the image of the newest release that ran | ### The schema is marked dirty The migration tool marks the schema dirty when a migration fails halfway, and refuses to continue until you say what happened. ```bash kubectl -n gen0sec-system exec -it core-0 -c postgres -- \ psql -U postgres -d ax_prod -c 'select * from schema_migrations' ``` `force ` means "**this version is applied**, continue with the next one". So to re-run a failed migration, force the version **before** it. Migrations are not idempotent. ## Kafka ### Kafka will not start Broker crash loops with `Invalid cluster.id ... Expected X, but read Y`. The volume survived a previous `helm uninstall`, which is deliberate, and the recreated cluster generated a new identity. Either restore the original cluster or wipe the volumes: ```bash kubectl -n gen0sec-system delete kafkanodepool kafka kubectl -n gen0sec-system delete pvc -l strimzi.io/cluster=core helm upgrade g0s-data ... ``` **This deletes the message data on those volumes.** ### No KafkaTopic resources exist although the chart rendered them Helm can skip re-applying objects that were deleted out of band. Re-apply the stored manifest: ```bash helm get manifest g0s-data -n gen0sec-system | kubectl apply -f - ``` ### Rule changes made through the signal API never appear The request is accepted, but the rule is not created. The signal worker writes an audit record in the same transaction as the rule, and drops the change if that fails. ```bash kubectl -n gen0sec logs deploy/g0s-signal | grep 'Failed to process v2 signal' ``` `permission denied for table audit_log` means the grant was skipped during the migration. `no partition of relation "audit_log"` means the audit log ran out of partitions. See [Known limitations](./known-limitations.mdx#the-audit-log-needs-new-partitions-after-two-years). ### Block events are missing from the dashboard Rows the database refused are parked on the `block_events_dlq` Kafka topic, with the reason in a `dlq_reason` header. Only the refused row is parked, not the batch it arrived in. telemetry-api also logs what it dropped or stored without a field, once a minute per organization: ```bash kubectl -n gen0sec logs deploy/g0s-telemetry-api | grep 'OTLP block events rejected' ``` ## Object store and cache ### Services cannot reach the object store The object store Service is `rustfs-svc`, with the suffix. `S3_ENDPOINT` and `ARXIGNIS_DATA_URL` must match it. ```bash kubectl -n gen0sec-system get svc | grep rustfs kubectl -n gen0sec get deploy g0s-download-api \ -o jsonpath='{.spec.template.spec.containers[0].env}' | tr ',' '\n' | grep -E 'S3_ENDPOINT|DATA_URL' ``` See [Install step 7](./../05-install.mdx#7-install-the-platform). ### 500 NoSuchBucket The bucket does not exist. [Install step 5](./../05-install.mdx#5-create-the-object-store-bucket) creates it. Nothing in the charts does. ### Everything works but reads are slow The shared cache is never fatal. Missing or unreachable, each service logs a warning once and runs on its in-process cache only. Confirm from the service side, not the pod list: ```bash kubectl -n gen0sec logs deploy/g0s-auth-api | grep -i "L2 cache" ``` `L2 cache (Redis) connected to server` means the cache is in use. `falling back to L1-only cache` means `REDIS_URL` is wrong or unreachable. ```bash kubectl -n gen0sec-system get svc dragonfly ``` The Service is deliberately not prefixed with the release name, because `REDIS_URL` names it directly. ### The cache pod crash loops ```bash kubectl -n gen0sec-system logs deploy/dragonfly --previous ``` | Log shows | Cause | Fix | |---|---|---| | `io_uring` errors | Blocked by the host kernel or a seccomp profile | Add `--force_epoll` to `dragonfly.extraArgs` in the infra values | | Read-only filesystem errors | The pod runs with a read-only root, which is why the chart disables snapshotting | Remove any flag you added that re-enables it | ### The cache is being evicted or OOM-killed `maxMemory` is the eviction threshold and the memory limit is the kill threshold. Keep the first comfortably below the second, or the kubelet kills the pod before it gets a chance to evict. ## Install and workstation ### The bundler fails with a yq syntax error You have the wrong `yq`. Two unrelated tools share the name. ```bash yq --version ``` The output must contain `mikefarah`. If not, install the Go version and point the bundler at it with `MIKEFARAH_YQ_PATH`. See [Requirements](./../02-requirements.mdx#check-your-yq). ### zsh: no matches found, before helm runs `zsh` expands `[0]` in a `--set` argument as a glob. Single-quote the whole argument: ```bash --set 'services.ui.ingress.hosts[0].host=example.internal' ``` ### The namespace hangs in Terminating Strimzi puts a finalizer on all 23 KafkaTopics, and once the operator is gone nothing clears it. ```bash kubectl get kafkatopics.kafka.strimzi.io -n gen0sec-system -o name \ | xargs -r -n1 kubectl patch -n gen0sec-system --type=merge \ -p '{"metadata":{"finalizers":null}}' ``` `uninstall.sh` does this for you, in the right order. See [Uninstall](./../operate/uninstall.mdx). ### A pod reports Running but nothing works A pod in `CrashLoopBackOff` still reports `status.phase: Running`. Read the `READY` column, never a field selector on phase. ```bash kubectl -n gen0sec get pods ``` ### Agents get 401 on every relayed artifact Your Gen0Sec API key is wrong, or the secret holds a trailing newline. ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` ``` artifact relayed from upstream path=/v1/ids-rules/version status=401 ``` Recreate the secret with `printf`, not `echo`. `--from-file` stores the bytes exactly, so a newline becomes part of the key. See [Data relay](./../configure/relay.mdx#getting-the-key-right). A missing secret does not produce this: the relay pod fails to start instead. See the next entry. ### CreateContainerConfigError on the relay pod The pod references a secret that does not exist. Almost always `gen0sec-download-proxy`: `downloadProxy.apiKeySecret.name` is set but the secret was never created. ```bash kubectl -n gen0sec describe pod -l app.kubernetes.io/name=download-api | tail -20 kubectl -n gen0sec get secret gen0sec-download-proxy ``` Create it. See [Install step 2](./../05-install.mdx#your-gen0sec-api-key). This failure is the visible one. The dangerous case is the opposite: leaving `apiKeySecret.name` unset renders no secret reference at all, the pod starts, and the failure is the silent 401 above. ### The relay reports 502 on every relayed route The upstream is unreachable. A 502 is deliberate rather than a 404 or 500: a polling agent reads it as "ask again later" and keeps the artifact it already holds. ```bash kubectl -n gen0sec exec deploy/g0s-download-api -- curl -fsS localhost:9999/status/healthz ``` Check that `api.gen0sec.com:443` is allowed. See [Ports and egress](./../security/egress.mdx). If you route egress through an explicit proxy, that is the cause: proxies are not honoured in-cluster. See [Known limitations](./known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy). ## Routing ### An agent gets HTML, or a redirect, from an API The request reached the dashboard's catch-all instead of the intended service. Routing is by longest matching prefix and the dashboard owns `/`, so any path missing from the route table lands there. ```bash kubectl -n gen0sec get ingress -o wide ``` Compare against [Publish the endpoints](./../06-publish.mdx#the-route-table). The failures this causes are quiet by design of HTTP, not by ours: | Missing route | What the agent sees | |---|---| | `/v1/authcheck` | A redirect and HTML. The agent cannot distinguish a bad key from a routing mistake and continues unverified | | `/v1/agents/events` | `SSE connect returned HTTP 404`, then silent fallback to polling, which looks like a slow platform | | `/v1/ban-ledger` | A redirect the agent reads as success. The contribution is lost silently | ### kubectl get ingress shows an empty HOSTS column The ingress controller ignores host-less rules: the object is created, nothing is logged, and no traffic is served. Set `entrypointHost`. See [Publish the endpoints](./../06-publish.mdx). ## Dashboard | Symptom | Cause | |---|---| | Components says it could not read infrastructure versions | The infra chart's `components.ui.*` values do not match the platform release, or a NetworkPolicy blocks the dashboard from the API server. See [Component versions](./../configure/components.mdx#verifying-what-you-are-actually-running) | | Components says data versions are unavailable | The relay or `ids-rules-api` is unreachable from the dashboard | | The online update check cannot reach `releases.gen0sec.com` | Expected when the cluster has no route out. Turn the check off | | No **Forgot your password?** link | SMTP is not configured. See [Invitations and email](./../configure/identity.mdx#invitations-and-email) | | Invitations show a link instead of being emailed | SMTP is not configured, or sending failed. Check `kubectl -n gen0sec logs deploy/g0s-ui` | ## Air gap and mirroring ### Confirming the cluster really is isolated From a pod, not from a node. ```bash kubectl -n gen0sec run netcheck --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/busybox:1.36.1-glibc -- wget -T5 -qO- https://ghcr.io/ ; echo "exit=$?" ``` A non-zero exit is the expected result. If it succeeds, the cluster still has egress and an install that appears to work may be pulling from the internet. ### An image is missing for one architecture You built the bundle with `--arch`. A single-architecture bundle fails when the kubelet tries to pull on a node of the other architecture, which on an isolated cluster is after the transfer. Rebuild without `--arch`, or with the architecture your nodes actually run. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). ## Still stuck [Collect diagnostics](./diagnostics.mdx), then send us the archive. --- # Verify the install Source: https://docs.gen0sec.com/docs/cerebellum/0.2.0/verify # Verify the install Running pods does not mean a working platform. A worker with no Service, a misconfigured credential, or a missing route all leave every pod `Ready` while no data moves. This page defines what "installed successfully" means. ## Before you start - [ ] [First sign-in](./07-first-signin.mdx) complete - [ ] An agent API key was created ## 1. Every pod is ready ```bash kubectl -n gen0sec-system get pods kubectl -n gen0sec get pods ``` Every pod reports `Running` and `n/n` in the `READY` column. On a correct install there are no exceptions: nothing is expected to sit unready. :::note[Check readiness, not phase] A pod in `CrashLoopBackOff` still reports `status.phase: Running`. A `--field-selector` on phase reports a broken install as clean, which is why the command above reads the `READY` column instead. ::: ## 2. The migration job is gone ```bash kubectl -n gen0sec get job g0s-db-migrate ``` The output is similar to this: ``` Error from server (NotFound): jobs.batch "g0s-db-migrate" not found ``` `NotFound` is the pass. Helm deletes the job the moment it succeeds. If it is still there, it failed: `kubectl -n gen0sec logs job/g0s-db-migrate`. ## 3. The relay is connected ```bash kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' ``` The output is similar to this: ```json { "status": "ready", "mode": "proxy", "upstream": "https://api.gen0sec.com", "external_relays": ["geoip", "ids-rules"] } ``` `"mode": "proxy"` confirms the relay came up. If the upstream is unreachable, relayed routes answer 502, which a polling agent reads as "ask again later" rather than "the artifact is gone". Then confirm it is actually authorized upstream, which the mode alone does not tell you: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means your Gen0Sec API key is wrong. ## 4. The platform serves an agent This is the check that matters. It makes the same requests an agent makes, from outside the cluster, over your published address. ```bash export GEN0SEC_BASE_URL=https://$ENTRYPOINT_HOST export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) $KIT/scripts/verify-deployment.sh ``` It checks four things: 1. The API refuses unauthenticated callers. 2. Configuration is served. 3. Threat intelligence and IDS rule bundles are published. 4. A downloaded bundle matches the SHA-256 its manifest advertises. It is read-only and safe to run against production. It exits non-zero and names the failing check. **Run it after every install and every upgrade.** It is the cheapest way to catch a deployment that looks healthy and serves nothing. ## Offline bundle only: confirm nothing pulls from outside ```bash kubectl -n gen0sec-system get pods -o yaml | grep -E '^\s+image:' | sort -u kubectl -n gen0sec get pods -o yaml | grep -E '^\s+image:' | sort -u ``` Every image listed is under your registry. Anything else means a values file was not passed with `-f`. :::note[Why both namespaces, and why read the YAML] The application runs in `gen0sec`; only the operators, object store and cache run in `gen0sec-system`. Checking one namespace covers well under half the images. Reading the YAML rather than a container template also catches init containers, which is where the object store's init image is. ::: ## Definition of installed successfully All five are true: 1. Every pod in both namespaces reports `Running` and `n/n` ready. 2. `kubectl -n gen0sec get job g0s-db-migrate` returns `NotFound` or appears as `Completed`. 3. `download-api` reports `"mode": "proxy"` with your upstream, and its log shows no `status=401` or `status=402` on relayed paths. 4. `verify-deployment.sh` exits zero. 5. On the offline bundle method, every image reference is under your registry. If all five hold, the install is done. If any fail, start at the [symptom index](./troubleshoot/symptoms.mdx). ## You may be interested in... - [Upgrade](./operate/upgrade.mdx), so you know how to move before you need to - [Roll back](./operate/rollback.mdx), and where the data boundary is - [Known limitations in this release](./troubleshoot/known-limitations.mdx) --- # Overview Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/ # Overview ## What is Cerebellum? Cerebellum is the enterprise version of the Gen0Sec platform that you can host in your own Kubernetes cluster. ## Charts | Installation order | Chart | Namespace | What it installs | |---|---|---|---| | 1. | `gen0sec-infra` | `gen0sec-system` | Three operators (Postgres, Kafka, ingress), the object store, and two caches: one shared by the application, one for the ingress data plane | | 2. | `gen0sec-data` | `gen0sec-system` | The Postgres cluster, the Kafka cluster, topics and database roles | | 3. | `gen0sec-platform` | `gen0sec` | Schema migrations, then the application services and the dashboard | The installation order is mandatory, as each chart creates resources the next one depends on. ## Installation methods ### Installation from registry (Recommended) Choose it when, - your cluster can access our registry, - you don't mind depending on an external registry ### Install via creating an offline bundle Choose it when, - your cluster don't have access to our registry, - you'd rather host mirrored images yourself, - you're facing issues pulling images from our registry :::note[Same content] Worry not, the offline bundle builds the same release and installs the same charts. ::: If you are unsure, go with **Install from registry**. You can always switch later. :::info[One thing that can decide this for you] Our registry challenges pulls that arrive with no `User-Agent`, and on some Kubernetes distributions the kubelet's pull path sends none. That is deliberate policy on our side, and the result is a 403 the kubelet cannot recover from. If you cannot change container runtime configuration on your nodes, use **Create an offline bundle**. Your own registry does not inspect user agents. See [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: You will need to decide the installation method in the [Prepare the artifacts](./04-prepare-artifacts.mdx) step. ## Connectivity :::warning[What 'offline' means] The 'offline' part in the offline bundler is used to describe the installation method, not the resulting product. ::: It's important to note that even by creating an offline bundle, doesn't mean you can operate the platform in an airgapped cluster that is entirely isolated from reaching the public internet. Cerebellum needs exactly one outbound destination from your cluster: - `api.gen0sec.com` on port `443`. That single connection carries **threat intelligence, GeoIP databases, ML models and IDS rules**. All four are licensed by their upstream providers and are not included in any of the artifacts we ship. Every agent in your fleet reaches it through one in-cluster service, so it is one destination to allow and one to audit. Nothing else in either namespace needs a route out. The dashboard can optionally check for new versions and send email; see [Network and connectivity](./03-network.mdx). Read [Network and connectivity](./03-network.mdx) before you start. It is the page to send to whoever runs your firewall. ## How long it takes Set aside **half a day for a first install**, and about an hour once you have done one. The install is 26 numbered steps across five pages, and most of the elapsed time is spent waiting on image pulls and on two clusters becoming ready, not on typing. The waits are gated explicitly, so the ceiling is knowable. These are the timeouts the commands themselves carry — the upper bound before a step is treated as failed, not a prediction: | Stage | You are waiting on | Gate | |---|---|---| | [Prepare the artifacts](./04-prepare-artifacts.mdx) | Downloads and signature verification. From the registry this is minutes; building an offline bundle copies every image and is the longest single stage | None | | [Install](./05-install.mdx) infra | Three operators and the object store | None | | [Install](./05-install.mdx) data | Kafka ready, then Postgres `Running` | `15m` each | | [Install](./05-install.mdx) platform | Schema migrations, then every pod ready | `20m` + `10m` | | [Publish](./06-publish.mdx) | The ingress rollout | `20m` | | [First sign-in](./07-first-signin.mdx) | You: MFA enrolment, a forced password change, the onboarding wizard | None | | [Verify](./08-verify.mdx) | Four checks you run | None | What actually decides your number is how fast your nodes pull images. Everything else is bounded by the table above. :::note[The Quickstart is not a shortcut] It is the same procedure with one extra values file per chart, on one node. It is quicker because a single-node cluster pulls and schedules less — not because it skips steps. ::: ## Before you install 1. [Requirements](./02-requirements.mdx). Cluster, tools, and your access token. 2. [Network and connectivity](./03-network.mdx). What to allow, and why. 3. [Prepare the artifacts](./04-prepare-artifacts.mdx). Prepare for the installation. If you want a working instance to look at before you plan a real install, use the [Quickstart](./quickstart.mdx). It builds a throwaway single-node install and tears it down again. --- # Component versions Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/configure/components # Component versions :::note[Read this from your release, not from this page] Every release pins its own versions, and the authoritative list is in the install kit you downloaded. ```bash yq '.spec.images[] | [.image, .imageTag, .digest] | @tsv' $KIT/release-manifest.yaml yq '.spec.charts[] | [.chart, .version, .digest] | @tsv' $KIT/release-manifest.yaml ``` This page describes what those versions mean and which of them have surprising requirements. ::: {/* This page should be generated from release/components.yaml and release-manifest.yaml in CI, per the structure proposal section 7. Hand-maintained, it drifts within one release and is then worse than nothing, because operators trust it. */} ## Third-party components | Component | Version | Notes | |---|---|---| | Postgres operator | 2.0.1 | Zalando. **1.15.x cannot be used**: its schema rejects PostgreSQL 18 | | PostgreSQL | 18, with extensions | Includes AGE, TimescaleDB, `pg_cron`, `pg_net`, `pglogical`, `pgsodium` | | Connection pooler | 2.0.1 | Operator-launched. Services connect through it, never directly | | Logical backup | 2.0.1 | Only used when logical backups are enabled | | Kafka operator | 1.0.0 | Strimzi. v1 CRD API only | | Apache Kafka | 4.1.1 | **KRaft, no ZooKeeper.** Operator-launched | | Object store | Chart 0.10.0 | S3-compatible. Its Service has a `-svc` suffix | | Shared cache | 1.38.1 | Redis protocol. Application services degrade to in-process caching without it | | Ingress cache | Valkey | Deployed by the ingress chart and used only by it. Kept separate on purpose, so a shared-cache outage cannot take the ingress data plane with it. One replica: a restart costs a cold start, not data | | Ingress operator | Commit-tagged | `ingressClassName: synapse`. Built in a sibling repository, so the tag is a commit hash rather than a version | | Platform services | The release version | Resolved from the registry host, namespace and image tag | ## Mirror paths, for the offline bundle method Every mirrored image lands under `gen0sec/`. Two paths look wrong and are not negotiable. | Image | Must be at | |---|---| | Kafka broker | `/gen0sec/strimzi/kafka:1.0.0-kafka-4.1.1` | | Kafka entity operator | `/gen0sec/strimzi/operator:1.0.0` | **Why Kafka keeps an extra path segment.** The Kafka operator's image-registry setting replaces only the registry *host* and preserves the repository path. So the chart points it at `/gen0sec` rather than ``, and the upstream `strimzi/` segment survives inside our namespace. :::note[This exact thing shipped wrong once] With the namespace omitted, the chart asked for `/strimzi/operator` while the release mirrored `/gen0sec/strimzi/operator`. One path segment apart, and the entire Kafka stack failed to pull on a cluster with no fallback. Every chart is now rendered in CI and the build fails on any image reference not backed by the release manifest, so the two cannot drift again. That gate is also why the table above is trustworthy: it catches a chart asking for a tag nobody publishes, which is how the ingress operator once requested a version that was never released. ::: `bundler.sh mirror` reproduces these paths exactly. You do not compute them. ## Not everything a chart renders is mirrored The vendored Kafka chart hardcodes an image map covering four Kafka versions plus Connect and Bridge build tooling. This edition ships one Kafka version and no Connect clusters, so the rest are deliberately absent from the bundle. That is roughly 1.5 GB of images nothing ever pulls. **If you enable Kafka Exporter, Cruise Control, Kafka Bridge or Connect builds, mirror their images first.** The release does not carry them. ## Kubernetes | | | |---|---| | Supported | 1.23 or later | See [Requirements](./../02-requirements.mdx#kubernetes-versions). ## Support policy | | | |---|---| | Supported releases | The latest two minor trains | | Upgrade path | Sequential minors. **Skipping a minor is not supported.** Patches in place | | Upgrade order | Always infra, then data, then platform | | CRDs | Helm never upgrades them. Apply them before the chart. See [Upgrade](./../operate/upgrade.mdx) | | Downgrade | Not supported across a minor that ran a schema migration. Within a train, use rollback. See [Roll back](./../operate/rollback.mdx) | | Kubernetes upgrades | Independent of ours. Stay within the supported range | ## Verifying what you are actually running The dashboard shows it under **Organization settings**, **Components**: what each component is running, and the latest version known to this release. Each chart records its versions in a ConfigMap, which you can also read directly: ```bash kubectl -n gen0sec get cm g0s-installed-versions -o jsonpath='{.data.versions\.json}' kubectl -n gen0sec-system get cm g0s-infra-installed-versions -o jsonpath='{.data.versions\.json}' kubectl -n gen0sec-system get cm g0s-data-installed-versions -o jsonpath='{.data.versions\.json}' ``` Two things about that page: - **Check for updates online** is off by default and needs `org:write`. When on, the dashboard fetches the latest release's version list from `releases.gen0sec.com` at most once an hour. Nothing about your install is sent. Air-gapped installs leave it off. - Opening it creates an API key named **Components tab (internal)**, which it uses to read data versions from the relay. Deleting it is safe; it is recreated on the next visit. If the page says it could not read infrastructure versions, the infra chart's `components.ui.namespace` and `components.ui.serviceAccountName` do not match the platform release. See [Values reference](./values.mdx#gen0sec-infra). The manifest says what a release pins. This says what your cluster pulled: ```bash kubectl -n gen0sec-system get pods -o yaml | grep -E '^\s+image:' | sort -u kubectl -n gen0sec get pods -o yaml | grep -E '^\s+image:' | sort -u ``` Reading the YAML rather than a container template also catches init containers, which is where the object store's init image is. On the offline bundle method, every reference should be under your own registry. Anything else means a values file was not passed. --- # Sign-in and identity Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/configure/identity # Sign-in and identity ## What is available | Method | Status | Where credentials live | |---|---|---| | Email and password | **Available** | Your Postgres | | Passkeys | **Available** | Your Postgres holds the public key. The private key never leaves the user's device | | Multi-factor authentication | **Available, and required during onboarding** | Your Postgres | | Password reset by email | **Available when [SMTP is configured](#invitations-and-email)** | Your mail server sends the link | | Microsoft Entra ID | **In development, not supported at the moment ** | | **Nothing outside your cluster is involved in sign-in.** Accounts, password hashes, passkeys and sessions are all in your own database. If you configure SMTP, invitation and password-reset emails go out through your mail server. :::note[Single sign-on is not available yet] Microsoft Entra ID is in development. No generic OIDC provider, no SAML and no LDAP integration exists either. If single sign-on is a requirement for you, tell us which provider, because that shapes what ships first. ::: ## The first account Registration is invite-only and invites need an existing member, so a fresh install seeds one account with a **published password**. Signing in, completing onboarding and changing that password is a required install step, not an optional one. See [First sign-in](./../07-first-signin.mdx). ## The published address decides whether sign-in works The dashboard derives its sign-in base URL from the host you publish it on. If that does not match the address people actually type in the browser, every sign-in is rejected as an invalid origin. Two rules: 1. **Use the address people type.** Not an internal service name, not a load balancer's own hostname. 2. **If TLS terminates upstream of the ingress**, set the scheme explicitly, because the chart cannot infer it and guesses `http` while the browser is on `https`. ```bash --set externalScheme=https ``` Leave that unset when TLS terminates at the ingress. See [Publish the endpoints](./../06-publish.mdx#if-tls-terminates-upstream). ## Passkeys Users can register with a passkey instead of a password, and add, rename or remove passkeys under **Profile**. Signing in with a passkey does not ask for the authenticator-app code: the passkey's own PIN or biometric check stands in for it. Enrolling multi-factor is still required during onboarding. A passkey is bound to the host name the dashboard is published on, taken from its sign-in URL. So: - **They need HTTPS and a DNS name.** A plain-HTTP address or a bare IP cannot register one. - **Changing `entrypointHost` makes every registered passkey stop working.** Users then sign in with a password, or register a new passkey on the new address. - **A user who registered with a passkey only has no password to fall back on.** Ask them to set one under **Profile**, especially when SMTP is not configured and nobody can reset it by email. ## Invitations and email Invitations grant one or more workspaces, each with a role no higher than the inviter's own. Pending invitations are listed on the **Team** tab, where they can be revoked. A link expires after seven days, and inviting the same address again replaces the previous link. With SMTP configured, the invitation is emailed and users can reset a forgotten password. Without it, the dashboard shows the invitation link for you to copy and send yourself, and the sign-in page offers no password reset. SMTP is configured through the dashboard's environment. The chart has no dedicated keys for it, so set them under `services.ui` in your overrides file, with the credentials in a Secret you create: ```bash kubectl -n gen0sec create secret generic g0s-ui-smtp \ --from-literal=user='' --from-literal=password='' ``` ```yaml title="my-overrides.yaml" services: ui: env: HOST_EMAIL: smtp.example.com HOST_PORT: "587" HOST_ALIAS_EMAIL: no-reply@example.com HOST_APP_URL: https://cerebellum.example.com envSecrets: HOST_AUTH_USER: { name: g0s-ui-smtp, key: user } HOST_AUTH_PASS: { name: g0s-ui-smtp, key: password } ``` | Variable | Notes | |---|---| | `HOST_EMAIL` | The SMTP server's host name | | `HOST_PORT` | `587` by default, with STARTTLS. `465` switches to implicit TLS | | `HOST_AUTH_USER`, `HOST_AUTH_PASS` | **Required.** SMTP is only used when the host and both credentials are set. A relay without authentication is not supported | | `HOST_ALIAS_EMAIL` | **Required.** The sender address | | `HOST_APP_URL` | **Required.** The address people type, with its scheme. Links in emails are built from it, and it is not derived from `entrypointHost` | Pass the file to every later `helm upgrade` as well. The dashboard's outbound connection to your mail server is listed on [Ports and egress](./../security/egress.mdx). ## Sessions The signing secret lives in `g0s-db-ui`, created with all the other database secrets in [Install step 6](./../05-install.mdx#6-create-the-application-secrets). | Action | Effect on sessions | |---|---| | Re-running `make-db-secrets.sh` | None. The existing signing secret is preserved deliberately | | Rotating the signing secret | Every session ends and every outstanding invite link stops working. Passwords and passkeys are unaffected | | Changing `entrypointHost` | Registered passkeys stop working | | Upgrading | None. Upgrading from v0.1.x to v0.2.0 invalidates invite links issued before the upgrade | ## Turning the dashboard off entirely If your operators use the API only: ```bash --set services.ui.enabled=false ``` The agent-facing APIs are unaffected. Note that agent API keys are issued through the dashboard, so turn it off only once you have the keys you need. ## If it fails | Symptom | Cause | |---|---| | Sign-in rejected as an invalid origin | The published host does not match the address typed, or the scheme is wrong. See above | | Sessions drop after a re-run of the secret script | The signing secret changed. It should not; check whether the secret was deleted first | | Every sign-in fails after an upgrade | Check that `externalScheme` and the host survived the upgrade. `helm get values g0s -n gen0sec` | | The passkey prompt fails or offers no passkey | The address typed differs from `entrypointHost`, the page is plain HTTP, or the host changed after the passkey was registered | | Emailed links start with `undefined/` | `HOST_APP_URL` is not set | | Invitations show a link instead of being emailed | SMTP is not configured, or sending failed. Check `kubectl -n gen0sec logs deploy/g0s-ui` | --- # Reference configurations Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/configure/reference-configs # Reference configurations Two named configurations rather than a blank values file. Each ships in the install kit, and each is reproduced in full on this page, so you can read it before you download anything. | Configuration | For | Redundancy | Storage | |---|---|---|---| | [Single node](#single-node) | Evaluation, training, upgrade rehearsal | **None** | Smallest | | [Production](#production) | Anything you depend on | Full | Smallest production grade | ## Single node **Not for production.** Everything at one replica on one node. Losing the node loses the data. Ships as three overlays in the kit: ``` $KIT/examples/single-node/infra-values.yaml $KIT/examples/single-node/data-values.yaml $KIT/examples/single-node/platform-values.yaml ``` Layer each one **after** the production values file, so it wins. Full procedure: [Quickstart](./../quickstart.mdx). | | Production | Single node | |---|---|---| | Object store | 3 replicas, erasure coded, StatefulSet | 1 replica, standalone, **Deployment** | | Postgres | 3 instances, both poolers on | 1 instance, replica pooler off | | Kafka | 3 brokers, replication factor 3 | 1 broker, every factor 1 | | Services | 2 or 3 replicas | 1 each | | StorageClass | Yours, set explicitly | The cluster's default (empty in the overlay) | ### What the overlays set Verbatim from the v0.2.1 kit, so you can read them without unpacking it first.
infra-values.yaml — object store and cache ```yaml title="$KIT/examples/single-node/infra-values.yaml" # ============================================================================ # gen0sec-infra — SINGLE-NODE EVALUATION overlay # ============================================================================ # Layered ON TOP of values-onprem.yaml (pass both, in this order): # helm install g0s-infra charts/gen0sec-infra -n gen0sec-system \ # -f values/gen0sec-infra-values-onprem.yaml \ # -f examples/single-node/infra-values.yaml # # Shrinks the footprint to fit one worker node. NOT for production: the object # store loses erasure coding and every component becomes a single point of # failure. Use it for evaluation, training, and dry-running an upgrade. # ============================================================================ rustfs: # Distributed mode needs >= 2 nodes for erasure coding; on one node the pods # would schedule but the cluster is not fault-tolerant anyway, so run the # single-process server and be explicit about it. mode: standalone: enabled: true distributed: enabled: false replicaCount: 1 storageclass: # Empty = the cluster's default StorageClass, matching values-onprem.yaml. On # k3s that resolves to local-path, which this overlay used to hardcode — but a # literal `local-path` exists on no other distribution, so the PVCs bound # nowhere and only surfaced as pods Pending past the rollout timeout. name: "" dataStorageSize: 20Gi logStorageSize: 5Gi dragonfly: # Cache only — shrink it so the node's RAM goes to Postgres and Kafka. Keep # maxMemory under resources.limits.memory so it evicts instead of being killed. maxMemory: 256mb proactorThreads: 1 resources: requests: { cpu: 100m, memory: 128Mi } limits: { cpu: 500m, memory: 512Mi } ```
data-values.yaml — Postgres and Kafka ```yaml title="$KIT/examples/single-node/data-values.yaml" # ============================================================================ # gen0sec-data — SINGLE-NODE EVALUATION overlay # ============================================================================ # helm install g0s-data charts/gen0sec-data -n gen0sec-system \ # -f values/gen0sec-data-values-onprem.yaml \ # -f examples/single-node/data-values.yaml # # One Postgres instance, one Kafka node, replication factor 1. Any node loss is # data loss — evaluation only. # ============================================================================ postgres: numberOfInstances: 1 # With a single instance there is no replica to pool. enableReplicaConnectionPooler: false volume: size: 20Gi # Empty = the cluster's default StorageClass (see infra-values.yaml). storageClass: "" resources: requests: { cpu: 500m, memory: 1Gi } limits: { cpu: "4", memory: 6Gi } parameters: shared_buffers: 512MB effective_cache_size: 1536MB max_connections: "400" max_worker_processes: "16" max_parallel_workers: "8" timescaledb.max_background_workers: "8" kafka: replicas: 1 storage: size: 20Gi # Empty = the cluster's default StorageClass (see infra-values.yaml). class: "" # min.insync.replicas must stay < replicas, so every factor drops to 1 # together with the node count. config: offsets.topic.replication.factor: 1 transaction.state.log.replication.factor: 1 transaction.state.log.min.isr: 1 default.replication.factor: 1 min.insync.replicas: 1 log.compression.type: zstd topicDefaults: partitions: 1 replicas: 1 ```
platform-values.yaml — one replica per service ```yaml title="$KIT/examples/single-node/platform-values.yaml" # ============================================================================ # gen0sec-platform — SINGLE-NODE EVALUATION overlay # ============================================================================ # helm install g0s charts/gen0sec-platform -n gen0sec \ # -f values/gen0sec-platform-values-onprem.yaml \ # -f examples/single-node/platform-values.yaml \ # --set global.imageTag=$VERSION --timeout 20m # # One replica of everything: no HA, and a rolling update is a short outage per # service. Resource requests are unchanged — they are already small. # ============================================================================ services: auth-api: { replicaCount: 1 } ban-ledger-api: { replicaCount: 1 } config-api: { replicaCount: 1 } data-api: { replicaCount: 1 } download-api: { replicaCount: 1 } feeds-api: { replicaCount: 1 } ids-rules-api: { replicaCount: 1 } rules-validator: { replicaCount: 1 } service-graph-api: { replicaCount: 1 } signal-api: { replicaCount: 1 } telemetry-api: { replicaCount: 1 } threat-api: { replicaCount: 1 } agent-status: { replicaCount: 1 } ui: { replicaCount: 1 } ```
## Production The default. `values-onprem.yaml` in the bundle **is** the production configuration, so there is no overlay to apply: what you install in [Install](./../05-install.mdx) is this. What you set yourself: ```yaml # gen0sec-infra rustfs: storageclass: name: your-block-storage-class # gen0sec-data postgres: volume: storageClass: your-block-storage-class kafka: storage: class: your-block-storage-class ``` For requirements see [Sizing](./../02-requirements.mdx#sizing-a-production-install). ## Extending retention The shipped sizes are deliberately modest, so growing is the normal direction. Which key you grow depends on what you are retaining. See [Sizing storage for your retention](./../02-requirements.mdx#sizing-storage-for-your-retention). eg. ```yaml # gen0sec-data postgres: volume: size: 200Gi storageClass: your-block-storage-class kafka: storage: size: 200Gi class: your-block-storage-class ``` ```yaml # gen0sec-infra rustfs: storageclass: name: your-block-storage-class dataStorageSize: 100Gi ``` :::note[You can also grow later] Volume expansion grows a volume if your StorageClass has `allowVolumeExpansion: true`. Starting at the defaults and expanding when you need to is a reasonable plan, and it is the direction that works: nothing shrinks a volume. ::: ## Publishing endpoints Independent of the three above, and needed by all of them: ``` $KIT/examples/single-entrypoint/ingress-values.yaml ``` One address for the dashboard and every agent-facing API, routed by path prefix. See [Publish the endpoints](./../06-publish.mdx) for the route table and the two values you set.
The header of that file, with both customer inputs and why they exist The rest of the file is the per-service route list. It is reproduced as a table on [Publish the endpoints](./../06-publish.mdx#the-route-table) rather than twice here. ```yaml title="$KIT/examples/single-entrypoint/ingress-values.yaml (head)" # ============================================================================ # gen0sec-platform — SINGLE-ENTRYPOINT INGRESS overlay # ============================================================================ # Publishes the dashboard and the agent-facing APIs through ONE address, served # by the Synapse proxy that gen0sec-infra deploys for the `synapse` IngressClass. # # helm upgrade --install g0s charts/gen0sec-platform -n gen0sec \ # -f values/gen0sec-platform-values-onprem.yaml \ # -f examples/single-entrypoint/ingress-values.yaml # # Layered ON TOP of values-onprem.yaml, which leaves every ingress disabled. # That default is deliberate — turning this on makes the APIs reachable from # wherever the proxy Service is reachable — so this is opt-in, not the default. # # A HOST IS REQUIRED — set `entrypointHost`, either below or with # `--set entrypointHost=...`. The chart reads it and publishes every rule under # it, so it is named once and nowhere else. # # It used to be a YAML anchor aliased into each rule. That could only ever be # changed by editing this file: anchors are expanded by the YAML parser before # Helm merges `--set`, so `--set entrypointHost=...` rewrote the top-level key # while every aliased copy kept `gen0sec.internal`, and the install published # on a host nobody could reach. The chart consumes the value now, so the # documented `--set` works. # # Leaving it empty FAILS THE RENDER rather than publishing anything. Host-less # rules do not work with this controller: synapse's upstreams.yaml is keyed by # host, and the operator skips any rule with an empty host # (controllers/ingress_controller.go, "if host == \"\" || rule.HTTP == nil # { continue }"). Such a rule produces an Ingress object that its own controller # silently ignores, with no event and no log line, so the chart refuses to # render one at all. # # The dashboard needs this host to be the address people actually type: # BETTER_AUTH_URL is derived from it (services.ui.urlFromIngress), and sign-in # breaks otherwise. # # ROUTING IS BY PATH PREFIX, because one address serves every backend. The # prefixes below are the real route roots, read off each service's router -- # they are not a naming convention and do not survive being guessed at. # # One overlap to keep in mind, resolved by longest-prefix matching: # /v1/agents/events config-api serves this; data-api owns /v1/agents # If a request lands on the wrong backend, that is the first thing to check. # # /v1/ids-rules goes to download-api, NOT the in-cluster ids-rules-api: in proxy # mode IDS rules are relayed from the platform, and ids-rules-api serves a local # ET ingest that this edition does not run. # ============================================================================ # CUSTOMER-INPUT: the single address every rule below is published under. entrypointHost: "" # CUSTOMER-INPUT: the scheme users actually reach that address on. # # Only needed when TLS terminates UPSTREAM of the Ingress -- a cloud load # balancer, a CDN, a tunnel. The chart otherwise infers the scheme from the # Ingress's own `tls:` block, which is absent in that setup, so it guesses http # while the browser is on https. The dashboard hands this to Better Auth as its # base URL and every sign-in is then rejected with "Invalid origin". # # Leave it unset when TLS terminates at the Ingress (i.e. you set ingress.tls), # and the chart reads the scheme off that TLS block instead. # # A single service can still override this with # `services..ingress.externalScheme`, which wins over this value. externalScheme: https ```
## Keeping your overrides Put them in a file under version control, not in `--set` arguments. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ -f my-site/platform.yaml \ --set global.imageTag=$VERSION \ --timeout 20m ``` `helm upgrade` uses only what you pass it. Anything set with `--set` on a previous install and omitted now reverts to the chart default, silently. A file makes that impossible to forget. Keep `entrypointHost` in that file. The ingress overlay ships it empty, and the render fails without it. Pass your file after the overlay so your value wins. Two things that are awkward to keep in a file, because they are per-release or per-secret: | Keep as `--set` | Why | |---|---| | `global.imageTag` | Changes every release | | `downloadProxy.apiKeySecret.name` | Or put it in your file. Just never omit it. See [Data relay](./relay.mdx) | ## What the kit contains ```bash ls $KIT/examples/*/ ``` Everything under `examples/` is a starting point you are meant to read and edit, not a black box. Each file carries comments explaining which values are yours to set. --- # Threat intelligence and the data relay Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/configure/relay # Threat intelligence and the data relay The relay is one in-cluster service, `download-api`. It is the only component with a route out, and it is how your fleet gets threat intelligence. This is not a choice you make. It is a requirement. See [Known limitations](./../troubleshoot/known-limitations.mdx#no-zero-egress-deployment). ## What it relays Read-only, upstream-produced content only. | Class | Routes | Why it is not shipped | |---|---|---| | CTI indicators | `/v1/indicators/version`, `/v1/indicators/download` | Built by the upstream intelligence pipeline | | ML models | `/v1/models/*` and the per-model aliases | Trained and published by Gen0Sec | | GeoIP | `/v1/geoip/manifest`, `/v1/geoip/{file}/download` | Licensed by MaxMind and IPInfo. The IPInfo Lite bundle is the default source | | IDS rules | `/v1/ids-rules/*` | Licensed as part of the Emerging Threats ruleset | **It is a relay, not a cache.** Nothing is stored on the way through. There is no local artifact store to size, no synchronization job to supervise, and no staleness window. An agent that polls gets whatever upstream has at that moment. **The relayed URLs are identical to the direct ones**, so your agents need no configuration change. ## What it never relays | Class | Routes | Why | |---|---|---| | Workload identity | `/v1/identity/*` | Produced in your cluster, describes your workloads | | Policy edges | `/v1/policy-edges/*` | Pod IPs, namespaces, NetworkPolicy edges | | Fleet bans | `/v1/fleet-bans/*` | Your own agents' ban decisions | These three are tenant-scoped: they describe your infrastructure or your decisions. Relaying them would push that to a service you do not run. They keep the local object store path behind the local authorization check, in relay mode exactly as without it. Uploads are never relayed. The relay is read-only, so `PUT` and `POST` on the download surface are served locally or not at all. ## How authorization works Two hops, two credentials, checked in two places. | Hop | Credential | Checked by | |---|---|---| | Agent to relay | The agent's own key, issued by your dashboard | Your cluster. Relayed routes sit behind the local auth middleware and each carries the same permission it has in direct mode | | Relay to Gen0Sec | Your Gen0Sec API key | Gen0Sec | **A caller's key is never forwarded upstream on a self-hosted deployment.** A locally-issued key cannot authenticate to gen0sec. Forwarding it would leak a local credential to a third party without making the request succeed. With no Gen0Sec API key configured, the relay sends no credential at all. ## The setting that must be right ```yaml downloadProxy: enabled: true upstream: "https://api.gen0sec.com" apiKeySecret: name: gen0sec-download-proxy # named by default key: DOWNLOAD_PROXY_API_KEY ``` ### Why the key is required Your agents' keys are issued by **your** dashboard and exist only in **your** database, so they cannot authenticate to gen0sec. The relay presents your own Gen0Sec API key instead, and that is the only credential the upstream will accept from your installation. Without it, the relay sends no credential, Gen0Sec answers 401, and no relayed artifact reaches your fleet. :::note[The chart names the secret by default, so this fails loudly now] `apiKeySecret.name` defaults to `gen0sec-download-proxy`. If that secret does not exist, the relay pod never starts: `CreateContainerConfigError`, visible immediately. That is deliberate. An empty name renders no environment variable at all, the relay starts, and the only symptom is every relayed class answering 401 while agents report empty rulesets rather than errors. A failed rollout at install time is the cheaper failure. Blank the name only if you relay through a gateway that supplies its own credential. ::: ### Getting the key right ```bash printf '%s' '' > ~/.gen0sec/relay-key kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-file=DOWNLOAD_PROXY_API_KEY=$HOME/.gen0sec/relay-key ``` The value renders as a `secretKeyRef`, so it never appears in a manifest or in a values file. ## All settings | Key | Default | Meaning | |---|---|---| | `downloadProxy.enabled` | `true` | Turning it off is not supported | | `downloadProxy.upstream` | `https://api.gen0sec.com` | The upstream base URL. A malformed value **fails startup** rather than degrading | | `downloadProxy.external` | `geoip,ids-rules` | Which external classes to relay. An unknown name fails startup | | `downloadProxy.timeout` | `60s` | Upstream request timeout, matching the router's own | | `downloadProxy.apiKeySecret.name` | `""` | **Set this.** See above | | `downloadProxy.apiKeySecret.key` | `DOWNLOAD_PROXY_API_KEY` | The key within the secret | :::note[Failing startup is deliberate] A malformed upstream is not treated as "fall back to reading from the local object store". That would leave every agent poll erroring against a store this deployment does not populate. Failing fast is the safer wrong answer. ::: ## Verifying ```bash kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' ``` The output is similar to this: ```json { "status": "ready", "mode": "proxy", "upstream": "https://api.gen0sec.com", "external_relays": ["geoip", "ids-rules"] } ``` **That tells you the relay started, not that it is authorized.** For that, read the log: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means your Gen0Sec API key is missing or wrong. ## Response codes worth knowing | Code | Means | What an agent does | |---|---|---| | 200 | Relayed successfully | Uses the artifact | | **401** | The upstream rejected the credential | **Reports an empty ruleset, not an error.** This is the silent failure | | **502** | The upstream is unreachable | Reads it as "ask again later" and keeps what it holds. Correct degradation | | 404 | The artifact does not exist upstream | Treats it as withdrawn | The 401 and 502 rows are the reason this page exists. ## Preview a change before applying it ```bash helm template g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set downloadProxy.apiKeySecret.name=gen0sec-download-proxy \ | grep -A3 DOWNLOAD_PROXY ``` --- # Secrets Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/configure/secrets # Secrets **No chart contains a credential.** Every one is referenced by name. This page is explains what must exist, with which keys, before the platform starts. ## The secrets you create | Secret | Namespace | Keys | Created in | |---|---|---|---| | `gen0sec-registry` | **Both** | `.dockerconfigjson` | [Install step 2](./../05-install.mdx#the-image-pull-secret) | | `gen0sec-object-store` | `gen0sec-system` | `RUSTFS_ACCESS_KEY`, `RUSTFS_SECRET_KEY` | [Install step 2](./../05-install.mdx#the-object-store-credentials) | | `gen0sec-download-proxy` | `gen0sec` | `DOWNLOAD_PROXY_API_KEY` | [Install step 2](./../05-install.mdx#your-gen0sec-api-key) | The pull secret is needed in **both** namespaces. The application runs in `gen0sec`; the operators, object store and cache run in `gen0sec-system`. ## The twenty the script creates `make-db-secrets.sh` composes one secret per service from the passwords the Postgres operator generated. | Secret | Contains | Role it uses | Endpoint | |---|---|---|---| | `g0s-db-` | `DATABASE_URL`, plus object store keys for services that use it | Its own per-service role, read-only where it only reads | The connection pooler | | `gen0sec-db-migrate` | `DATABASE_URL` | `postgres`, superuser | The primary, unpooled | | `g0s-db-ui` | `DATABASE_URL`, `BETTER_AUTH_SECRET`, object store keys | `frontend` | The connection pooler | One more is optional and not created by the script: SMTP credentials for invitation and password-reset email, referenced from `services.ui.envSecrets`. See [Invitations and email](./identity.mdx#invitations-and-email). A referenced Secret that does not exist leaves the dashboard in `CreateContainerConfigError`. Every service connects as its own role. The migration job needs a superuser on the primary because it runs `CREATE EXTENSION`, which an owner role cannot do. :::note[The session signing key is preserved] `g0s-db-ui` carries the dashboard's session signing secret. Re-running the script keeps the existing value, so sessions and invite links stay valid across a re-run. Nothing else regenerates it. ::: Four services need no database connection and get no `DATABASE_URL`: the GeoIP updater, the GeoIP database updater, the model publisher and the rules validator. ## Where the database passwords come from The Postgres operator creates a credentials secret per role it provisions, named after the role: ``` .core.credentials.postgresql.acid.zalan.do ``` Underscores in the role become dashes in the secret name. Compose each connection string from that role's password: ``` postgres://:@core-pooler.gen0sec-system.svc.cluster.local:5432/ax_prod?sslmode=require ``` | Consumer | Role | Endpoint | |---|---|---| | Each service | Its own per-service role | `core-pooler` | | Migration job | `postgres` | `core`, the primary, with the on-prem marker appended | | Dashboard | `frontend` | `core-pooler` | The migration job's connection string goes to the primary and carries the on-prem marker, `options=-c%20gen0sec.deployment_mode%3Donprem`: ``` postgres://postgres:@core.gen0sec-system.svc.cluster.local:5432/ax_prod?sslmode=require&options=-c%20gen0sec.deployment_mode%3Donprem ``` Migrations read the marker to tell your database from our hosted one. Without it the seeded admin account is not created, and from v0.2.1 rule changes stop reaching agents. :::note[One deliberate exception] The dashboard's connection string uses `sslmode=no-verify`. Its client library verifies the certificate chain for `sslmode=require` and rejects the pooler's self-signed certificate; the Go library every other service uses reads the same value as encrypt-without-verify. **Do not copy `no-verify` onto the other services.** Their driver rejects it outright. ::: ## For production: use your secret manager `make-db-secrets.sh` is fine for evaluation. In production, treat its output as a **specification** rather than a tool, and have your secret manager produce secrets of the same shape. Two reasons: 1. The script writes plain Kubernetes secrets from operator-generated passwords. Your manager should be the source of truth, with the cluster holding copies. 2. Rotation is then a manager operation rather than a re-run plus a rollout. An external secret can template the connection string from the operator-generated password, so nothing is copied by hand: ```yaml # shape, not a working manifest: adapt to your operator data: - secretKey: DATABASE_URL remoteRef: key: g0s-auth-api-ro.core.credentials.postgresql.acid.zalan.do property: password # then template: postgres://:{{ .password }}@core-pooler...:5432/ax_prod?sslmode=require ``` To see the exact list your install needs, read it out of the values file rather than from this page: ```bash grep -oE 'secretRef:\s*\{?\s*name:\s*[a-z0-9-]+' \ $B/values/gen0sec-platform-values-onprem.yaml | sort -u ``` Or inspect your live installation inside your cluster. ## Rotation | Credential | How | |---|---| | Database role passwords | Rotate through the Postgres operator, then restart the consumers | | Object store keys | Update **both** `gen0sec-object-store` and every `g0s-db-` that carries them, then restart the consumers | | Registry pull credentials | Recreate `gen0sec-registry` in both namespaces. Existing pods keep running; new pulls use the new value | | Your Gen0Sec API key | Recreate `gen0sec-download-proxy`, then restart the relay | | Dashboard session key | Rotating invalidates every session and every outstanding invite link. Passwords and passkeys are unaffected | | SMTP credentials | Update the Secret, then `kubectl -n gen0sec rollout restart deploy/g0s-ui` | :::note[One credential pair, surfaced two ways] The object store keys appear as `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` to the object store itself, and as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to the services that read it. Same values, two naming conventions. Rotating means updating both places. ::: ## Verifying Nothing should be missing before you install the platform: ```bash $KIT/scripts/preflight.sh ``` After install, confirm the count: ```bash kubectl -n gen0sec get secret | wc -l # ~20 service secrets plus the ones you created kubectl -n gen0sec-system get secret | grep credentials | wc -l # expect 6 ``` ## What is never in a secret | | Where it is instead | |---|---| | The registry host and namespace | The values file | | The relay upstream URL | The values file | | The cache connection string | The values file, deliberately with no credential in it | If you enable authentication on the cache, remove that line from the values file and put the connection string in each service's secret instead. --- # Values reference Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/configure/values # Values reference :::note[This page is the reference] It covers the keys you actually set, which is a much shorter list than every key a chart accepts, and states which of them are required. For anything not listed here, read the chart's own `values.yaml` in the bundle. There is no generated key-by-key reference. The charts carry no JSON schema yet, so nothing can be generated from one; when that changes this page will link to it. ::: ## How values are layered Three files, applied in order, last one wins. ``` chart values.yaml the chart's own defaults, registry deliberately blank values-onprem.yaml shipped in the bundle, points everything at a registry your overrides what you set ``` **Always pass `-f values-onprem.yaml`.** The default `values.yaml` leaves the registry blank on purpose, so installing without it cannot resolve images. Put your overrides in a file rather than in `--set` arguments, because `helm upgrade` uses only what you pass it. Anything set with `--set` last time and omitted this time reverts to the chart default. ```bash helm get values g0s -n gen0sec # what the running release actually has ``` ## gen0sec-infra Top-level keys: `postgres-operator`, `strimzi-kafka-operator`, `synapse-operator`, `synapse`, `rustfs`, `dragonfly`, `components`. | Key | Default | Set it when | |---|---|---| | `rustfs.storageclass.name` | `""`, meaning the cluster default | You want a specific StorageClass | | `rustfs.storageclass.dataStorageSize` | `50Gi` | Sizing the object store. **Four drives per replica** | | `rustfs.storageclass.logStorageSize` | `10Gi` | Rarely | | `rustfs.replicaCount` | `3` | Fewer nodes. Erasure coding assumes they spread | | `rustfs.clusterDomain` | `cluster.local` | Your cluster uses a custom DNS domain | | `dragonfly.extraArgs` | `[]` | Adding `--force_epoll` when `io_uring` is blocked | | `dragonfly.maxMemory` | Chart default | Keep it comfortably under the memory limit, or the kubelet kills the pod before it can evict | | `components.ui.namespace` | `gen0sec` | You install the platform chart in another namespace. Names the dashboard's service account for the Components tab | | `components.ui.serviceAccountName` | `g0s-ui` | You install the platform chart under a release name other than `g0s`. Must be `-ui` | See [Sizing storage for your retention](./../02-requirements.mdx#sizing-storage-for-your-retention). ## gen0sec-data Top-level keys: `postgres`, `kafka`, `kafka2pg`. | Key | Default | Set it when | |---|---|---| | `postgres.numberOfInstances` | `3` | Never below 3 in production | | `postgres.volume.size` | `50Gi` | Sizing the database. This is the system of record, so it is the one to grow first | | `postgres.volume.storageClass` | `""` | You want a specific StorageClass. **Fast block storage** | | `postgres.resources` | 1 vCPU / 2 GiB requested, 8 vCPU / 12 GiB limit | Tuning. The limit sets the per-node minimum | | `kafka.replicas` | `3` | Fewer nodes. **Drop the replication factors together**, or brokers refuse writes | | `kafka.storage.size` | `50Gi` | Sizing the message log and the ingest buffer | | `kafka.storage.class` | `""` | You want a specific StorageClass | :::danger[Kafka replicas and replication factor move together] `min.insync.replicas` must stay below `replicas`. Reducing brokers without reducing the factors gives you a cluster that schedules and then refuses every write. ::: ## gen0sec-platform Top-level keys: `global`, `migration`, `downloadProxy`, `services`. ### global | Key | Default | Notes | |---|---|---| | `global.registry.host` | `registry.gen0sec.com` | Rewritten by the bundler on the offline path | | `global.registry.namespace` | `gen0sec` | Every image lives under this | | `global.imageTag` | Empty | **Required.** Set it to the release version | | `global.imagePullSecrets` | `gen0sec-registry` | Must exist in both namespaces | | `global.commonEnv.S3_ENDPOINT` | The object store service URL | Must match the object store Service, which has a `-svc` suffix | | `global.commonEnv.ARXIGNIS_DATA_URL` | The same URL plus `/platform-data` | **The bucket name must match the bucket you created** | | `global.commonEnv.REDIS_URL` | The cache service URL | No credential in it, deliberately | | `global.podSecurityContext` | Non-root, uid 65532 | Do not weaken. See [Hardening](./../security/hardening.mdx) | | `global.securityContext` | Read-only root, no capabilities | Do not weaken | | `global.ingress.className` | `synapse` | The ingress controller the infra chart installs | | `global.commonEnv.CACHE_L1_TTL` | Unset, so `30s` | Default lifetime of an in-process cache entry, as a duration such as `45s`. An invalid value is ignored | ### Publishing `entrypointHost` is passed from the first install, and `externalScheme` by the [single-entrypoint overlay](./../06-publish.mdx). `values-onprem.yaml` sets neither and leaves every ingress disabled. Both are top level, and the chart applies both to every rule the overlay writes, which is why they are listed here rather than under `services`. | Key | Default | Notes | |---|---|---| | `entrypointHost` | `""` | **Required from the first install**, published or not. The one address that serves the dashboard and every agent-facing API, and the source of the dashboard's sign-in URL even before its Ingress exists. Empty fails the render and names the value to set, unless the dashboard is disabled or reached without an Ingress (see below). It must also be the address people actually type: the dashboard derives its sign-in URL from it | | `externalScheme` | Unset | Set to `https` **only when TLS terminates upstream** of the Ingress — a cloud load balancer, a CDN, a tunnel. With no `tls:` block to read, the chart otherwise infers `http` while the browser is on `https`, and every sign-in is rejected with `Invalid origin`. Leave it unset when TLS terminates at the Ingress | ### downloadProxy | Key | Default | Notes | |---|---|---| | `downloadProxy.enabled` | `true` | Turning it off is not supported | | `downloadProxy.apiKeySecret.name` | `""` | **Required. Empty is a silently broken deployment** | Full detail on [Data relay](./relay.mdx). This is the single most consequential key in the chart. ### services Twenty-one entries. Each one accepts the same shape, and you rarely need to touch any of it. | Key pattern | Notes | |---|---| | `services..enabled` | `threat-api` is not shipped at this release | | `services..replicaCount` | 2 or 3 in production, 1 in the quickstart overlay | | `services..resources` | Requests and limits. See [Sizing](./../02-requirements.mdx#sizing-a-production-install) | | `services..env` | Per-service environment, including `PORT` | | `services..ingress` | Off by default. Set by the entrypoint overlay | | `services..volumes`, `.volumeMounts` | **Do not remove these.** They exist because the read-only root filesystem needs a writable path where each service expects one | | `services.ui.enabled` | `true`. The dashboard | | `services.ui.env.BETTER_AUTH_URL` | Derived. Set it only when the dashboard is reached without an Ingress, such as through `kubectl port-forward` or your own proxy. It then wins over `entrypointHost` | | `services.ui.serviceAccount.automount` | `true`. The dashboard reads the installed-versions ConfigMaps for the Components tab. Every other service keeps `false` | | `services.ui.entraId.enabled` | `false`. Entra ID is in development and not supported at this release. See [Sign-in and identity](./identity.mdx) | :::danger[Removing a service's volume mount breaks that service] Several services default to writing into a path relative to their working directory, which fails on a read-only root filesystem. Each one has an explicit writable volume mounted where it writes. Those mounts are load-bearing, not leftovers. ::: ## Checking what a change will do Always render before you apply. ```bash helm template g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f my-overrides.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ | less ``` To diff against what is running: ```bash helm get manifest g0s -n gen0sec > running.yaml helm template g0s ... > proposed.yaml diff running.yaml proposed.yaml ``` ## Named starting points Two tested configurations rather than a blank page: [Reference configurations](./reference-configs.mdx). --- # First sign-in Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/first-signin ## Why this page exists Registration is invite-only, and invites can only be created by an existing member. A fresh install can be logged in by a pre-seeded admin account. ## The seeded account | | | |---|---| | Email | `admin@gen0sec.local` | | Password | `Gen0Sec!ChangeMe` | :::danger[This password is published, and it is the same on every install] It is a constant in our migration file, so anyone who has read our documentation knows it. The dashboard will not let you past it: the account is created with a must-change flag, so setting a new password is a step you cannot skip or postpone. What that does not do is protect the account between the moment you publish the address and the moment you first sign in. **Whoever signs in first is who sets the new password.** Sign in before you give the address to anyone else. If you have already shared it, sign in now and treat the interval as exposed. ::: ## 1. Sign in Open your published address and sign in with the details above. ```bash echo https://$ENTRYPOINT_HOST ``` If sign-in is rejected as an invalid origin, the published host or scheme does not match what you typed. See [Sign-in and identity](./configure/identity.mdx#the-published-address-decides-whether-sign-in-works). ## 2. Complete first-run setup The dashboard holds you at each of these in turn, and it will not let you skip ahead. A deep link to any other page returns you to the step you are on. 1. **Enroll multi-factor authentication.** Nothing else opens until you confirm a six-digit code. 2. **Set a new password.** Step 3 below. 3. **Create your organization.** 4. **Finish the remaining onboarding steps.** The wizard creates your first API key, then walks the WAF, access rule and monitoring surfaces. Multi-factor comes first, so the published password stops being enough to sign in before you have even reached the password screen. ## 3. Mandatory password change Because the password is preseeded, after setting up MFA, you'll be asked to change your password. :::note[The change signs out this account everywhere else] Every other session on the seeded account ends when the password changes. On a first install there are none, which is the point: if the address was reachable before you signed in, this is what closes that off. ::: The seed is also idempotent on the email address. Re-running the migration or the secret script neither duplicates the account nor re-flags it, and it does not reset a password you have already changed. ## 4. Invite the people who need access Invite your operators rather than sharing the seeded account. Under **Organization settings**, **Team**, pick the workspaces each person gets and a role for each. If [SMTP is configured](./configure/identity.mdx#invitations-and-email) the invitation is emailed. Otherwise the dashboard shows the link: copy it and send it yourself. Links expire after seven days. Invitees register with a passkey or a password, and enroll multi-factor like you did. ## 5. Mint an agent API key Create an agent API key in the dashboard, then save it where the verification step reads it from: ```bash mkdir -p ~/.gen0sec printf '%s' 'yor-agent-key' > ~/.gen0sec/agent-key ``` ## If it fails | Symptom | Cause | |---|---| | The seeded account does not exist | The migration ran without the on-prem marker. Secrets were provisioned without `make-db-secrets.sh` | | Sign-in rejected as an invalid origin | The published host or scheme does not match the address you typed | | The dashboard loads but onboarding will not advance | Check `kubectl -n gen0sec logs deploy/g0s-ui` | | The password is rejected | It is case-sensitive and contains a `!`. Watch your shell if you pasted it | | The new password is rejected as too weak | 12 characters, one symbol, and three of the four character classes. The server applies the same rules as the form | | The password screen appears again on a later sign-in | The change did not complete. The flag clears only on success | | An invitation link says it is no longer valid | It was revoked, replaced by a newer invitation, already used, expired, or issued before an upgrade from v0.1.x | --- # Install Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/install # Install ## Before you start - [ ] [Prepare the artifacts](./04-prepare-artifacts.mdx) complete - [ ] `$KIT`, `$B`, `$REGISTRY`, `$REGISTRY_USER`, `$REGISTRY_TOKEN` are set in your shell - [ ] `$G0S_DOWNLOAD_API_KEY` is set. Step 2 writes it into a secret, and an unset variable produces an empty one that fails silently - [ ] You are logged in to the registry - [ ] The address the dashboard will be reached on, in `$ENTRYPOINT_HOST`. Step 7 needs it, and [Publish the endpoints](./06-publish.mdx) publishes under it ```bash export ENTRYPOINT_HOST=cerebellum.example.internal ``` ## 1. Check the cluster ```bash $KIT/scripts/preflight.sh ``` This checks the prerequisites the charts assume but do not create: a reachable cluster, a StorageClass, and the namespaces and secrets created in step 2. On a fresh cluster it reports the namespaces and secrets as missing, which is expected at this point. Pass a StorageClass name to pin it instead of accepting the default: ```bash $KIT/scripts/preflight.sh gen0sec-system gen0sec my-block-storage ``` ## 2. Create the namespaces and the two bootstrap secrets ```bash for ns in gen0sec-system gen0sec; do kubectl create namespace $ns --dry-run=client -o yaml | kubectl apply -f - done ``` Written to converge, so re-running step 2 after a failure further down is safe rather than a wall of `AlreadyExists`. ### The image pull secret Both namespaces need it. The application runs in `gen0sec`; the operators, object store and cache run in `gen0sec-system`. ```bash for ns in gen0sec-system gen0sec; do kubectl -n $ns create secret docker-registry gen0sec-registry \ --docker-server=$REGISTRY \ --docker-username=$REGISTRY_USER \ --docker-password=$REGISTRY_TOKEN \ --dry-run=client -o yaml | kubectl apply -f - done ``` ### The object store credentials :::danger[Run this once, and only once] Unlike the rest of step 2 this does **not** converge. It mints new random keys every time it runs, and the object store's existing data was written with the old ones — recreating the secret on a second pass strands that data behind credentials nothing holds any more. The guard below stops before that happens. ```bash kubectl -n gen0sec-system get secret gen0sec-object-store >/dev/null 2>&1 \ && echo "already exists — skip to step 3" \ || echo "not present — run the block below" ``` ::: ```bash umask 077 S=$(mktemp -d) openssl rand -hex 16 | tr -d '\n' > "$S/RUSTFS_ACCESS_KEY" openssl rand -hex 32 | tr -d '\n' > "$S/RUSTFS_SECRET_KEY" kubectl -n gen0sec-system create secret generic gen0sec-object-store --from-file="$S" ``` **Save both files somewhere safe before you delete them.** Nothing regenerates these keys, and the data in the object store is written with them. ```bash mkdir -p ~/gen0sec-object-store-keys cp "$S"/RUSTFS_* ~/gen0sec-object-store-keys/ # or load them into your secret manager rm -rf "$S" ``` Step 5 reads them back from that directory. ### Your Gen0Sec API key The relay presents this key to Gen0Sec when it fetches threat intelligence, GeoIP, models and IDS rules. The chart expects this exact secret name, so the relay pod will not start without it. See [Credentials from gen0sec](./02-requirements.mdx#your-gen0sec-api-key). ```bash # Refuses to run rather than creating an empty secret. An empty key is accepted # by Kubernetes and by the chart, and only surfaces later as every relay fetch # returning 401 across the whole fleet — with nothing in the install log. : "${G0S_DOWNLOAD_API_KEY:?not set — see Prepare the artifacts, step 0}" kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-literal=DOWNLOAD_PROXY_API_KEY=$G0S_DOWNLOAD_API_KEY ``` ## 3. Install infra ```bash helm install g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml ``` This installs three operators (Postgres, Kafka, ingress), the object store and the shared cache. ### Wait for all five The next step creates resources these operators have to reconcile. Do not continue until every one is available. ```bash kubectl -n gen0sec-system rollout status deploy/g0s-infra-postgres-operator kubectl -n gen0sec-system rollout status deploy/strimzi-cluster-operator kubectl -n gen0sec-system rollout status deploy/g0s-infra-synapse-operator kubectl -n gen0sec-system rollout status deploy/dragonfly kubectl -n gen0sec-system rollout status \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" ``` The output for each is similar to this: ``` deployment "g0s-infra-postgres-operator" successfully rolled out ``` :::note[Why the last one resolves its own kind] The object store is a StatefulSet in the production configuration and a Deployment in the single-node configuration. Resolving the kind means the same command works in both. ::: ## 4. Install data ```bash helm install g0s-data $B/charts/gen0sec-data-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml ``` This creates the Postgres cluster, the Kafka cluster, its topics, and the database roles. ### Wait for both clusters Both take several minutes. Nothing after this works until Postgres reports `Running`. ```bash kubectl -n gen0sec-system wait --for=condition=Ready kafka/core --timeout=15m kubectl -n gen0sec-system wait \ --for=jsonpath='{.status.PostgresClusterStatus}'=Running postgresql/core --timeout=15m ``` ## 5. Create the object store bucket The object store starts empty and nothing creates the bucket for you. Create it now, before the platform install. Services that read or write the object store fail until it exists. This step is required on every install. The two datasets your cluster produces about itself, `identity` and `policy-edges`, live here and are never sent anywhere. See [Network and connectivity](./03-network.mdx#what-never-leaves-your-cluster). Run it as a throwaway pod in the cluster. The pod reads the keys straight from the `gen0sec-object-store` secret, so they never reach your shell and never touch your local AWS configuration. ```bash kubectl -n gen0sec-system run s3-init --rm -i --restart=Never \ --image=amazon/aws-cli:2.15.0 \ --overrides='{"spec":{"containers":[{"name":"s3-init","image":"amazon/aws-cli:2.15.0", "env":[{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"name":"gen0sec-object-store","key":"RUSTFS_ACCESS_KEY"}}}, {"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"name":"gen0sec-object-store","key":"RUSTFS_SECRET_KEY"}}}, {"name":"AWS_EC2_METADATA_DISABLED","value":"true"}, {"name":"AWS_REGION","value":"us-east-1"}], "args":["--endpoint-url","http://rustfs-svc.gen0sec-system.svc.cluster.local:9000","s3","mb","s3://platform-data"]}]}}' ``` Expect `make_bucket: platform-data`. `BucketAlreadyOwnedByYou` is also a pass, so this step converges: running it twice is harmless. :::note[Offline installs: mirror this one image first] `amazon/aws-cli` is **not** in the offline bundle. The bundle carries only what `release-manifest.yaml` pins, and its `SHA256SUMS` covers the whole tree — the bundler refuses a package containing anything the checksums do not list — so it cannot be added after the fact. Mirror it alongside the bundle, then point the pod at your copy: ```bash skopeo copy docker://amazon/aws-cli:2.15.0 docker://$REGISTRY/amazon/aws-cli:2.15.0 ``` Substitute `$REGISTRY/amazon/aws-cli:2.15.0` for `amazon/aws-cli:2.15.0` in both the `--image` flag and the `image` field of the overrides above. They must match: the `--image` flag sets the pod's image, and the override object sets the container's, and Kubernetes uses the latter. ::: Verify by running the same command with `"s3","ls"` in place of `"s3","mb","s3://platform-data"`. The bucket should be listed: ``` 2026-09-01 14:02:11 platform-data ``` An empty result means the bucket was not created, and the services in step 7 will fail with `500 NoSuchBucket`. Do not continue. :::note[If you changed the bucket name] `platform-data` is the value of `global.commonEnv.S3_BUCKET`. If you overrode it, use your name instead. ::: :::caution[Do not export the object-store keys into your shell] The in-cluster method above is recommended because the credentials never enter your shell. That matters most on managed Kubernetes. **On EKS, exporting them breaks `kubectl`.** The kubeconfig authenticates through an `aws eks get-token` exec plugin that reads `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` — the same two variables. Setting them to the object-store keys makes every later command in that shell fail with: ``` error: You must be logged in to the server (Unauthorized) ``` The error names kubectl, not the variables, which makes it hard to attribute. The in-cluster method avoids it entirely. If you use the workstation method below, scope the credentials to the one command with `env ...` rather than exporting them. **If the object store returns `401`,** the keys are not the ones this install generated. Keys left over from a previous install of the same cluster do this, and so does a stale copy in `~/gen0sec-object-store-keys/`. Credentials are genuinely validated: wrong keys return `InvalidAccessKeyId`, and none at all returns `403`. :::
Workstation method, for reference This needs no S3 client image in the cluster, which is why it was preferred: on the offline bundle method an in-cluster image has to be mirrored first, so the bucket comes to depend on a step that exists only to create the bucket. Use it if that trade-off suits you — but read the caution above first, because exporting the object-store keys will break `kubectl` on EKS. ### 5a. Open a port forward In a second terminal: ```bash kubectl -n gen0sec-system port-forward svc/rustfs-svc 9000:9000 ``` Leave it running. The output is similar to this: ``` Forwarding from 127.0.0.1:9000 -> 9000 ``` ### 5b. Create the bucket Back in your first terminal, use the two keys you generated in step 2. Scope them to the one command with `env` rather than exporting them, so they cannot clobber the variables your kubeconfig's exec plugin reads: ```bash env AWS_ACCESS_KEY_ID="$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ AWS_SECRET_ACCESS_KEY="$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" \ AWS_REGION=us-east-1 AWS_EC2_METADATA_DISABLED=true \ aws --endpoint-url http://localhost:9000 s3 mb s3://platform-data ``` The output is similar to this: ``` make_bucket: platform-data ``` `BucketAlreadyOwnedByYou` is also a pass. With the MinIO client instead: ```bash mc alias set g0s http://localhost:9000 \ "$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ "$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" mc mb g0s/platform-data mc ls g0s ``` ### 5c. Verify it exists ```bash env AWS_ACCESS_KEY_ID="$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ AWS_SECRET_ACCESS_KEY="$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" \ AWS_REGION=us-east-1 AWS_EC2_METADATA_DISABLED=true \ aws --endpoint-url http://localhost:9000 s3 ls ``` The output is similar to this: ``` 2026-09-01 14:02:11 platform-data ``` An empty result means the bucket was not created, and the services in step 7 will fail with `500 NoSuchBucket`. Do not continue. Then stop the port forward with `Ctrl-C` in the second terminal.
## 6. Create the application secrets ```bash $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml ``` This composes around twenty secrets from the passwords the Postgres operator generated in step 4. That is why it cannot run earlier. Each service gets its own database role through the connection pooler. The migration job gets a superuser connection direct to the primary, because it runs `CREATE EXTENSION`. :::note[In production, treat this script's output as a specification] `make-db-secrets.sh` is fine for evaluation. For production, have your secret manager produce secrets of the same shape. Every name and key is listed in [Secrets](./configure/secrets.mdx). ::: ## 7. Install the platform ```bash helm install g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` **`entrypointHost` is required here**, before anything is published. The dashboard builds its sign-in URL from it, and without it the render fails with `entrypointHost is required`. Use the address people will type, the one you publish under in the next step. :::note[Nothing to set for the relay] The chart already points at `gen0sec-download-proxy` by default. You created that secret in step 2, and if it is missing the relay pod fails to start with `CreateContainerConfigError` rather than coming up half-configured. ::: Schema migrations run first, as a pre-install hook. If they fail, Helm aborts and nothing is deployed. That is deliberate: a partially migrated database is worse than no install. ### Check the migration ```bash kubectl -n gen0sec get job g0s-db-migrate ``` The output is similar to this: ``` Error from server (NotFound): jobs.batch "g0s-db-migrate" not found ``` **`NotFound` is the success case.** The job is a Helm hook with `hook-delete-policy: hook-succeeded`, so Helm deletes it the moment it passes. If the job is still there, it may have failed: ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` ### Wait for the pods ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m ``` :::note[Check readiness, not phase] A pod in `CrashLoopBackOff` still reports `status.phase: Running`. A `--field-selector` on phase reports a broken install as clean. ::: :::note[Two values that must match your cluster] Both are set in `values-onprem.yaml` and both point at the in-cluster object store. Check them if anything cannot read its data. | Key | Must be | |---|---| | `global.commonEnv.S3_ENDPOINT` | `http://rustfs-svc.gen0sec-system.svc.cluster.local:9000` | | `global.commonEnv.ARXIGNIS_DATA_URL` | The same endpoint, plus the bucket: `.../platform-data` | `ARXIGNIS_DATA_URL` is where in-cluster services read their datasets from. It points at your object store, not at Gen0Sec. **The bucket name at the end of it must match the bucket you created in step 5.** If you used a different name there, change it here too. The Service is `rustfs-svc`, with the `-svc` suffix. That suffix is the single most common cause of services that come up and then cannot read anything. ::: ## If it fails | Symptom | Start here | |---|---| | `helm install` fails with `entrypointHost is required` | `entrypointHost` was not passed in step 7. Set it and re-run | | `ImagePullBackOff` | [ImagePullBackOff](./troubleshoot/symptoms.mdx#imagepullbackoff) | | Install aborts at `g0s-db-migrate` | [Migration failures](./troubleshoot/symptoms.mdx#migration-failures) | | `postgresql/core` never reaches `Running` | [Postgres will not start](./troubleshoot/symptoms.mdx#postgres-will-not-start) | | Kafka broker crash loops | [Kafka will not start](./troubleshoot/symptoms.mdx#kafka-will-not-start) | --- # Network and connectivity Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/network # Network and connectivity ## Why the platform must reach Gen0Sec No platform is useful without data, neither is Gen0Sec. Because the datasets are licensed per customer, they are not in the install kit, not in the offline bundle, and not in any container image. The platform fetches them at runtime from `api.gen0sec.com`. | Dataset | Licensed by | |---|---| | CTI indicators | Upstream threat intelligence providers | | GeoIP databases | MaxMind, IPInfo | | IDS rules | Emerging Threats ruleset | | ML models | Trained and published by Gen0Sec | One in-cluster service, `download-api`, does the fetching. It relays each request upstream and streams the response straight back. It is a relay, not a cache, so there is no artifact bucket to size, no sync job to supervise and no staleness window. An agent that polls gets whatever upstream has at that moment. This also means you get the most recent data we can provide. ## One hop, not many Your agents never talk to the internet. They talk to `download-api`, and `download-api` is the only service in either namespace that needs a route out. The dashboard makes optional calls of its own, listed in the [egress list](#egress-list); none of them carries agent data. ``` agents -> download-api -> api.gen0sec.com:443 ^ | the only egress point in the deployment ``` That means one destination to allow, one to audit. It also means the relayed URLs are identical to the direct ones, so agents need no configuration change. ## What never leaves your cluster Two datasets describe your own infrastructure, and they are never relayed: | Dataset | What it contains | Where it stays | |---|---|---| | `identity` | Workload identity produced in your cluster | Your object store | | `policy-edges` | Pod IPs, namespaces, NetworkPolicy edges | Your object store | No hidden switches, no configuration options, it is not implemented. Your data is yours and yours only. Uploads are not relayed either. The relay is read-only, so `PUT` and `POST` on the download surface are served locally or not at all. ## How authorization works **Your agents' API keys never leave your cluster.** Agents authenticate to your installation with keys your own dashboard issued, held in your own database. On a correct configuration the relay does not forward those keys upstream. It presents **your own Gen0Sec API key**, the one issued with your subscription, on the second hop only. ``` agent --[your agent's key]--> relay --[your Gen0Sec API key]--> api.gen0sec.com agent key stops here ``` | | | |---|---| | What Gen0Sec sees | One credential, belonging to your installation | | What Gen0Sec never sees | Any individual agent key, or how many you have issued | | Revoking an agent | Entirely local, in your dashboard | | Revoking your installation's access | Revoke or rotate your Gen0Sec API key | :::danger[This key is required, and omitting it fails silently] With no Gen0Sec API key configured, the relay falls back to forwarding the caller's own key. That is correct for the hosted product, where the same platform issued it. Here it is always wrong: your keys exist only in your cluster, so Gen0Sec rejects every one with a 401, and **the agent reports an empty ruleset rather than an error**. The install may look healthy, but the fleet quietly stops receiving threat intelligence, GeoIP, models and IDS rules. Set up in [Install](./05-install.mdx) step 2, enabled in step 7, and checked by [Verify the install](./08-verify.mdx). ::: Routes that are not relayed, including `identity` and `policy-edges`, keep the local authorization check against your own database. ## Egress list All connections are outbound, TCP, TLS. Nothing needs an inbound rule from the internet. | Destination | Port | Needed by | Purpose | Required | |---|---|---|---|---| | `api.gen0sec.com` | 443 | Cluster pods (`download-api` only) | Threat intelligence, GeoIP, ML models, IDS rules | **Always** | | `releases.gen0sec.com` | 443 | Your workstation | Download the install kit and its signature | **Always** | | `registry.gen0sec.com` | 443 | Your workstation | `helm pull` for charts, `skopeo copy` for images | **Always** | | `registry.gen0sec.com` | 443 | Cluster nodes (kubelet) | Image pulls at install time | Registry install only | | `fulcio.sigstore.dev` | 443 | Your workstation | `cosign` certificate verification | **Always** | | `rekor.sigstore.dev` | 443 | Your workstation | `cosign` transparency log verification | **Always** | | `tuf-repo-cdn.sigstore.dev` | 443 | Your workstation | `cosign` trust root refresh | **Always** | | Your own registry | 443 or 80 | Cluster nodes, and the host that runs `bundler.sh apply` | Image pulls at install time | Offline bundle only | | `helm.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest agent version, shown on the Clients and Components pages | No. Always attempted. Blocking it only hides the latest version | | `releases.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest component versions | No. Only if an administrator turns on **Check for updates online** | | Your SMTP server | 587 or 465 | Cluster pods, the dashboard only | Invitation and password-reset email | No. Only if you [configure SMTP](./configure/identity.mdx#invitations-and-email) | {/* ASSUMPTION: this list is derived from the release workflow, the bundler and the chart values. Confirm it is complete, and confirm the Sigstore host set, which changes with cosign versions. */} :::note[The Sigstore hosts are the ones people forget] `cosign verify-blob` checks a signature against a public transparency log. Without those three destinations it fails, and it fails on the first command of the install. Allow them on the workstation, not on the cluster. ::: ### Inside the cluster No NetworkPolicy is required for the install. If you enforce a default-deny policy, the two namespaces need to reach each other: the application in `gen0sec` talks to Postgres, Kafka, the object store and the cache in `gen0sec-system`. ## What this rules out `api.gen0sec.com` on port 443 is required for the platform to serve data. An agent fleet with the platform hosted on a cluster that is permitted no outbound connections at all cannot function correctly. The offline bundle method removes every dependency on gen0sec-hosted registries: your cluster pulls images only from your own. It does not remove the runtime connection, because the mentioned datasets above are not ours to ship. If your policy allows no cluster egress, talk to us. We would rather hear it now than during your evaluation. > **Air-gapped.** Often used to mean "no connection of any kind". Under that definition Cerebellum > is not deployable. If it means "no dependency on vendor-hosted registries and one audited > outbound destination", the offline bundle method fits, and this page is the list your firewall team > needs. --- # Back up and restore Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/operate/backup # Back up and restore ## What to back up Not everything in the deployment is equally valuable. Backing up the wrong things is how backup jobs become expensive and untested. | Component | Back up | If you lose it | |---|---|---| | **Postgres** | **Yes, always** | Everything. Configuration, agents, policy, history, the audit log. This is the system of record | | **Object store** | **Yes** | The datasets and the artifacts your cluster produced about itself. Datasets are re-fetchable, `identity` and `policy-edges` are not | | Kafka | No | In-flight messages only. It is a transport, not a store. Producers re-send | | Shared cache | No | Nothing. It repopulates | | Secrets | **Yes** | Access to your own volumes. See the warning below | :::danger[Back up the volumes along with the secrets, or the volumes are useless] The database passwords are generated by the Postgres operator and stored as Kubernetes secrets. Data is written with those credentials. A volume backup without the matching secret backup restores data you cannot open. ::: ## Postgres The Postgres operator ships a logical backup image, already mirrored as part of the release. Enable it in the infra values rather than scripting `pg_dump` yourself. ```yaml # gen0sec-infra values postgres-operator: configLogicalBackup: logical_backup_schedule: "0 2 * * *" logical_backup_provider: s3 ``` Verify a backup ran: ```bash kubectl -n gen0sec-system get cronjob | grep logical-backup kubectl -n gen0sec-system get job | grep logical-backup ``` :::note[A backup you have not restored is not a backup] Restore into a scratch namespace once, before you need to. The [Quickstart](./../quickstart.mdx) cluster is a reasonable place to rehearse it, and rehearsing a restore is also a rehearsal of an upgrade rollback. ::: ## Object store Two prefixes matter, and they matter differently. | Prefix | Recoverable without a backup? | |---|---| | `identity/`, `policy-edges/` | **No.** Produced by your cluster about your cluster | | Dataset prefixes | Yes. They are fetched from Gen0Sec | Back up the whole bucket with any S3-compatible tool, with an in-cluster job or manually, over a port forward as in [Install step 5](./../05-install.mdx#5-create-the-object-store-bucket): ```bash kubectl -n gen0sec-system port-forward svc/rustfs-svc 9000:9000 ``` ```bash aws --endpoint-url http://localhost:9000 s3 sync s3://platform-data ./platform-data-backup ``` Restore is the same command with the arguments reversed. ## Secrets ```bash kubectl -n gen0sec-system get secret -o yaml > secrets-gen0sec-system.yaml kubectl -n gen0sec get secret -o yaml > secrets-gen0sec.yaml ``` **These files contain live credentials in base64.** Treat them exactly as you treat the database itself: encrypted at rest, access controlled, and out of any shared drive. If you run an external secret manager, you do not need this: your manager is already the source of truth and the cluster holds copies. See [Secrets](./../configure/secrets.mdx). ## Recovery point and recovery time | | Value | Determined by | |---|---|---| | **Recovery point** | Your backup interval | The logical backup schedule. Nightly gives up to 24 hours of loss | | **Recovery time** | Measure it | Database size and your storage throughput. Rehearse it and record your own figure | We do not publish an RTO as it depends entirely on your configuration specifics and the data size. Time your own rehearsal and use that number in your own runbook. ## Restoring Order matters, and it is the install order. 1. Install infra and data as on a [first install](./../05-install.mdx), so the operators and empty clusters exist. 2. Restore the secrets, before anything tries to use the volumes. 3. Restore Postgres from the logical backup. 4. Restore the object store bucket. 5. Install the platform. **The migration hook runs against restored data**, so the restored schema must match the release you are installing. Restore into the version it came from, then upgrade. 6. Run the smoke test: `$KIT/scripts/verify-deployment.sh`. :::note[Do not restore an old database into a new release] The migration chain expects to run forward from the schema version in the dump. Restoring a v0.1 backup and installing v0.3 skips the intermediate migrations. Restore v0.1, then upgrade one minor at a time. See [Upgrade](./upgrade.mdx). ::: --- # Monitor Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/operate/monitor # Monitor ## What the platform exposes today | Signal | Where | Notes | |---|---|---| | Health and readiness | `/status/healthz` and `/status/readyz` on each service | Three services serve only `healthz`. See [Known limitations](./../troubleshoot/known-limitations.mdx#three-services-report-readiness-from-a-weaker-probe) | | Relay mode and upstream | `/status/healthz` on the relay | Returns the mode, the upstream and the active relay classes | | Prometheus metrics | Worker port 8080 | Exposed, but no Service is created for workers, so scraping needs a direct pod target | | Kubernetes state | Pods, Deployments, custom resources | The primary signal | | Block events refused by the database | Kafka topic `block_events_dlq`, 14 days retention | Each message carries the reason in its `dlq_reason` header. Steady growth means events are being lost | | Block events dropped at ingest | telemetry-api log line `OTLP block events rejected, or stored without a field they carried` | At most once a minute per organization, with counts per reason | | Installed and latest versions | Dashboard, **Organization settings**, **Components** | See [Component versions](./../configure/components.mdx#verifying-what-you-are-actually-running) | | Who changed what | Dashboard, **Organization settings**, **Audit Log** | Owner and Admin roles only | **No ServiceMonitor, alert rules or dashboards ship with the charts at the moment.** If you run Prometheus, you build the scrape configuration. ## The four checks worth alerting on If you set up nothing else, set up these. ### 1. Pods not ready ```bash kubectl -n gen0sec get pods --no-headers \ | awk '{split($2,r,"/"); if (r[1] != r[2]) print}' ``` Any output is a problem. **Read the ready column, never a field selector on phase:** a pod in `CrashLoopBackOff` still reports `status.phase: Running`. ### 2. The database cluster is not Running ```bash kubectl -n gen0sec-system get postgresql core \ -o jsonpath='{.status.PostgresClusterStatus}' ``` Expect `Running`. Anything else and nothing downstream works. ### 3. The relay lost its upstream ```bash kubectl -n gen0sec exec deploy/g0s-download-api -- \ curl -fsS localhost:9999/status/healthz ``` Expect `"status": "ready"` and `"mode": "proxy"`. An unreachable upstream answers **502** on relayed routes rather than 404 or 500. That is deliberate: a polling agent reads 502 as "ask again later" and keeps the artifact it already holds, instead of treating it as withdrawn. So a broken upstream degrades your fleet slowly rather than immediately, which is safer and also easier to miss. Alert on it. ### 4. The platform stopped serving agents ```bash $KIT/scripts/verify-deployment.sh ``` Read-only, safe against production, exits non-zero and names the failing check. Running it on a schedule is the single most useful synthetic check available, because it is the only one that proves data actually moves. ## Signals that are noisy on purpose Do not page on these. | Signal | Why it is not an incident | |---|---| | `falling back to L1-only cache` | The shared cache is never fatal. Services run on their in-process cache. Reads get slower, nothing breaks | | A 502 from a relayed route, once | Upstream hiccup. Agents retry and keep what they have | | The migration job absent | That is success. Helm deletes it on completion | | `unknown_source` in a telemetry-api warning | An agent newer than the platform. The event is stored, without its source | ## Logs ```bash kubectl -n gen0sec logs deploy/ --tail=200 kubectl -n gen0sec-system logs deploy/g0s-infra-postgres-operator --tail=200 ``` All services log to standard output as structured JSON, so any collector that reads container stdout works without configuration. Log level is set per service through the values file. See [Values reference](./../configure/values.mdx). ## What a healthy install looks like Five statements, the same five as [Verify the install](./../08-verify.mdx#definition-of-installed-successfully). If you build a dashboard, build it to answer these: 1. Every pod in both namespaces is `Running` and fully ready. 2. `postgresql/core` reports `Running`. 3. `kafka/core` reports `Ready`. 4. The relay reports `"mode": "proxy"` with your upstream. 5. The smoke test exits zero. --- # Roll back Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/operate/rollback # Roll back Read the boundary section first. It decides whether a rollback is safe or whether you need a restore. ## The boundary, stated plainly **`helm rollback` reverts manifests. It does not revert your database.** That single sentence is the whole risk. Helm restores the previous Deployments, images and configuration. Any schema change the upgrade's migration hook applied is still applied. If the old application cannot read the new schema, rolling back the manifests gives you old code against a new database. | Situation | Rollback is | |---|---| | Patch upgrade within a train, no migration ran | Safe. Just do it | | Minor upgrade whose migrations only added things | Safe. Old code ignores what it does not know | | Minor upgrade whose migrations removed or renamed something the old code reads | **Not safe.** Restore from backup instead | | You are not sure which | Check the release notes. If they do not say, ask us before rolling back | :::note[v0.2.1 back to v0.2.0] Safe, and there is nothing to run: all 17 v0.2.1 down migrations are no-ops, and v0.2.0 works against the v0.2.1 schema. Both v0.2.1 fixes live in the database, so rolling back does not bring either defect back. While rolled back, do not `helm upgrade` the v0.2.0 chart: its migration job does not know the v0.2.1 schema version and fails with `no migration found for version`. ::: :::note[v0.2.0 back to v0.1.x] Rolling back the platform alone is safe: every v0.2.0 migration only adds, so v0.1.x runs against the v0.2.0 schema. **Do not run the down migrations.** There are 18, and they delete the audit log, registered passkeys and pending invitations. While rolled back, do not `helm upgrade` the v0.1.x chart: its migration job does not know the v0.2.0 schema version and fails with `no migration found for version`. ::: :::note[Why we prefer expand and contract] Migrations that add in one train and remove in a later one mean a one-release rollback never needs a schema change. That is the design intent, and it is what makes most rollbacks safe. It is not a guarantee for every release, which is why this page exists rather than a one-line reassurance. ::: ## Before you start - [ ] You know which revision you are going back to - [ ] You have read the release notes for the version you are leaving - [ ] The previous release's images are still in your registry **Offline bundle method:** mirroring is additive, so the old tags are still there unless you deleted them. Do not prune your registry between an upgrade and the point at which you are confident. ## 1. Find the revision ```bash helm history g0s -n gen0sec ``` The output is similar to this: ``` REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Mon Sep 1 10:14:02 2026 superseded gen0sec-platform-0.1.0 0.1.0 Install complete 2 Mon Sep 1 15:41:18 2026 deployed gen0sec-platform-0.2.0 0.2.0 Upgrade complete ``` ## 2. Roll back the platform Start here. In most cases this is the only step you need, because the application is what broke. ```bash helm rollback g0s 1 -n gen0sec ``` ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` If the smoke test passes, stop. Do not roll back infra or data unless they are the problem. ## 3. Roll back the schema, only if you must Needed when the upgrade applied a change the old code cannot work with. Check the release notes first. ```bash kubectl -n gen0sec run db-migrate-down --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/db-migrate: \ --env DATABASE_URL="$OWNER_DSN" -- down ``` Use the `db-migrate` image of the release you are **leaving**, not the one you are returning to. Only it contains the down migrations for its own schema; an older image fails with `no migration found for version `. `down` reverts one migration per run, so run it once per migration the release added. **Never run `down all`**: it reverts the entire schema. `$OWNER_DSN` is the superuser connection string from the `gen0sec-db-migrate` secret. See [Secrets](./../configure/secrets.mdx). :::danger[A down migration can lose data] Reversing a migration that dropped a column recreates the column, not its contents. If the upgrade removed data, only a restore brings it back. See [Back up and restore](./backup.mdx). The v0.2.0 down migrations drop the audit log, passkeys and pending invitations, and clear the `fleet_ban` source from stored block events. One of them also decompresses recent block events and locks that table while it runs. ::: ## 4. Roll back infra and data, only if they are the problem Same order as the upgrade, in reverse: platform, then data, then infra. ```bash helm rollback g0s-data 1 -n gen0sec-system helm rollback g0s-infra 1 -n gen0sec-system ``` :::note[CRDs do not roll back] Helm never manages CRDs after the first install, in either direction. A rolled-back operator runs against the newer CRD schema. That works when the change was additive, which is the usual case, and it is another reason to roll back the platform alone where possible. ::: ## When a restore is the right answer instead Rollback is for "the new version misbehaves". Restore is for "the data is wrong". | You want to | Use | |---|---| | Undo a bad release | Rollback | | Undo a destructive migration | Restore | | Recover from data corruption | Restore | | Return to a point in time | Restore | [Back up and restore](./backup.mdx). ## If it fails | Symptom | Cause | |---|---| | `ImagePullBackOff` after rollback | The previous version's images were pruned from your registry | | Old pods start, then error on database queries | Schema mismatch. Go to step 3, or restore | | `helm rollback` reports no such revision | `helm history` was truncated. `helm get manifest g0s --revision N` to confirm what exists | ## Next [Uninstall](./uninstall.mdx), or [Back up and restore](./backup.mdx). --- # Rotate credentials Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/operate/rotate # Rotate credentials Credentials rotates differently, and some of them have a consequence worth knowing before you start. | Credential | Rotating it interrupts | Reversible | |---|---|---| | [Database role passwords](#database-role-passwords) | The services that use them, briefly | Yes | | [Object store keys](#object-store-keys) | Everything that reads the object store | Yes, **if you kept the old values** | | [Registry pull credentials](#registry-pull-credentials) | Nothing running. New pulls only | Yes | | [Your Gen0Sec API key](#your-gen0sec-api-key) | Artifact delivery, briefly | Yes | | [Dashboard session signing key](#dashboard-session-signing-key) | **Every session and every invite link** | No | If you run an external secret manager, rotation happens there and the cluster picks up the new value. This page is for installs holding plain Kubernetes secrets. See [Secrets](./../configure/secrets.mdx). ## Database role passwords Rotate through the Postgres operator, not by editing the secret. The operator owns these values, and an edited secret is overwritten on its next reconciliation. After the operator has regenerated the credentials, recompose the application secrets and restart the consumers: ```bash $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy ``` The script preserves the dashboard's session signing key, so re-running it does not sign people out. ## Object store keys These are the ones you generated during install, and **nothing regenerates them**. Rotating means updating two places. ```bash umask 077 S=$(mktemp -d) openssl rand -hex 16 | tr -d '\n' > "$S/RUSTFS_ACCESS_KEY" openssl rand -hex 32 | tr -d '\n' > "$S/RUSTFS_SECRET_KEY" kubectl -n gen0sec-system delete secret gen0sec-object-store kubectl -n gen0sec-system create secret generic gen0sec-object-store --from-file="$S" ``` Then restart the object store, and recompose the application secrets, because the services receive the same values under different names: ```bash kubectl -n gen0sec-system rollout restart \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy ``` Save the new files where you kept the old ones, then remove the temporary directory. :::danger[Do not lose these] Data in the object store was written with them. A retained volume without the matching keys is data you cannot open. See [Back up and restore](./backup.mdx). ::: ## Registry pull credentials Recreate the secret in **both** namespaces. Running pods keep running; the new value applies to the next pull. ```bash umask 077 AUTH=$(printf '%s:%s' "$REG_USER" "$(cat $REG_PASS_FILE)" | base64 | tr -d '\n') printf '{"auths":{"%s":{"auth":"%s"}}}' "$REGISTRY" "$AUTH" > dockerconfig.json unset AUTH for ns in gen0sec-system gen0sec; do kubectl -n $ns delete secret gen0sec-registry kubectl -n $ns create secret generic gen0sec-registry \ --type=kubernetes.io/dockerconfigjson \ --from-file=.dockerconfigjson=dockerconfig.json done rm -f dockerconfig.json ``` Rotate before the old credential expires, not after. An expired pull secret is invisible until something reschedules. ## Your Gen0Sec API key ```bash printf '%s' '' > $RELAY_KEY_FILE kubectl -n gen0sec delete secret gen0sec-download-proxy kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-file=DOWNLOAD_PROXY_API_KEY=$RELAY_KEY_FILE kubectl -n gen0sec rollout restart deploy/g0s-download-api ``` Confirm it worked, because a wrong key fails silently: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means the new key is wrong, or has a trailing newline. Use `printf`, not `echo`. See [Data relay](./../configure/relay.mdx). ## Dashboard session signing key :::danger[This signs everyone out] Rotating it invalidates every active session and every outstanding invite link. There is no partial rotation and no grace period. ::: Only do this if you believe the key is compromised. Delete it and let the secret script regenerate it: ```bash kubectl -n gen0sec delete secret g0s-db-ui $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy/g0s-ui ``` The script preserves an existing key, which is why deleting the secret first is the step that actually rotates it. ## Certificates | Certificate | Owned by | |---|---| | Ingress TLS | You. Rotate in your ingress or through cert-manager | | Postgres pooler | The Postgres operator, self-signed, rotated by it | | Relay to Gen0Sec | Public certificate on our side. Nothing to do | The platform presents no certificate of its own outside the ingress you configure. ## After any rotation ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` The smoke test is the only check that proves the platform still serves an agent rather than merely starting. --- # Uninstall Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/operate/uninstall # Uninstall ## What you are about to delete | | Default | `--keep-data` | `--keep-crds` | |---|---|---|---| | The three Helm releases | Removed | Removed | Removed | | Objects the operators created outside the releases | Removed | Removed | Removed | | Kafka topic finalizers | Cleared | Cleared | Cleared | | Namespaces `gen0sec` and `gen0sec-system` | **Deleted** | Kept | Deleted | | Volumes: Postgres, Kafka, object store | **Deleted** | Kept | Deleted | | Secrets, including the database passwords | **Deleted** | Kept | Deleted | | Operator CRDs | **Removed** | Removed | Kept | :::danger[The default deletes your data, and it asks you to prove you mean this cluster] Without `--keep-data` the databases and the object store go, and nothing can undo that. So that run also **asks you to type the current context name back**, and refuses to run at all with no terminal attached. `--keep-crds` does not trigger that prompt, because a schema the next install reinstalls anyway is no reason to skip asking about the databases. ::: ## Remove everything ```bash $KIT/scripts/uninstall.sh --yes ``` It prints the cluster context, the release list and the volume count, then waits for you to type the context name. Read what it prints before you type. ## Keep the data ```bash $KIT/scripts/uninstall.sh --yes --keep-data ``` Stops after clearing the topic finalizers. The namespaces stay, and so do the volumes and the secrets in them. No context confirmation, because nothing irreversible happens. :::note[Why the secrets have to stay with the volumes] A reinstall over retained volumes needs the original database passwords: the data was written with them and a fresh Postgres operator generates different ones. Keeping the volumes without the secrets leaves you with data you cannot open. ::: ## Keep the CRDs ```bash $KIT/scripts/uninstall.sh --yes --keep-crds ``` Use this when something else in the cluster runs the same operators, or when you want the next install to skip reinstalling schemas. ## How CRD removal is made safe CRDs are cluster-scoped, and **deleting one deletes every resource of that kind in every namespace**. Another Postgres or Kafka cluster run by the same operators would go with them. So the script checks first: if any of those kinds is in use outside the two namespaces it is removing, **it skips the CRDs** and tells you. `--keep-crds` skips them unconditionally. That check is why removing CRDs by default is safe. It is not a blanket delete. If you want to see what would be affected before you run anything: ```bash kubectl get postgresqls,kafkas,kafkatopics,kafkanodepools -A 2>/dev/null | grep -v 'gen0sec' ``` Any output there means the script will leave the CRDs alone. ## The Kafka finalizer, and why order matters Strimzi puts a `strimzi.io/topic-operator` finalizer on all 23 KafkaTopics. Once the operator is gone, nothing clears it, and deleting the namespace leaves it in `Terminating` indefinitely. The script uninstalls the releases first, then clears the finalizers, then deletes the namespaces. That order is the whole reason the script exists. If you already deleted the namespace and it is stuck, this is the fix: ```bash NS=gen0sec-system kubectl get kafkatopics.kafka.strimzi.io -n $NS -o name \ | xargs -r -n1 kubectl patch -n $NS --type=merge \ -p '{"metadata":{"finalizers":null}}' ``` ## Reinstalling over retained volumes Two things bite here. **Kafka.** The volumes survive on purpose, and a recreated cluster generates a new identity. Brokers then crash loop with `Invalid cluster.id`. See [Kafka will not start](./../troubleshoot/symptoms.mdx#kafka-will-not-start). **Postgres.** You need the original role passwords. If you kept the secrets, the operator adopts the existing data. If you did not, the volumes are unreadable. ## Uninstalling by hand The script does more than these commands: it also removes objects the operators created outside the releases, and it handles the CRD safety check. Use it if you can. ```bash helm uninstall g0s -n gen0sec helm uninstall g0s-data -n gen0sec-system helm uninstall g0s-infra -n gen0sec-system # clear the finalizers before touching the namespaces kubectl get kafkatopics.kafka.strimzi.io -n gen0sec-system -o name \ | xargs -r -n1 kubectl patch -n gen0sec-system --type=merge \ -p '{"metadata":{"finalizers":null}}' kubectl delete namespace gen0sec kubectl delete namespace gen0sec-system ``` Do not pass `--wait` to `helm uninstall`. The Kafka topics cannot finish deleting until the finalizers above are cleared, so waiting only burns the timeout. --- # Upgrade Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/operate/upgrade # Upgrade When this page is done you are running the new release, with the same data. Read [Roll back](./rollback.mdx) before you start. The rollback boundary changes what a safe upgrade looks like. ## Before you start - [ ] Read the release notes for every version between yours and the target - [ ] A current backup. See [Back up and restore](./backup.mdx) - [ ] The new release's install kit, verified - [ ] A maintenance window. The application is briefly unavailable while pods roll ## Rules that are not negotiable | Rule | Why | |---|---| | One minor at a time | Skipping a minor is not supported. Each train's migrations assume the previous train ran | | Patches in place | Within a train, upgrade directly | | Same order as install: infra, then data, then platform | Each chart depends on the previous one | | Apply CRDs by hand first | Helm never upgrades CRDs. See step 1 | | Back up first | The migration hook changes your schema, and `helm rollback` does not undo that | ## 1. Apply the new CRDs **Do this before anything else.** Helm installs a chart's CRDs on first install and ignores them from then on. An upgrade that ships a new CRD schema fails with a server-side-apply error, or worse, silently rejects fields it does not know. ```bash cd $B/charts tar xzf gen0sec-infra-*.tgz kubectl apply --server-side --force-conflicts \ -f gen0sec-infra/charts/postgres-operator/crds/ kubectl apply --server-side --force-conflicts \ -f gen0sec-infra/charts/strimzi-kafka-operator/crds/ ``` The output is similar to this: ``` customresourcedefinition.apiextensions.k8s.io/postgresqls.acid.zalan.do serverside-applied ``` :::note[This is not hypothetical] The Postgres operator 1.15 to 2.0 change adds PostgreSQL 18 to the `postgresqls` schema. Without the CRD apply, the `postgresql/core` resource is rejected outright and the database never starts. ::: ## 2. Get the new artifacts Follow [Prepare the artifacts](./../04-prepare-artifacts.mdx) for the new release. Both methods work the same as on a first install, with one addition for the offline bundle. **Offline bundle only.** Mirroring is additive: the new tags land beside the old ones. Your running install keeps working until you upgrade, and a rollback still has its images. Do not reuse the previous bundle's directory. The bundler refuses to write into an existing one, and each bundle's values files are targeted at the version they shipped with. ## 3. Upgrade infra Run every command from the **new** bundle directory, so `charts/` and `values/` are the new release's. ```bash helm upgrade g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml ``` If you installed the platform under a release name or namespace other than `g0s` in `gen0sec`, add `--set components.ui.serviceAccountName=-ui,components.ui.namespace=` here. The infra chart binds the dashboard's service account by name, and a wrong name fails silently: the Components tab then cannot show infrastructure versions. Wait for the operators to finish rolling before continuing. ```bash kubectl -n gen0sec-system rollout status deploy/g0s-infra-postgres-operator kubectl -n gen0sec-system rollout status deploy/strimzi-cluster-operator kubectl -n gen0sec-system rollout status deploy/g0s-infra-synapse-operator kubectl -n gen0sec-system rollout status deploy/dragonfly kubectl -n gen0sec-system rollout status \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" ``` ## 4. Upgrade data ```bash helm upgrade g0s-data $B/charts/gen0sec-data-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml ``` ```bash kubectl -n gen0sec-system wait --for=condition=Ready kafka/core --timeout=15m kubectl -n gen0sec-system wait \ --for=jsonpath='{.status.PostgresClusterStatus}'=Running postgresql/core --timeout=15m ``` ## 5. Upgrade the platform The schema migration runs first, as a **pre-upgrade** hook, before any new pod rolls. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` Upgrading from v0.1.x, `entrypointHost` is required even if you never published the endpoints: the dashboard now builds its sign-in URL from it, and the render fails without it. `$KIT` is the **new** release's kit. Pass every file you layered at install time, in the same order: `helm upgrade` applies only what you give it. Leaving out the ingress overlay removes every Ingress, and leaving out `single-node/platform-values.yaml` on a quickstart install scales every service back to its production replica count. See [Publish the endpoints](./../06-publish.mdx). **If the migration fails, Helm aborts and the old application keeps running.** That is the designed behavior and it is why the hook runs before the rollout. ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` Fix the cause, then re-run the same command. See [Migration failures](./../troubleshoot/symptoms.mdx#migration-failures). :::note[v0.2.0: one migration needs a quiet moment] `20261004150000` changes a constraint on the block events table and waits at most 10 seconds for its lock. On a busy install the job fails with `canceling statement due to lock timeout` and leaves the schema dirty. Nothing was changed. Pause the block events pipeline, mark the previous version and re-run this step: ```bash kubectl -n gen0sec-system scale deploy/g0s-data-kafka2pg-block-events --replicas=0 kubectl -n gen0sec run db-migrate-force --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/db-migrate:$VERSION \ --env DATABASE_URL="$OWNER_DSN" -- force 20261004120000 ``` Scale the pipeline back to 1 once the upgrade completes. Events wait in Kafka meanwhile. ::: :::note[v0.2.1: check the on-prem marker first] The migration job's connection string must carry the on-prem marker, or v0.2.1 stops rule changes reaching agents. `make-db-secrets.sh` always adds it. If your secret manager produces `gen0sec-db-migrate`, confirm before you run this step: ```bash kubectl -n gen0sec get secret gen0sec-db-migrate -o jsonpath='{.data.DATABASE_URL}' \ | base64 -d | grep -c 'gen0sec.deployment_mode%3Donprem' ``` `1` means it is there. See [Secrets](./../configure/secrets.mdx#where-the-database-passwords-come-from). A `WARNING: ... left NOT VALID` in the migration log is not a failure. It means existing rows break a check that new writes now have to pass, and the migration continues. ::: ## 6. Verify ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` Run the smoke test after every upgrade, not just after installs. It is the only check that proves the platform still serves an agent rather than merely starting. Full checklist: [Verify the install](./../08-verify.mdx). ## If you changed values along the way `helm upgrade` uses the values files you pass, not the ones from last time. Anything you set with `--set` on the previous install must be set again, or it reverts to the chart default. Check what the running release actually has: ```bash helm get values g0s -n gen0sec ``` Keep your overrides in a file rather than in shell history. See [Reference configurations](./../configure/reference-configs.mdx). ## If it fails | Symptom | Start here | |---|---| | CRD schema error on `helm upgrade` | You skipped step 1 | | `Unsupported value` from the Postgres operator | You skipped step 1 | | Migration job fails | [Migration failures](./../troubleshoot/symptoms.mdx#migration-failures) | | Migration job fails with `lock timeout` | A busy block events table. See the note in step 5 | | Pods roll but the smoke test fails | [Roll back](./rollback.mdx) | | `ImagePullBackOff` on new tags | The new images were not mirrored. Offline bundle method, step 2 | ## Next [Roll back](./rollback.mdx). --- # Prepare the artifacts Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/prepare-artifacts import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; # Prepare the artifacts ## Before you start - [ ] [Requirements](./02-requirements.mdx) met, tools installed, `yq` version checked - [ ] [Network and connectivity](./03-network.mdx) configured by your network team - [ ] Your [registry token](./02-requirements.mdx#the-registry-token), created on the dashboard under **Repository Access** - [ ] Your [Gen0Sec API key with the required scopes](./02-requirements.mdx#your-gen0sec-api-key) ## 0. Set your keys Create your token files. ```bash mkdir -p ~/.gen0sec cd ~/.gen0sec/ && touch .registry-token .download-api-key ``` Then open up an editor of your choice to fill each file with their respective content. Then run ```bash export REGISTRY_TOKEN=$(cat ~/.gen0sec/.registry-token) export G0S_DOWNLOAD_API_KEY=$(cat ~/.gen0sec/.download-api-key) ``` :::tip[Manage your keys securely] By storing keys in files, you make sure they don't get etched into your shell history. If you have history turned off or you know what you're doing, you can also go with ```bash export REGISTRY_TOKEN="your-registry-token" export G0S_DOWNLOAD_API_KEY="your-gen0sec-api-key" ``` ::: ## 1. Set the release Pin the release explicitly. A pinned tag installs the same bytes in six months; a resolved one does not. You can find available release versions on the [Releases page](https://releases.gen0sec.com/cerebellum/). ```bash export TAG=v0.2.1 export VERSION=${TAG#v} ``` To always use the latest version instead: ```bash export TAG=$(curl -fsSL https://releases.gen0sec.com/api/repos/cerebellum/latest | jq -r .tag_name) export VERSION=${TAG#v} ``` :::note[Release candidates are not published here] Only stable releases are mirrored, so an `-rc.N` tag returns 404 no matter how it is spelled. The [release index](https://releases.gen0sec.com/cerebellum/) is the list of what exists. ::: ## 2. Download the install kit ```bash curl -fsSLO https://releases.gen0sec.com/cerebellum/$TAG/cerebellum-install-kit.tar.gz curl -fsSLO https://releases.gen0sec.com/cerebellum/$TAG/cerebellum-install-kit.tar.gz.cosign.bundle ``` ## 3. Verify the signature Run this before you unpack anything. ```bash cosign verify-blob \ --bundle cerebellum-install-kit.tar.gz.cosign.bundle \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/release\.yaml@refs/tags/' \ cerebellum-install-kit.tar.gz ``` The output is similar to this: ``` Verified OK ``` Anything else means stop. Do not unpack the archive. :::note[Why both certificate flags are required] `cosign` refuses to verify without an expected issuer and an expected identity, so a shortened command fails rather than passing weakly. The identity is pinned to one workflow on a tag reference. A broader pattern would accept a signature produced by any workflow in our organization. The signature covers the file's contents, so it proves the bytes are ours regardless of how they reached you. ::: ## 4. Unpack it ```bash tar -xzf cerebellum-install-kit.tar.gz export KIT=$PWD/cerebellum-install-kit-$VERSION cd $KIT ``` The kit contains the bundler, the release manifest, the helper scripts used later in this install, and the sizing overlays. It does not contain images or charts. ## 5. Log in to our registry ```bash export REGISTRY=registry.gen0sec.com export REGISTRY_USER=any printf '%s' "$REGISTRY_TOKEN" | helm registry login $REGISTRY -u "$REGISTRY_USER" --password-stdin printf '%s' "$REGISTRY_TOKEN" | skopeo login $REGISTRY -u "$REGISTRY_USER" --password-stdin ``` Both are needed. `helm` and `skopeo` keep separate credential stores, and the next step uses each of them. `cosign` is a third store again, and it needs no login of its own. The bundler lends it the credential `skopeo` holds. ## 6. Get the charts Pull the charts and their values files. Images are not pulled: your cluster pulls each one from `registry.gen0sec.com` when it needs it. :::note[Confirm your nodes can pull from us before you go further] Our registry challenges pulls that send no `User-Agent`, which on some distributions is what the kubelet's pull path does. This step succeeds either way, because it runs from your workstation. The failure appears at step 3 of [Install](./05-install.mdx), as a 403 on every image. Check it now: [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: ```bash ./bundler.sh assemble-charts release-manifest.yaml export B=$PWD/gen0sec-cerebellum-multiarch-$VERSION ``` The output ends with a per-chart digest confirmation. :::note[What "digest-verified" means here] `assemble-charts` checks two digests per chart: the registry's manifest against the digest this release pins, and the chart archive against the layer named in that manifest. What you install provably came out of the chart this release published. ::: Each chart carries its own `values-onprem.yaml`, extracted into `$B/values/`. That file is what points the install at a registry. The default `values.yaml` leaves the registry blank on purpose, so installing without `-f` will not resolve images. ### 6a. Build the bundle on your workstation ```bash ./bundler.sh create-package release-manifest.yaml ``` This pulls every image and chart the release pins by digest, verifies every signature, then packages the result as `gen0sec-cerebellum-multiarch-$VERSION.tar.gz`. It carries both `linux/amd64` and `linux/arm64`, so it installs on any node in your cluster. Signature verification happens here, on the connected side, because it needs the public transparency log. Everything downstream is protected by checksums instead. :::note[Before you use --arch to halve the transfer] `--arch amd64` or `--arch arm64` extracts one manifest out of each multi-architecture index. That changes every image digest by definition. Per-architecture signatures still verify, because each one is signed individually. **SBOM attestations do not**, because they are attached to the index, and the index is what a single-architecture bundle leaves behind. The bundler warns at the time and annotates the bundle. If your acceptance process checks attestations, stay on the default. If your nodes are mixed, or you are unsure what they run, stay on the default: a missing architecture fails when the kubelet tries to pull, on an isolated cluster, after the transfer. A single-architecture bundle also renames the tarball and the directory inside it, so substitute the name in every command that follows. ::: ### 6b. Move it across In this step you take your bundle onto an intermediary machine inside the network where your internal registry and the target cluster is located you'll install the bundle onto. After transferring the bundle, you can clean up all artifacts it has created. eg. ```bash scp gen0sec-cerebellum-multiarch-$VERSION.tar.gz :~/ # `create-package` ran inside the kit directory, so the bundle is sitting in it. Copying the kit first # would transfer tens of gigabytes twice. `cleanup` removes only what the bundler created. ./bundler.sh cleanup cd .. && scp -r cerebellum-install-kit-$VERSION :~/ # You can even remove the installation kit if you don't intend to keep it. # cd $KIT/.. && rm -rf cerebellum-install-kit.tar.gz $KIT ``` ### 6c. Load it into your registry Log in to your intermediary machine and use the bundler to load the images into your internal registry. ```bash export REGISTRY=registry.internal.example export REGISTRY_USER=your-registry-user export REGISTRY_TOKEN=$(cat path/to/your-registry-token) # Carried across because this is a different machine and Install step 2 needs # it. Unset, the secret it creates is empty and every relay fetch 401s. export G0S_DOWNLOAD_API_KEY=$(cat path/to/your-gen0sec-api-key) export KIT=~/cerebellum-install-kit-$VERSION export VERSION=$(cat $KIT/release-manifest.yaml | yq ".spec.version") tar -xzf gen0sec-cerebellum-multiarch-$VERSION.tar.gz export B=~/gen0sec-cerebellum-multiarch-$VERSION cd $B printf '%s' "$REGISTRY_TOKEN" | skopeo login $REGISTRY -u "$REGISTRY_USER" --password-stdin ./bundler.sh apply release-manifest.yaml -r $REGISTRY ``` `apply` pushes every bundled image into your registry, then rewrites the bundled values files to point at it. The charts are not pushed: you install them from `$B/charts/*.tgz`. The bundle is checked against its own `SHA256SUMS` before anything is pushed, so a corrupted transfer fails here rather than at install time. :::note[For a plain-HTTP registry] Set `MIRROR_TLS_VERIFY=false` before `apply`. This affects the push only, and it is the one place in this install where TLS verification can be disabled. Prefer a certificate from your internal CA. ::: ## Checkpoint Whichever method you used, you should now have the following: | Variable | Points at | |---|---| | `$KIT` | The unpacked install kit | | `$B` | A directory containing `charts/` and `values/` | | `$REGISTRY` | The registry your cluster will pull from | | `$REGISTRY_USER` | The username for that registry (`any` for ours) | | `$REGISTRY_TOKEN` | The password or token for that registry | | `$G0S_DOWNLOAD_API_KEY` | Your Gen0Sec API key, set in step 0. [Install](./05-install.mdx#your-gen0sec-api-key) step 2 puts it in a secret | You are logged in to `$REGISTRY`. Every page after this one assumes exactly that and nothing else, which is why nothing after this point branches. --- # Publish the endpoints Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/publish # Publish the endpoints The ingress controller is installed and running. However it's not configured to serve anything by default. That is deliberate, because creating those routes makes the APIs reachable from wherever the controller is reachable, so it is your decision, not ours. The data plane's LoadBalancer Service already has an external address at this point. Until you complete this page it serves nothing, because no rule names a host. :::note[Ingress and TLS are yours to configure] The platform ships the routing rules. Terminating TLS, issuing the certificate and exposing the LoadBalancer are your cluster's responsibility. For the configuration options, see the [Kubernetes ingress reference](https://docs.gen0sec.com/docs/client/kubernetes-ingress). ::: ## Before you start - [ ] [Install](./05-install.mdx) complete, all pods running - [ ] A DNS name you control, resolving to your ingress controller's address - [ ] A LoadBalancer address for the ingress data plane, or a Service type your cluster can expose - [ ] A TLS certificate for that name, or cert-manager configured ## 1. Choose the address One address serves the dashboard and every agent-facing API. Requests are routed by path prefix. Use the address you set as `entrypointHost` in [Install step 7](./05-install.mdx#7-install-the-platform). It is already in `$ENTRYPOINT_HOST` if you followed the install in this shell: ```bash export ENTRYPOINT_HOST=cerebellum.example.internal ``` **The host is required, and the chart enforces it.** A rule with no host is silently ignored by the ingress controller: the Ingress object is created, no event is logged, and nothing serves traffic. Rather than publish one, the chart refuses to render and names the value to set, so this fails at `helm upgrade` instead of in production. **Use the address people will actually type.** The dashboard derives its sign-in URL from this host, and sign-in fails if the two do not match. ## 2. Publish The kit ships the overlay. Layer it on top of the values file you have been using. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` :::warning[Re-pass every values file you installed with] `helm upgrade` applies only the files you pass it. Anything layered at install time and omitted here reverts to the chart default. On the [Quickstart](./quickstart.mdx) you also passed `-f $KIT/examples/single-node/platform-values.yaml`, so include it in this upgrade too (before the ingress overlay), or every service scales back to its production replica count: ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-node/platform-values.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` ::: :::note[Quote any --set argument containing brackets] `zsh` treats `[0]` as a glob and fails before `helm` runs. Single-quote the whole argument: `--set 'services.ui.ingress.hosts[0].host=...'`. ::: ## 3. Verify the routes exist ```bash kubectl -n gen0sec get ingress ``` The output is similar to this: ``` NAME CLASS HOSTS ADDRESS PORTS AGE g0s-platform synapse cerebellum.example.internal 10.0.0.42 80 30s ``` An empty `HOSTS` column means the host was not applied, and nothing will serve traffic. The upgrade fails before it can leave an Ingress in that state, so if you see one it was created by an earlier install. Then check from outside the cluster. The dashboard is published at `/` and returns a redirect to its sign-in page, so follow it with `-L`: ```bash curl -fsSL -o /dev/null -w '%{http_code}\n' https://$ENTRYPOINT_HOST/ ``` The output is similar to this: ``` 200 ``` :::note[There is no health endpoint on the ingress] `/status/healthz` is served per pod inside the cluster and is deliberately **not** published through the ingress. A request to it from outside falls through to the dashboard catch-all and returns `404`, so do not use it as an external check. The dashboard root above confirms reachability; the [smoke test](./08-verify.mdx#4-the-platform-serves-an-agent) confirms the platform serves agents. ::: ## Find the address to point DNS at The ingress data plane is a `LoadBalancer` Service. ```bash kubectl -n gen0sec-system get svc g0s-infra-synapse -o wide ``` Point your DNS name at the external address. If `EXTERNAL-IP` stays ``, your cluster has no LoadBalancer implementation: change the Service type to something it can expose. It cannot stay cluster-internal. ## The route table You do not configure these. The overlay sets them, read off each service's own router. They are here because they are the answer to "a request landed on the wrong backend", and because a missing route fails quietly rather than loudly. | Path prefix | Served by | |---|---| | `/` | Dashboard (catch-all, shortest prefix, so it is last resort) | | `/v1/indicators`, `/v1/models`, `/v1/geoip`, `/v1/ids-rules` | Relay | | `/v1/penalty-model`, `/v1/traffic-model`, `/v1/ja4-classifier`, `/v1/flow-classifier`, `/v1/jepa-embedder` | Relay, per-model aliases | | `/v1/identity`, `/v1/policy-edges`, `/v1/fleet-bans` | Relay, served locally | | `/v1/authcheck`, `/v1/auth` | Authentication | | `/v1/events`, `/v1/agents`, `/v1/log` | Agent ingest | | `/v1/agents/events` | Configuration push (longer prefix than `/v1/agents`, so it wins) | | `/v1/config` | Configuration | | `/v1/telemetry`, `/v1/logs` | Telemetry | | `/v1/signal`, `/v1/rules` | Signal | | `/v1/ban-ledger` | Ban ledger | | `/v1/threat` | Threat feeds | | `/v1/service-graph` | Service graph | Routing is by **longest matching prefix**, because one address serves every backend. The configuration push stream on `/v1/config` is long-lived, so the chart sets read, write and idle timeouts of 3600 seconds on that route. Do not lower them: agents fall back to polling with no push, which looks like a slow platform rather than a timeout. :::note[Why a missing route is worse than a broken one] The dashboard owns `/`, so any path absent from this table falls through to it and gets a redirect or HTML. HTTP has no way for the caller to tell that apart from a real answer, so the failures are silent: | Missing route | What the agent actually does | |---|---| | `/v1/authcheck` | Receives HTML, cannot distinguish a bad key from a routing mistake, and continues unverified | | `/v1/agents/events` | Logs `SSE connect returned HTTP 404`, then falls back to polling. Looks like a slow platform, not a routing bug | | `/v1/ban-ledger` | Reads the redirect as success. The contribution is lost with no error anywhere | This is why the overlay exists rather than a list of prefixes for you to type. ::: :::note[/v1/ids-rules goes to the relay, not to the in-cluster rules service] IDS rules are relayed from gen0sec. The in-cluster rules service builds rules from a local ingest pipeline that this edition does not run. Route this prefix there and you reach a service with no rules, and the relay is never used. ::: ### If TLS terminates upstream If a load balancer, CDN or tunnel terminates TLS in front of the ingress, the chart cannot infer the external scheme and guesses `http` while the browser is on `https`. The dashboard then builds its sign-in URL with the wrong scheme and rejects every sign-in with `Invalid origin`. Set it explicitly: ```bash --set externalScheme=https ``` Leave it unset when TLS terminates at the ingress, because then the chart reads it from that service's ingress TLS block. One service can differ from the rest with `services..ingress.externalScheme`, which wins over the install-wide value above. Only a service that derives its own external URL reads it, and the dashboard is the only one that does. ## The dashboard The dashboard is enabled by default and is published at the root of your address. Sign-in is email and password, held in your own Postgres, so nothing outside the cluster is involved. Its secret, `g0s-db-ui`, was created in [Install](./05-install.mdx) step 6 with all the others. Re-running that script keeps the existing signing secret, so sessions and invite links stay valid. To leave the dashboard out: ```bash --set services.ui.enabled=false ``` ## If it fails | Symptom | Cause | |---|---| | `helm upgrade` fails with `entrypointHost is required` | Expected, and the point: `entrypointHost` was not passed. Set it and re-run | | `kubectl get ingress` shows an empty `HOSTS` column | An Ingress left by an earlier install, created when a host-less rule could still render. Re-run the upgrade with `entrypointHost` set | | Sign-in is rejected with `Invalid origin` | The dashboard's URL does not match the address people type. Check `entrypointHost`, and `externalScheme` if TLS terminates upstream | | `curl` times out | DNS does not resolve to your ingress controller, or a firewall blocks the port | | The dashboard loads but sign-in redirects to the wrong address | The sign-in URL is derived from `entrypointHost`. It must be the address people type | | A request reaches the wrong service | Longest-prefix routing. Check the `/v1/agents` overlap above | --- # Quickstart on one node Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/quickstart # Quickstart on one node :::danger[Not for production] This runs every component at a single replica on a single node. There is no redundancy anywhere: **losing the node loses the data.** Postgres has no standby, Kafka has one broker at replication factor 1, and the object store has no erasure coding. Use it for evaluation, training and upgrade rehearsals. For anything you depend on, follow [Sizing a production install](./02-requirements.mdx#sizing-a-production-install), which runs on three nodes. ::: ## What you get A working instance you can sign in to, on one machine, without planning a cluster. This is not a separate procedure. It is the same install as the rest of this documentation, with one extra values file per chart. ## What you need | | | |---|---| | One node | 12 vCPU, 32 GB RAM, 200 GB disk | | Kubernetes | k3s | | StorageClass | Any default block StorageClass. k3s installs `local-path` as the default, which is enough | | Tools | The same list as [Requirements](./02-requirements.mdx#tools-on-your-workstation) | | Credentials | The same two as [Requirements](./02-requirements.mdx#credentials-from-gen0sec). The relay is required here too | ## How it differs from production The kit ships an overlay per chart that reduces everything to one replica. | | Production | This quickstart | |---|---|---| | Object store | 3 replicas, erasure coded, distributed | 1 replica, **standalone mode**, and a Deployment rather than a StatefulSet | | Postgres | 3 instances | 1 instance, no replica pooler | | Kafka | 3 brokers, replication factor 3 | 1 broker, every factor 1 | | Application services | 2 or 3 replicas each | 1 each | | Cache | Chart default | 256 MB, one thread | | Storage | Your block StorageClass, 500 GiB per node minimum | The cluster's default StorageClass, about 65 GiB of volumes | | Survives losing a node | Yes | No | :::note[The connection pooler stays on] Only the **replica** pooler is disabled, because there is no replica to pool. Every service still connects through `core-pooler`, exactly as in production, so the secrets that `make-db-secrets.sh` composes are the same shape here. ::: :::note[Two things that surprise people] The object store stops being a StatefulSet and becomes a Deployment of the same name. That is why the readiness checks in [Install](./05-install.mdx) resolve the kind instead of naming it. Volume claims from a previous multi-replica install are not reused. If you are converting an existing install, delete them first. ::: ## Run it Follow the spine exactly as written, starting at [Prepare the artifacts](./04-prepare-artifacts.mdx). Three commands change. Nothing else does. | Spine step | On this install | |---|---| | [Prepare the artifacts](./04-prepare-artifacts.mdx) | Unchanged. Either method works, and the registry method is fewer steps | | [Install](./05-install.mdx) steps 1 and 2 | Unchanged. Same namespaces, same three secrets | | [Install](./05-install.mdx) steps 3, 4 and 7 | Add the overlay. See below | | [Install](./05-install.mdx) steps 5 and 6 | Unchanged. You still create the bucket, and you still run the secret script | | [Publish the endpoints](./06-publish.mdx) | Unchanged. See the note on the address below | | [First sign-in](./07-first-signin.mdx) and [Verify the install](./08-verify.mdx) | Unchanged | Pass the overlay **after** the production values file, so it wins. ```bash helm install g0s-infra $B/charts/gen0sec-infra-*.tgz -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml \ -f $KIT/examples/single-node/infra-values.yaml ``` ```bash helm install g0s-data $B/charts/gen0sec-data-*.tgz -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml \ -f $KIT/examples/single-node/data-values.yaml ``` ```bash helm install g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-node/platform-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` The wait gates after each one are the same as on [Install](./05-install.mdx). Read the files before you use them: each overlay carries comments explaining what it trades away. :::caution[Your cluster needs a default StorageClass] The overlays leave the storage class empty in all three places — `rustfs.storageclass.name` in `infra-values.yaml`, `postgres.volume.storageClass` and `kafka.storage.class` in `data-values.yaml` — which means "use the cluster's default", the same as the production values file. On k3s that resolves to `local-path` and there is nothing to do. If your cluster has **no** default, every PVC stays `Pending`. `helm install` still reports success, and the only symptom is the wait gate above timing out without naming a cause. Check first: ```bash kubectl get storageclass # exactly one row must be marked (default) ``` To pin a class explicitly instead of relying on the default, set those three keys, or layer a fourth `-f` file last. ::: ## The address [Publish the endpoints](./06-publish.mdx) is unchanged, including the requirement for a host, which the platform install above already needs in `$ENTRYPOINT_HOST`. On k3s the bundled LoadBalancer implementation gives the ingress data plane the node's own address, so there is nothing further to provision. For an evaluation, a name in `/etc/hosts` on the machine you browse from is enough. It has to be the name you actually type in the browser, because the dashboard builds its sign-in URL from it. :::note[About TLS terminatotion] In case your TLS termination does not happen on a higher level, configure Synapse Ingress to work with cert-manager (see the [Kubernetes ingress reference](https://docs.gen0sec.com/docs/client/kubernetes-ingress)). The shipped ingress overlay sets `externalScheme: https` and configures no certificate at the ingress. A single-node evaluation usually has none. Either provision a certificate, or serve plain HTTP: set `externalScheme` to `http` in the ingress overlay and use an `http://` base URL in the sign-in and verify steps below. ::: ## First sign-in Nothing is different here, and it is worth knowing before you get there: the dashboard makes you enroll multi-factor authentication and replace the seeded password before anything opens. See [First sign-in](./07-first-signin.mdx). ## Verify Run the same checks as a production install, from [Verify the install](./08-verify.mdx). The smoke test is the one that matters, because it proves the platform serves an agent rather than merely starting: ```bash # Save the agent key you minted at first sign-in, then read it back below. mkdir -p ~/.gen0sec printf '%s' '' > ~/.gen0sec/agent-key export GEN0SEC_BASE_URL=https://$ENTRYPOINT_HOST export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) $KIT/scripts/verify-deployment.sh ``` A single-node install that passes it is a good rehearsal for a real one. That is most of the value of building this. ## Tearing it down ```bash $KIT/scripts/uninstall.sh --yes ``` It asks you to type the cluster context name back before deleting anything, so it will not run unattended. See [Uninstall](./operate/uninstall.mdx). ## When you are ready for production [Sizing a production install](./02-requirements.mdx#sizing-a-production-install) has the node counts, per-node capacity and storage figures. There is no in-place conversion from this install to a production one: build the production cluster separately. --- # Bundler CLI Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/reference/bundler # Bundler CLI `bundler.sh` ships in the install kit. It builds an offline bundle on a connected host and loads it into your registry on a host that has no internet access. Its only input is the `release-manifest.yaml` inside the kit. No access to our source is needed, so you can build your own bundle from any published release. For the procedure rather than the reference, see [Prepare the artifacts](./../04-prepare-artifacts.mdx). ## Which commands need internet access This is the table an isolated site actually needs. The pipeline is split on internet access, not on environment. | Command | Internet | Talks to | |---|---|---| | `validate` | **Required** | Parses the manifest, resolves tooling, then fetches one image manifest to confirm registry access | | `assemble` | **Required** | Source registry: images and charts | | `assemble-charts` | **Required** | Source registry: charts only | | `assemble-images` | **Required** | Source registry: images only | | `verify-assembly` | **Required** | Source registry for signature artifacts, plus Sigstore trust material | | `create-package` | **Required** | Inherits `assemble` and `verify-assembly` | | `package` | None | Local `tar` | | `verify-package` | None | Local extract and checksum comparison | | `mirror` | None | Destination registry, over your local network | | `retarget` | None | Local files | | `apply` | None | Destination registry. This is `mirror` plus `retarget` | | `cleanup`, `version`, `help` | None | Local | **Split the pipeline at the tarball, not earlier.** `create-package` cannot run on an isolated host: it fails in preflight with no route to the source registry. ## Commands | Command | What it does | |---|---| | `validate` | Preflight. The manifest exists and its schema is sane, every image and chart is digest-pinned, the tooling resolves, and the source registry is readable. Fetches one image manifest, so a bad host, missing login, TLS mismatch or unpublished digest fails here rather than partway through a multi-gigabyte download | | `assemble` | Creates the bundle directory, pulls every image and chart, extracts each chart's production values file, copies in the manifest and itself. Fails if the directory exists | | `assemble-charts` | The charts half. **What the registry install method uses**, since the cluster pulls its own images | | `assemble-images` | The images half. Separable so a failed multi-hour pull can be retried without re-fetching charts | | `verify-assembly` | Keyless signature verification of every pinned image and chart, then writes `SHA256SUMS` and prints the bundle **fingerprint**. Reports every failure and refuses to continue on any | | `package` | Tars the bundle directory into a tarball **beside** it, never inside | | `verify-package` | Unpacks to a temporary directory and re-checks `SHA256SUMS` | | `create-package` | `assemble`, `verify-assembly`, `package`, `verify-package`, in order | | `mirror` | Verifies the bundle against `SHA256SUMS`, then pushes every image to your registry | | `retarget` | Rewrites the registry host in the bundle's own values files, in place, and updates the affected `SHA256SUMS` entries so the bundle stays self-consistent | | `apply` | `mirror` then `retarget`. The usual customer step | | `cleanup` | Removes every directory and archive the bundler created | | `version` | The bundler's version, stamped at release | | `help` | Usage | Every command runs `validate` first, so any of them is safe to invoke on its own. ## Options | Option | Effect | |---|---| | `--arch amd64` or `--arch arm64` | Single-architecture bundle, roughly half the size. **Loses SBOM attestations.** See below | | `-o ` | A different tarball name | | `-r ` | Destination registry for `mirror`, `retarget` and `apply` | | `--dangerously-skip-cosign-verify` | Drops signature verification and its tooling requirement. The bundle is annotated as unverified | ### The --arch trade-off A single-architecture bundle extracts one manifest out of each multi-architecture index, which changes every image digest by definition. | | Default | With `--arch` | |---|---|---| | Image signatures | Verify | **Verify.** Each per-architecture manifest is signed individually | | SBOM attestations | Verify | **Lost.** They hang off the index, which is what is left behind | | Digests match the manifest | Yes | No | | Tarball and directory name | `multiarch` | Named after the architecture, so substitute it everywhere | If your acceptance process checks attestations, or your nodes are mixed, or you are not certain what they run: take the default. A missing architecture fails when the kubelet tries to pull, on an isolated cluster, after the transfer. ## Environment variables | Variable | Purpose | |---|---| | `MIKEFARAH_YQ_PATH` | Path to the Go `yq`, when the one on `PATH` is the Python one | | `BUNDLE_TLS_VERIFY=false` | Disables TLS verification on the **pull** side | | `MIRROR_TLS_VERIFY=false` | Disables TLS verification on the **push** side, for a plain-HTTP registry | The two TLS variables are the only places in the whole install where verification can be disabled. Prefer a certificate from your internal CA. See [Hardening](./../security/hardening.mdx). ## Credentials, and the three stores Three tools, three credential stores. You log in to two of them. | Tool | Store | Login needed | |---|---|---| | `skopeo` | The containers store | Yes, both source and destination | | `helm` | `HELM_REGISTRY_CONFIG` | Yes, source only | | `cosign` | The Docker config | **No.** The bundler lends it the one `skopeo` holds | If you see the bundler say it is reusing skopeo's credential for cosign, that is the expected path and not a warning. ## Behavior worth knowing | | | |---|---| | `assemble` refuses to write into an existing directory | So a stale bundle cannot be silently mixed with a new one | | `retarget` takes no file arguments | Rewriting arbitrary paths handed in on a command line is not something an isolated install should be able to ask for | | `retarget` verifies integrity **before** editing, never after | That order is what stops a checksum update from re-blessing a bundle that arrived altered | | `retarget` is idempotent | Running `apply` twice is safe | | `mirror` runs before `retarget` inside `apply` | So a registry failure leaves your values files untouched | | Mirroring is additive | New tags land beside old ones, so a running install keeps working and a rollback still has its images | | An egress proxy is fine for the bundler | Provided the host can reach the registry and Sigstore. Unlike the relay, these are ordinary CLI tools that honour proxy variables | ## Bundle contents ``` gen0sec-cerebellum--/ ├── oci/ every image, as an OCI layout ├── charts/ installable chart archives ├── values/ each chart's production values, retargeted by `apply` ├── release-manifest.yaml the digests this bundle was built from ├── bundler.sh a copy └── SHA256SUMS over every file above ``` `assemble` extracts each chart's values file as `values/-values-onprem.yaml`. There is no bare `values-onprem.yaml` in the bundle root, which is why the install commands name the file in full. The bundle does **not** carry the install kit's `scripts/` or `examples/`. Those cross the air gap inside the kit directory. See [Install kit contents](./kit.mdx). --- # Release notes Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/reference/changelog # Release notes ## v0.2.1 A patch release. Upgrade in place from v0.2.0 following [Upgrade](./../operate/upgrade.mdx). From v0.1.x, upgrade straight to v0.2.1, and read the v0.2.0 notes below first. It carries 17 schema migrations, which run as the pre-upgrade hook. They bring our hosted and on-premises databases onto one migration chain, so most of them re-apply schema an on-premises database already has and change nothing there. What they do change on yours is listed below. None removes anything and every down migration is a no-op, so rolling back to v0.2.0 is safe; see [Roll back](./../operate/rollback.mdx). :::warning[Check the on-prem marker before upgrading] Migrations tell an on-premises database from our hosted one by a marker on the migration job's connection string. Without it, v0.2.1 points rule-change notifications at an address that exists only in our hosted deployment, and rule changes stop reaching agents. `make-db-secrets.sh` always adds it. If your secret manager produces `gen0sec-db-migrate`, confirm it is there: ```bash kubectl -n gen0sec get secret gen0sec-db-migrate -o jsonpath='{.data.DATABASE_URL}' \ | base64 -d | grep -c 'gen0sec.deployment_mode%3Donprem' ``` `1` means it is there. On `0`, add it before upgrading; see [Secrets](./../configure/secrets.mdx#where-the-database-passwords-come-from). ::: ### Fixed **Rule changes did not reach agents.** Creating, editing or deleting an access, firewall, WAF, smart firewall or IDS rule, or switching a managed rule on or off, saved the change but did not regenerate the configuration agents fetch. The fleet kept enforcing the previous rules, deleted ones included. A database trigger asks the config generator to regenerate after every rule write, and it needs elevated privileges to make that call. A migration shipped in v0.1.0 replaced the trigger without them, so the call was refused for the dashboard's and the services' database roles. The refusal was logged as a PostgreSQL warning and the rule write still succeeded, so nothing in the dashboard showed it: ``` WARNING: notify_config_generator: POST to http://g0s-config-generator... failed: permission denied for function http_post ``` The trigger runs with those privileges again. Rules saved before the upgrade are picked up by the next rule change in the same workspace, which regenerates that workspace's whole configuration. **Managed WAF rules did not work.** On an on-premises database the column linking a workspace to a managed WAF rule was named `rule_managed_id`, while the dashboard, the config generator and the database trigger all read `waf_rules_managed_id`. So: - The dashboard could not list managed WAF rules or switch them on or off. - Managed WAF rules switched on for a workspace were never sent to its agents. - Changing a managed WAF rule failed with `column s.waf_rules_managed_id does not exist`. v0.2.0 listed the last of these as fixed. That fix was correct for our hosted database, whose column already had the new name, and it is what produced this error on yours. The column is now renamed on upgrade, so every reader finds it. **Creating your organization ended on a page not found.** Right after first-run setup created the organization, the dashboard sent you to `//check-workspace`, which does not exist. You now land on the workspace overview. ### Changed **The overview no longer offers Threats.** The Threats module is not part of the self-hosted release, and its navigation item was already hidden, but the overview still showed its card and counted it. The overview now shows the six modules you have. **Some roles gain permissions.** Where a role, built-in or custom, already holds the first permission, the upgrade grants it the second: | Holds | Gains | |---|---| | `api_keys:read` | `registry_keys:read` | | `api_keys:write` | `registry_keys:write` | | `smart_firewall_rules:read` | `fleet_bans:read` | | `smart_firewall_rules:write` | `fleet_bans:write` | So anyone who could manage API keys can now manage registry tokens under **Repository Access**, and anyone who could manage smart firewall rules can now manage fleet bans. Review custom roles after the upgrade if either should stay separate. Nothing is revoked. ### Security - The dashboard's database role can no longer hard-delete firewall rules. Firewall rules are soft-deleted, so the dashboard never needed it. ### Documentation - [Secrets](./../configure/secrets.mdx#where-the-database-passwords-come-from) now spells out the on-prem marker the migration connection string needs, rather than only naming it. ## v0.2.0 A minor release. Upgrade from v0.1.0 or v0.1.1 following [Upgrade](./../operate/upgrade.mdx). It carries 18 schema migrations, which run as a pre-upgrade hook, so take a backup first. Every one only adds, so rolling the platform back to v0.1.x afterwards is safe; see [Roll back](./../operate/rollback.mdx). :::warning[Fleet bans start being enforced] On v0.1.x the fleet-ban artifact was written in a format agents could not read, so every agent loaded zero fleet bans while the publishing job reported success. From v0.2.0 they load. Agents with `platform.fleet_bans.enforce` on will start refusing traffic they did not refuse before. Review your fleet bans, or turn enforcement off, before you upgrade. ::: :::warning[Invitation links issued on v0.1.x stop working] Invitations are now tracked in the database, and links issued before the upgrade are not in it. Re-invite anyone who has not accepted yet. ::: :::warning[Check `externalScheme` in your values before upgrading] `externalScheme` was ignored on v0.1.x and is read from v0.2.0. If your values carry a top-level `externalScheme` that has never had any effect, **it takes effect on upgrade** and changes the scheme in the dashboard's sign-in URL. A scheme that does not match the address people actually type is rejected as `Invalid origin`. ```bash helm get values g0s -n gen0sec | grep externalScheme ``` Remove it or correct it before upgrading. Nothing to do if you pass `examples/single-entrypoint/ingress-values.yaml` unchanged: it ships `externalScheme: https`, which is the scheme v0.1.x effectively used anyway. ::: ### Added - **Passkeys.** Users can register and sign in with a passkey, and manage them under **Profile**. Passkeys are bound to the published host name and need HTTPS. See [Sign-in and identity](./../configure/identity.mdx#passkeys). - **Invitation and password-reset email**, through your SMTP server. Without SMTP, invitations fall back to a link you copy. See [Invitations and email](./../configure/identity.mdx#invitations-and-email). - **Role-scoped invitations.** An invitation grants specific workspaces, each with a role no higher than the inviter's. Pending invitations are listed and can be revoked, and access to one workspace can be revoked without removing the person. - **Audit log**, under **Organization settings** for the Owner and Admin roles. It records changes made in the dashboard and rule changes made through the signal API, with source IP and user agent. It is kept indefinitely; see [Known limitations](./../troubleshoot/known-limitations.mdx). - **Components**, under **Organization settings**: what each component runs, against the latest version known to the release. An online check for newer versions is off by default. See [Component versions](./../configure/components.mdx#verifying-what-you-are-actually-running). - **WAF rules accept `regex_replace()` and the `http.request.raw_path` field.** See [Known limitations](./../troubleshoot/known-limitations.mdx#new-waf-rule-functions-need-current-agents) for agent versions. - `CACHE_L1_TTL` sets the default lifetime of the in-process cache, as a duration. Default `30s`, up from a fixed `15s`. Set it through `global.commonEnv`. ### Fixed **Fleet bans never reached agents.** See the warning above. **Block events were lost in several ways.** Each of these applied to v0.1.x: - A batch of more than 20,000 records was refused whole, and OTLP senders do not retry that. The first 20,000 are now kept. - One row the database refused sent its whole insert batch, other organizations' rows included, to the dead-letter topic. Only the refused row is parked now, with the reason in a `dlq_reason` header. - A NUL byte in any field, a `source_ip` that was not an IP address, or a non-ASCII country code made the database refuse the row. These are now cleaned or dropped, and the event is kept. - A non-integer hit count stalled the dashboards' aggregates for every organization. It is now dropped, and the event kept. - An agent newer than the platform could report a source the platform did not know, and the event was refused. It is now stored without its source. `fleet_ban` is also accepted as a source. - IDS alerts were rejected as malformed block events and warned about once a minute. Records that are not block events are now skipped quietly. **Block events lacked country and ASN** whenever the GeoIP lookup missed, although the agent had sent them. The country is also stored in upper case, so `us` and `US` no longer count twice. **Drops made by the agent's kernel IP filter were not stored.** The agent reports them as `dropped_ip` records rather than block events, and those were skipped, so for an agent enforcing in the IP filter the access-rules views stayed empty. A drop the access rules caused is now stored as an access-rules block event, with its count as the hits and `bpf:ip_filter` as the rule. **Writing a managed WAF rule failed** with `column s.rule_managed_id does not exist`. **`--set entrypointHost` had no effect.** Following the [Publish the endpoints](./../06-publish.mdx) procedure verbatim published every route on the placeholder `gen0sec.internal` whatever address you passed. The overlay carried the host as a YAML anchor, and anchors are expanded when the file is parsed, before Helm merges `--set`, so the flag rewrote the top-level key while all ten rules kept the value the file shipped with. No template read the value either, so removing the anchor alone would not have helped. The failure was silent. The Ingress objects were created and looked correct, but Synapse keys its upstreams by host and serves nothing for an address no rule names. The chart now resolves the entrypoint itself and publishes every rule under it. **The dashboard's sign-in URL could be absent.** `BETTER_AUTH_URL` was derived from the first entry of a service's `ingress.hosts`. With the address named once in `entrypointHost` that list is empty, so the variable was not wrong but missing, and Better Auth rejected every sign-in with `Invalid origin`. It now follows the same entrypoint the rules do. ### Changed **An empty `entrypointHost` now fails the render** instead of publishing host-less rules. Such a rule matches every `Host` header and Synapse's operator skips it, so it could only ever be a misconfiguration that reported nothing. `helm upgrade` now stops and names the value to set. Rules that carry their own `ingress.hosts[].host` are unaffected and still need no `entrypointHost`. **`entrypointHost` is required from the first install**, before anything is published. The dashboard's sign-in URL now always comes from, in order: `services.ui.env.BETTER_AUTH_URL`, the dashboard's own `ingress.hosts[0].host`, then `entrypointHost`, whether or not its Ingress is enabled. With none of them the render fails and names `entrypointHost`. It was previously derived only once the dashboard's Ingress was enabled. [Install](./../05-install.mdx#7-install-the-platform) and [Upgrade](./../operate/upgrade.mdx#5-upgrade-the-platform) pass it. **`--set externalScheme` had no effect either.** Same defect, same shape: nothing read the top-level value, and the single-entrypoint overlay pinned the dashboard's own scheme on top of it, so the flag was inert twice over. It only ever looked correct because what it pinned was `https`, which is what a right answer produces anyway. The wrong scheme in the dashboard's base URL is rejected as `Invalid origin`, so this mattered to anyone terminating TLS upstream who needed anything else. The chart reads the top-level value now, and the overlay stops pinning the dashboard. `services..ingress.externalScheme` still overrides it for one service. A value that was inert becomes live. See the warning at the top before you upgrade. **The dashboard holds one namespaced permission.** The infra chart adds the Role and RoleBinding `g0s-ui-read-installed-versions` in `gen0sec-system`, allowing `get` on two named ConfigMaps, and the dashboard's service account token is now mounted. Nothing to do on the documented release names. Otherwise set `components.ui.*`; see [Upgrade](./../operate/upgrade.mdx#3-upgrade-infra) and [Privileges and RBAC](./../security/privileges.mdx). **The dashboard makes optional outbound calls**, none carrying agent data. See [Ports and egress](./../security/egress.mdx). **telemetry-api logs what it drops**, once a minute per organization, and its existing decode warnings are rate-limited the same way. See [Monitor](./../operate/monitor.mdx). **Rule changes made through the signal API are audited** in the same transaction. If the audit record cannot be written, the rule change is not applied. See [Symptoms](./../troubleshoot/symptoms.mdx#rule-changes-made-through-the-signal-api-never-appear). **The Agents item is hidden** from the dashboard navigation on self-hosted installs. **One migration waits at most 10 seconds for a lock** on the block events table. On a busy install it can fail and is safe to retry. See the note in [Upgrade](./../operate/upgrade.mdx#5-upgrade-the-platform). ### Security - gRPC 1.83.2, fixing an HTTP/2 memory exhaustion reachable in the APIs (GO-2026-6348), and OpenTelemetry 1.47.0. Other Go modules updated for advisories that were not reachable. - Next.js 16.3.7 and React 19.3.0, fixing CVE-2026-94545 as first shipped in v0.1.1. - better-auth 1.7.7, and the dashboard's dependency overrides enforced so `pnpm audit` is clean. ### Documentation - These docs are now versioned, with a version selector. v0.1.0 and v0.1.1 each keep their own version, with both defects above listed in their known limitations. - [Publish the endpoints](./../06-publish.mdx#if-tls-terminates-upstream) documents `--set externalScheme=https` again, now that it works. - [Upgrade](./../operate/upgrade.mdx) now passes the ingress overlay to the platform upgrade. Without it, the upgrade removes every Ingress. - [Roll back](./../operate/rollback.mdx) uses the `db-migrate` image of the release you are leaving. The previous image cannot run the down migrations. ## v0.1.1 A security patch for the dashboard. It fixes [CVE-2026-94545](https://nvd.nist.gov/vuln/detail/CVE-2026-94545) (medium, CVSS 4.0 score 5.3), in which Satori, the HTML-to-SVG library Next.js bundles for `next/og`, did not escape certain values in the SVG it generates. The dashboard does not render images through `next/og`, but the vulnerable copy shipped in its image and scanners flag it. The fix updates Next.js from 15.5.24 to 16.3.7, and React from 19.2.3 to 19.3.0. It upgrades in place from v0.1.0, with no migration and no values changes. Nothing else changed: the procedure is v0.1.0's, and its known limitations still apply. ## v0.1.0 First on-premises release. ### Installing Two methods, both documented on one procedure that branches once: [install from our registry](./../04-prepare-artifacts.mdx), or [create an offline bundle](./../04-prepare-artifacts.mdx) and mirror it into your own. ### Requires | | | |---|---| | Kubernetes | 1.23 or later | | Nodes | 3 for production, 16 vCPU and 32 GiB each recommended | | Storage | [sized to your retention](./../02-requirements.mdx#sizing-storage-for-your-retention) | | Cluster egress | `api.gen0sec.com:443`, required | ### Credentials you need **Two**, and the second is easy to miss. Both are self-service from the dashboard: 1. A registry token, for pulling charts and images. Created under **Repository Access**. The install kit itself is public and needs no credentials. 2. **Your Gen0Sec API key**, which the relay uses to fetch data from us. Without it the whole fleet receives empty artifacts with no error. See [Credentials from gen0sec](./../02-requirements.mdx#credentials-from-gen0sec). ### Not in this release | | | |---|---| | Zero-egress deployment | [Known limitations](./../troubleshoot/known-limitations.mdx#no-zero-egress-deployment) | | Explicit egress proxy for the relay | [Known limitations](./../troubleshoot/known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy) | | Single sign-on of any kind | Email and password only. Microsoft Entra ID is in development | | Shipped dashboards and alert rules | [Monitor](./../operate/monitor.mdx) lists what to watch meanwhile | | The threat service | Not shipped at this release | --- ## How we version | | | |---|---| | Scheme | Semantic versioning. Each minor is a release train | | Supported | The latest two minor trains | | Upgrades | Sequential minors, patches in place. **Skipping a minor is not supported** | | Release candidates | Not published to `releases.gen0sec.com`. Only stable releases are mirrored | Full policy: [Component versions](./../configure/components.mdx#support-policy). ## Finding a release ```bash curl -fsSL https://releases.gen0sec.com/api/repos/cerebellum/latest | jq -r .tag_name ``` Returns the latest full release. **Pin the tag explicitly for an install you want to reproduce**, so a rebuild in six months fetches the same bytes. ## What every release publishes | Artifact | Where | |---|---| | Install kit and its signature | `https://releases.gen0sec.com/cerebellum//` | | Images and charts | The registry, each pinned to a digest in the kit's release manifest | | Signatures and SBOM attestations | Alongside each image and chart, as OCI referrers | See [Supply chain](./../security/supply-chain.mdx). --- # Install kit contents Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/reference/kit # Install kit contents ## What you download Two files, published per release at `https://releases.gen0sec.com/cerebellum//`. | File | What | |---|---| | `cerebellum-install-kit.tar.gz` | The kit | | `cerebellum-install-kit.tar.gz.cosign.bundle` | Its detached signature | The release also publishes `latest-known-versions.json` there. You do not download it: the dashboard reads it only when its online update check is on. It is display data, not signed and not part of the kit. Verify before unpacking. See [Prepare the artifacts](./../04-prepare-artifacts.mdx#3-verify-the-signature). ## Layout ``` cerebellum-install-kit-/ ├── bundler.sh builds and loads the offline bundle ├── release-manifest.yaml every image and chart, pinned to a digest ├── scripts/ │ ├── preflight.sh cluster checks the charts assume but do not create │ ├── make-db-secrets.sh composes the ~20 application secrets │ ├── uninstall.sh the three releases, finalizers, namespaces │ └── verify-deployment.sh external smoke test, read-only ├── examples/ │ ├── single-node/ three overlays, one replica each │ └── single-entrypoint/ one address for the dashboard and every API ├── offline-data/ │ └── README.txt └── SHA256SUMS over every file above ``` **No images and no charts.** Those come from the registry, or from the offline bundle. ## The scripts | Script | Run it | Documented in | |---|---|---| | `preflight.sh` | Before installing | [Install step 1](./../05-install.mdx#1-check-the-cluster) | | `make-db-secrets.sh` | After the databases are ready, before the platform | [Install step 6](./../05-install.mdx#6-create-the-application-secrets) | | `verify-deployment.sh` | After every install and every upgrade | [Verify the install](./../08-verify.mdx#4-the-platform-serves-an-agent) | | `uninstall.sh` | When removing the installation | [Uninstall](./../operate/uninstall.mdx) | `verify-deployment.sh` reads two environment variables: ```bash export GEN0SEC_BASE_URL=https://cerebellum.example.internal export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) ``` ## The examples Everything under `examples/` is a starting point you are meant to read and edit. Each file carries comments marking which values are yours. | Directory | Purpose | |---|---| | `single-node/` | Three overlays that reduce every component to one node. [Quickstart](./../quickstart.mdx) | | `single-entrypoint/` | One address for the dashboard and every agent-facing API. [Publish the endpoints](./../06-publish.mdx) | ## Verifying the kit's own integrity ```bash cd $KIT && sha256sum -c SHA256SUMS ``` `SHA256SUMS` covers the kit **as delivered**. Adding files makes it no longer match, which is expected once you start using it. ## What the offline bundle method has to transfer Two things, and the order matters. | Transfer | Contains | |---|---| | The bundle tarball | Images, charts, values, the manifest, a copy of the bundler | | The kit directory | `scripts/`, `examples/`, and the manifest | The bundle carries its own copy of `bundler.sh`, so loading it needs no tooling from the kit. **The kit still has to cross**, because the install uses its scripts and its examples. Copy the bundle first, then clean up, then the kit. `create-package` runs inside the kit directory, so the bundle is sitting in it, and copying the kit first transfers tens of gigabytes twice. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). Both archives unpack side by side: ``` ~/gen0sec-cerebellum-multiarch-0.2.1/ the bundle ~/cerebellum-install-kit-0.2.1/ the kit ``` ## The offline-data directory The kit creates `offline-data/` containing a single `README.txt`, and nothing else. It is a placeholder for a data-delivery path that is not part of this release: the four datasets the platform needs are fetched at runtime through the relay, not delivered as files. See [Data relay](./../configure/relay.mdx). If you were told to populate it, check with us first. At this release you should not need to. --- # Requirements Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/requirements # Requirements ## Cluster | | Requirement | |---|---| | Kubernetes | 1.23 or later | | Access | `cluster-admin`, for the install only. See [Privileges and RBAC](./security/privileges.mdx) for what it is used for and what it leaves behind | | StorageClass | A default StorageClass on block storage. **Volume expansion recommended** | | Supported architecture | `linux/amd64` or `linux/arm64`. Mixed-architecture clusters are supported | | LoadBalancer | The ingress data plane is exposed as a `LoadBalancer` type Service by default. See the note below | :::note[The ingress needs to be reachable from outside the cluster] The ingress data plane is a `LoadBalancer` type Service, because nothing it programs can serve traffic if it stays cluster-internal. Your cluster needs a LoadBalancer implementation. If your cluster fronts ingress another way, a NodePort, a hardware balancer or an external proxy, change the Service type to match. ::: Cerebellum brings its own data plane - which contains Postgres, Kafka, object store and cache. You do not connect it to existing instances of any of them. :::info[Maintenance] We've deliberately chosen to use well-known solutions from the software industry in our data layer to ease the burden of maintenance. While Cerebellum ships this data layer by default, to make the setup easier, the data in it is the highest importance to you. So make sure you monitor the dataplane and do backups and maintenances regularly. ::: :::note[Container runtime, for the registry install method only] Our registry challenges pulls that carry no `User-Agent`, and on some distributions the kubelet's pull path sends none. containerd's own user agent passes; the problem is its absence. Either configure one on every node, or use the offline bundle method, which pulls into your own registry instead. Settle this before you choose a method. See [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: ### Kubernetes versions 1.23 is the floor because `autoscaling/v2` is the newest API the charts use, and it reached GA in that release. While anything from 1.23 and up should work, we recommend you use the latest stable version. ## Sizing a production install Production runs on **3 nodes**. Several data plane members each form a three-member quorum, so three is the floor for redundancy, not a suggestion. ### CPU & memory per node | | Resources | |---|---| | **Minimum** | 8 vCPU, 16 GiB | | **Recommended** | 16 vCPU, 32 GiB | ### Storage per node We recommend to start your nodes with `500 GiB` **minimum**. Several of the data plane members use PVCs. While the defaults are kept low - as your platform matures the data will grow and you'll need to expand the storage volumes. ### Sizing storage for your retention The shipped sizes are a starting point, not a capacity plan. They are deliberately modest, so for most production workloads you will be growing them rather than shrinking them. Four keys control everything: | Key | Chart | Default | |---|---|---| | `postgres.volume.size` | `gen0sec-data` | 50 GiB | | `kafka.storage.size` | `gen0sec-data` | 50 GiB | | `rustfs.storageclass.dataStorageSize` | `gen0sec-infra` | 50 GiB | | `rustfs.storageclass.logStorageSize` | `gen0sec-infra` | 10 GiB | Which one to grow depends on what you are retaining: | If you need | Grow | |---|---| | Longer event and telemetry history | `postgres.volume.size`. This is the system of record | | A longer Kafka retention window, or a bigger ingest buffer | `kafka.storage.size` | | More room for datasets and locally-produced artifacts | `rustfs.storageclass.dataStorageSize` | Example for longer retention: ```yaml # gen0sec-data postgres: volume: size: 200Gi kafka: storage: size: 200Gi ``` ```yaml # gen0sec-infra rustfs: storageclass: dataStorageSize: 100Gi ``` :::note[Plan for growth] Volume expansion grows a volume if your StorageClass supports it, and it is straightforward. Nothing shrinks one: reducing a size means recreating the volumes and losing what is on them. Starting at the shipped defaults and growing is therefore the recommended direction. It is the reason for low initial limits. Check that your StorageClass has `allowVolumeExpansion: true` before you install. ::: ### What is not on this page **Throughput.** How many agents, events per second or lookups a given topology sustains is not directly derivable from the charts' parameters. We suggest you scale the platform components and storage sizes according to your topology's specific needs. {/* Four workloads declare no CPU or memory requests: the object store, the Kafka brokers, valkey, and the Postgres operator. They are therefore BestEffort or Burstable and are evicted first under node pressure, which is why the per-node recommendation is driven by limits rather than by the requests total. Raised separately as a chart issue. */} ### Evaluating on one node Use the [Quickstart](./quickstart.mdx). It runs everything at a single replica on one node, for evaluation, training and upgrade rehearsals. ## Tools on your workstation Install these on the machine you run the commands from, not on the cluster. | Tool | Version | Used for | Install | |---|---|---|---| | `helm` | 3.16 or later | Installing the charts, and pulling them from the registry | [helm.sh](https://helm.sh/docs/intro/install/) | | `kubectl` | matching your cluster | Everything cluster-side | [kubernetes.io](https://kubernetes.io/docs/tasks/tools/#kubectl) | | `cosign` | 2 or later | Verifying signatures on the install kit, images and charts | [docs.sigstore.dev](https://docs.sigstore.dev/cosign/system_config/installation/) | | `skopeo` | any | Copying images between registries | [github.com/containers/skopeo](https://github.com/containers/skopeo/blob/main/install.md) | | `yq` | 4 or later, **mikefarah/yq only** | Reading the release manifest | [github.com/mikefarah/yq](https://github.com/mikefarah/yq#install) | | `jq` | any | Reading the release API | [jqlang.github.io](https://jqlang.github.io/jq/download/) | | `aws` or `mc` | any | Creating the object store bucket, one step during install | [aws-cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) or [MinIO client](https://min.io/docs/minio/linux/reference/minio-mc.html) | | `curl`, `tar`, `openssl`, `sha256sum` or `shasum` | any | Downloading, unpacking, generating secrets, checking checksums | Your OS packages | ### Check your yq There are two separate tools that are called `yq`. The Python one (`kislyuk/yq`) cannot parse the expressions the bundler uses, and it fails with a syntax error partway through. ```bash yq --version ``` The output must contain `mikefarah`. If it does not, install the Go version and point the bundler at it: ```bash export MIKEFARAH_YQ_PATH=/path/to/mikefarah-yq ``` ## Credentials from Gen0Sec You need **two**, and they are different things. Missing the second one produces a working-looking install whose entire fleet silently receives nothing. | Credential | Used by | Used for | |---|---|---| | **Registry token** | The bundler, on your workstation | Pulling charts and images from our registry | | **Your Gen0Sec API key** | The relay pod, inside your cluster | Fetching threat intelligence, GeoIP, models and IDS rules from Gen0Sec | Both are self-service from the dashboard. Neither requires contacting us. :::note[The install kit itself needs no credentials] The kit is published openly. Downloading and verifying it, in [Prepare the artifacts](./04-prepare-artifacts.mdx), happens before you use either credential — so if that download fails, the token is not what to check. The usual cause is a tag that is not published; the [release index](https://releases.gen0sec.com/cerebellum/) lists the ones that are. ::: ### The registry token Create it on the dashboard, under **Repository Access**: ``` https://app.gen0sec.com//repositoryAccess ``` Your workspace id is in the address bar once you are signed in, and the same page is linked from **Clients**. The token is shown **once**, at creation — store it before leaving the page, because it cannot be retrieved afterwards and a lost one has to be replaced. **The username is literally `any`.** Wherever a command asks for one, type `any`. Only the token is checked, so the username can be anything at all — `any` is simply what these docs use so the commands are copy-pasteable. ### Your Gen0Sec API key You can create your API key on the [SaaS Dashboard](https://app.gen0sec.com) Scopes required: - `ids_rules:read`, - `indicators:read`, - `ml_models:read` :::note[Why this one is separate from the registry token] Your agents get their API keys from your own dashboard, and those keys exist only in your cluster's database, so they cannot authenticate to Gen0Sec. The API key created in this step is used by your in-cluster relay to authenticate against the Gen0Sec SaaS, so it can fetch data related to threat intelligence, GeoIP, ML models and IDS rules. The chart expects it as a secret named `gen0sec-download-proxy`, and the relay pod will not start without it, so a missing key is a failed rollout rather than a silently degraded fleet. You will create this secret in the step 2 of [Install](./05-install.mdx). ::: --- # Architecture and data flows Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/security/architecture # Architecture and data flows Written for a security reviewer. Everything here is observable in a running install, and every claim has a page you can check it against. ## What runs where Two namespaces, three charts, one ingress point. ``` internet │ │ api.gen0sec.com:443 (the only required outbound path) │ ┌───────────────────────────┼──────────────────────────────────────────────┐ │ your cluster │ │ │ ▲ │ │ ┌─────────────────────── │ ─────────────────────────────────────────┐ │ │ │ namespace: gen0sec │ │ │ │ │ │ │ │ │ │ dashboard relay(download-api) 13 application services │ │ │ │ │ │ │ └───────────────────────────────┬───────────────────────────────────┘ │ │ │ in-cluster only │ │ ┌───────────────────────────────┼───────────────────────────────────┐ │ │ │ namespace: gen0sec-system │ │ │ │ │ ▼ │ │ │ │ Postgres (3) Kafka (3) object store (4) caches (2) │ │ │ │ Postgres operator Kafka operator ingress operator │ │ │ └───────────────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────┘ ▲ │ one address, TLS, path-prefix routed │ your agents and your users ``` | Namespace | Contains | Reachable from outside | |---|---|---| | `gen0sec` | The application services and the dashboard | Yes, through the ingress you publish | | `gen0sec-system` | Postgres, Kafka, object store, two caches, and the three operators | No | Nothing in `gen0sec-system` is published. The application reaches it by cluster DNS. ## Trust boundaries Four, and they are worth naming separately because they fail differently. | Boundary | Crossed by | Controlled by | |---|---|---| | **Internet to cluster** | The relay, outbound only, one destination | Your firewall. See [Ports and egress](./egress.mdx) | | **Your network to cluster** | Agents and users, inbound, one address | Your ingress and its TLS | | **Namespace to namespace** | The application reading its databases | Cluster DNS. No NetworkPolicy ships. See [Hardening](./hardening.mdx) | | **Cluster to registry** | Image pulls | The image pull secret, one per namespace | ## Data flows ### Inbound: agents and users | Flow | Path | Authentication | |---|---|---| | Agent telemetry and events | Your address, then the ingest services, then Kafka, then Postgres | Agent API key | | Agent configuration pull and push | Your address, then the configuration services | Agent API key | | Agent artifact download | Your address, then the relay, then Gen0Sec | Agent API key, validated locally. The relay uses your installation's own key upstream, so agent keys never leave. See [Data handling](./data-handling.mdx) | | Dashboard sign-in | Your address, then the dashboard, then Postgres | Email and password, or a passkey, held in your Postgres. No external identity provider at this release | ### Outbound: the relay, plus optional dashboard calls The dashboard can also check for new versions and send email. All of it is optional and carries no agent data. See [Data handling](./data-handling.mdx#the-dashboards-optional-connections). The relay has one destination and one purpose: fetching four licensed datasets that are in no artifact we ship. | Dataset | Why it cannot be shipped | |---|---| | CTI indicators | Licensed by upstream providers | | GeoIP databases | Licensed by MaxMind and IPInfo | | ML models | Published by gen0sec, refreshed independently of releases | | IDS rules | Licensed as part of the Emerging Threats ruleset | The relay is a **relay, not a cache**. Nothing is stored on the way through, so there is no local artifact store to size and no staleness window. ### Never outbound | Dataset | What it is | Where it stays | |---|---|---| | `identity` | Workload identity produced in your cluster | Your object store | | `policy-edges` | Pod IPs, namespaces, NetworkPolicy edges | Your object store | Relaying either would push a map of your workload topology to a service you do not run. It is not configurable and there is no flag that enables it. Uploads are not relayed either: the relay is read-only. ## Where data is stored | Store | Contains | Backed up by you | |---|---|---| | Postgres | The system of record: configuration, agents, policy, history | Yes, always | | Object store | Datasets, and the two locally-produced sets above | Yes | | Kafka | In-flight events between ingest and persistence | No, it is transport | | Cache | Nothing durable | No | See [Back up and restore](./../operate/backup.mdx). ## What the platform requires of your cluster | Requirement | Detail | |---|---| | `cluster-admin` during install | For CRDs and cluster-scoped RBAC. See [Privileges and RBAC](./privileges.mdx) | | Custom resource definitions | Postgres and Kafka operator CRDs, cluster-scoped by nature | | Its own ingress controller | Installed by the infra chart, `ingressClassName: synapse` | | One required outbound destination | `api.gen0sec.com:443` | | No node access, no host mounts, no privileged containers | See [Hardening](./hardening.mdx) | ## Questions a reviewer usually asks next | Question | Page | |---|---| | What exactly can leave the cluster? | [Data handling](./data-handling.mdx) | | What ports and destinations do I open? | [Ports and egress](./egress.mdx) | | Why does it need `cluster-admin`? | [Privileges and RBAC](./privileges.mdx) | | How do I verify what you shipped me? | [Supply chain](./supply-chain.mdx) | | What is the container security posture? | [Hardening](./hardening.mdx) | | What does not work yet? | [Known limitations](./../troubleshoot/known-limitations.mdx) | --- # Data handling Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/security/data-handling # Data handling ## What leaves your cluster One required connection, outbound, to one destination. The dashboard can make three optional ones, listed after it. | | | |---|---| | Destination | `api.gen0sec.com:443` | | Made by | The relay pod only | | Direction | Outbound, request and response | | Purpose | Fetching four licensed datasets | **What is in the request.** A path naming the artifact, and your own Gen0Sec API key. Nothing else. The relay does not add a customer identifier, a cluster identifier, an agent identifier, telemetry or usage data. **What is in the response.** The artifact: a CTI indicator database, a GeoIP database, an ML model, or an IDS rule bundle. ### The dashboard's optional connections | Destination | When | What is sent | |---|---|---| | `helm.gen0sec.com:443` | Opening the Clients or Components page | A request for the public chart index. Nothing about your install | | `releases.gen0sec.com:443` | Only if **Check for updates online** is on | A request for the latest release's public version list. Nothing about your install | | Your SMTP server | Only if you configure SMTP | The recipient's address, your organization's name, and an invitation or password-reset link | All three fail without affecting anything else. See [Ports and egress](./egress.mdx). ## What never leaves your cluster | Data | What it is | |---|---| | `identity` | Workload identity produced inside your cluster | | `policy-edges` | Pod IP addresses, namespaces, NetworkPolicy edges | | Agent telemetry and events | Everything your agents report | | Configuration and policy | Everything you configure | | Logs | All of it, to container stdout in your cluster | | Database contents | Postgres never leaves | | Dashboard user accounts, passwords and passkeys | Held in your Postgres | | Audit log | Who changed what, in the dashboard or through the API: the user or API key, the action, the changed fields with secret-looking fields redacted, source IP and user agent | `identity` and `policy-edges` deserve the explicit statement, because they are the two that are technically served by the same service that does the relaying. **They are never relayed.** Doing so would push a map of your workload topology to a service you do not run. It is not configurable, and no flag enables it. The upload surface is not relayed either, because the relay is read-only. ## Who validates agent credentials **Your agents' keys are validated in your cluster, and they never leave it.** Every relayed route sits behind the local authentication middleware, and each one carries the same permission it carries in direct mode. So a key that would be refused when served locally is refused it when relayed. ``` agent --[agent key, authorized locally, permission checked per route]--> relay --[your Gen0Sec API key]--> api.gen0sec.com agent key stops here ``` Two separate concerns, deliberately separated: | Hop | Credential | Checked by | |---|---|---| | Agent to relay | The agent's own key | Your cluster, against your installation, with the artifact's permission | | Relay to Gen0Sec | Your Gen0Sec API key | Gen0Sec | | | | |---|---| | Credentials Gen0Sec receives | One, belonging to your installation | | Credentials Gen0Sec never receives | Your individual agent key | | What Gen0Sec can infer about your fleet | Nothing from the credential. Requests carry no agent or cluster identifier | | Revoking one agent | Local, in your dashboard, immediate | | Revoking your installation's upstream access | Revoke or rotate your Gen0Sec API key | This is the answer to "does our agent inventory leave our perimeter". It does not. :::note[Agent keys are never forwarded upstream, even by accident] On a self-hosted deployment the relay never puts a caller's key in an outbound request, in any configuration. A locally-issued key could not authenticate to Gen0Sec anyway, so forwarding it could only leak a local credential to a third party without making the request succeed. With no Gen0Sec API key configured the relay sends no credential at all rather than sending yours. ::: ## Credentials at rest | Credential | Where it lives | Generated by | |---|---|---| | Database role passwords | Kubernetes secrets | The Postgres operator | | Object store keys | A Kubernetes secret | You, during install | | Registry pull credentials | A Kubernetes secret per namespace | You, during install | | Dashboard session signing key | A Kubernetes secret | Generated once, preserved across regeneration | | SMTP credentials | A Kubernetes secret | You, optional | **No credential appears in a chart, a values file or a rendered manifest.** Every one is referenced by name. See [Secrets](./../configure/secrets.mdx). The install procedure reads every credential from a file or from standard input rather than passing it as a command-line argument, so nothing lands in shell history or in the process list. ## Credentials in transit | Hop | Encrypted | |---|---| | Agents and users to your ingress | Yes, TLS you terminate | | Relay to `api.gen0sec.com` | Yes, TLS | | Services to Postgres | Yes, TLS to the pooler | | Services to the object store | Plain HTTP, in-cluster only | | Services to the cache | Plain, in-cluster only, no credential in the URL | The last two are in-cluster traffic on the pod network. If your threat model includes an attacker on the pod network, add NetworkPolicies and a service mesh. See [Hardening](./hardening.mdx). :::note[One deliberate exception, and why] The dashboard's database connection verifies encryption without verifying the certificate chain, because its client library rejects the pooler's self-signed certificate outright while every other service's library accepts encrypt-without-verify. The traffic is encrypted and in-cluster. Do not copy that setting onto the other services: their driver rejects it. ::: ## Data retention Retention is yours. Nothing expires data on your behalf, and nothing is sent anywhere before it is deleted. | Store | Grows with | Sized by | |---|---|---| | Postgres | Agents, events, history | `postgres.volume.size` | | Postgres audit log | Administrative changes, kept indefinitely | Small. See [Known limitations](./../troubleshoot/known-limitations.mdx#the-audit-log-is-never-pruned) | | Kafka | In-flight events, and its retention window | `kafka.storage.size` | | Object store | Datasets and locally-produced artifacts | `rustfs.storageclass.dataStorageSize` | See [Sizing](./../02-requirements.mdx#sizing-a-production-install). ## Deleting everything [Uninstall](./../operate/uninstall.mdx) without `--keep-data` deletes the namespaces and every persistent volume in them: the database, the message log and the object store. There is no copy anywhere else. --- # Ports and egress Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/security/egress # Ports and egress Same as [Network and connectivity](./../03-network.mdx), reproduced here for better visibility. **One canonical source.** If this table and the install page ever disagree, that is a bug in our publishing, not a judgement call for you. Report it. ## Outbound All TCP, all TLS, all outbound. Nothing needs an inbound rule from the internet. | Destination | Port | Needed by | Purpose | Required | |---|---|---|---|---| | `api.gen0sec.com` | 443 | Cluster pods, the relay only | Threat intelligence, GeoIP, ML models, IDS rules | **Always** | | `releases.gen0sec.com` | 443 | Workstation | Download the install kit and its signature | **Always** | | `registry.gen0sec.com` | 443 | Workstation | Pull charts and images | **Always** | | `registry.gen0sec.com` | 443 | Cluster nodes | Image pulls at install time | Registry install only | | `fulcio.sigstore.dev` | 443 | Workstation | Signature certificate verification | **Always** | | `rekor.sigstore.dev` | 443 | Workstation | Transparency log verification | **Always** | | `tuf-repo-cdn.sigstore.dev` | 443 | Workstation | Signature trust root refresh | **Always** | | Your own registry | 443 or 80 | Cluster nodes, and the host that loads the bundle | Image pulls at install time | Offline bundle only | | `helm.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest agent version, shown on the Clients and Components pages | No. Always attempted. Blocking it only hides the latest version | | `releases.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest component versions | No. Only if an administrator turns on **Check for updates online** | | Your SMTP server | 587 or 465 | Cluster pods, the dashboard only | Invitation and password-reset email | No. Only if you [configure SMTP](./../configure/identity.mdx#invitations-and-email) | **Only one row is required by the running cluster:** `api.gen0sec.com:443`, and only the relay pod uses it. The dashboard rows are optional: blocking them hides version information, or leaves invitations and password resets without email. Everything else is install-time, and most of it is your workstation rather than your cluster. ## Inbound | Source | Destination | Port | Purpose | |---|---|---|---| | Your agents | Your published address | 443 | Telemetry, events, configuration, artifact download | | Your users | Your published address | 443 | Dashboard | One address serves both, routed by path prefix. See [Publish the endpoints](./../06-publish.mdx#the-route-table). ## In-cluster No NetworkPolicy is required, and none ships. If you enforce default-deny, these are the flows to allow. | From | To | Port | |---|---|---| | `gen0sec` services | `core-pooler.gen0sec-system` (Postgres pooler) | 5432 | | Migration job | `core.gen0sec-system` (Postgres primary) | 5432 | | `gen0sec` services | `core-kafka-bootstrap.gen0sec-system` | 9092 | | `gen0sec` services | `rustfs-svc.gen0sec-system` (object store) | 9000 | | `gen0sec` services | `dragonfly.gen0sec-system` (cache) | 6379 | | Ingress controller | `gen0sec` services | Service ports | | `gen0sec` services | `gen0sec` services | Service ports | | Dashboard (`ui`) pods | The Kubernetes API server | 443 or 6443 | The dashboard reads two ConfigMaps from the API server for the Components tab. Find its address with `kubectl get endpoints kubernetes -n default`. Without that flow only the Components tab degrades. See [Hardening](./hardening.mdx) for what we recommend beyond that. ## Notes that save time **The Sigstore hosts are the ones people forget.** Signature verification is the first command of the install, and it fails without them. They are needed on the workstation, not on the cluster. **A proxy does not help for the cluster's one destination.** Explicit forward proxies are not honoured in-cluster. See [Known limitations](./../troubleshoot/known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy). **The offline bundle removes registry egress, not all egress.** The runtime connection stays. See [What this rules out](./../03-network.mdx#what-this-rules-out). --- # Hardening Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/security/hardening # Hardening Two lists: what the charts already do, and what is left to you. The second list is short but it is not empty, and we would rather name it than let you assume otherwise. ## Hardened by default Every application service runs with this security context, and it is not opt-in. ```yaml podSecurityContext: runAsNonRoot: true runAsUser: 65532 runAsGroup: 65532 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: [ALL] ``` | Control | Status | |---|---| | Runs as non-root | Yes, uid 65532 | | Read-only root filesystem | Yes | | Privilege escalation | Disabled | | Linux capabilities | All dropped | | Privileged containers | None | | Host network, PID or IPC | None | | Host path mounts | None | | Node access | None | :::note[Read-only root filesystems were not free] Several services defaulted to writing into a relative path under their working directory, which fails outright on a read-only root. Each one now gets an explicit writable volume mounted where it expects to write. That is why the values file carries per-service volume mounts: they are the cost of this control, not decoration. If you add a service-level override that removes one, that service will fail to start. ::: ## Pod Security Admission The application pods satisfy the **restricted** profile. Label the namespace to enforce it: ```bash kubectl label namespace gen0sec \ pod-security.kubernetes.io/enforce=restricted \ pod-security.kubernetes.io/enforce-version=latest ``` **Do not label `gen0sec-system` restricted yet.** The Postgres and Kafka pods there are created by upstream operators and have not been assessed against that profile. Use `baseline` if you need something, and tell us if you require restricted there. ## What you should add None of these ship, and all of them are worth doing. ### NetworkPolicies **No NetworkPolicy ships with the charts.** On a cluster without default-deny, that means the platform namespaces are as open as everything else. Add default-deny in both namespaces, then allow exactly these: | From | To | Port | |---|---|---| | `gen0sec` services | `core-pooler.gen0sec-system` | 5432 | | Migration job | `core.gen0sec-system` | 5432 | | `gen0sec` services | `core-kafka-bootstrap.gen0sec-system` | 9092 | | `gen0sec` services | `rustfs-svc.gen0sec-system` | 9000 | | `gen0sec` services | `dragonfly.gen0sec-system` | 6379 | | Ingress controller | `gen0sec` services | Service ports | | `gen0sec` services | `gen0sec` services | Service ports | | Dashboard (`ui`) pods | The Kubernetes API server | 443 or 6443 | | Relay pod | `api.gen0sec.com:443` | 443, egress | **Egress default-deny is the high-value one here.** The relay is meant to be the only pod that needs a route out. The dashboard's optional egress is listed on [Ports and egress](./egress.mdx). A NetworkPolicy is what turns that from a design intention into an enforced property, and it is the single strongest control you can add to this deployment. ### TLS everywhere it is not already | Hop | Today | Worth adding | |---|---|---| | Ingress to users and agents | TLS you terminate | Already yours | | Relay to Gen0Sec | TLS | Nothing to do | | Services to Postgres | TLS to the pooler | Nothing to do | | Services to the object store | Plain HTTP, in-cluster | A service mesh, if your threat model includes the pod network | | Services to the cache | Plain, in-cluster, no credential in the URL | Enable cache authentication, then move the connection string into each service's secret | ### Resource limits on the four workloads that have none The object store, the Kafka brokers, the ingress cache and the Postgres operator declare no CPU or memory requests. That makes them BestEffort or Burstable, so the kubelet evicts them first under node pressure. For the object store, holding your data, that is the wrong priority order. Set requests on them, or set a namespace `LimitRange` so nothing lands unbounded. See [Sizing](./../02-requirements.mdx#sizing-a-production-install). ### Secrets from an external manager `make-db-secrets.sh` is fine for evaluation. In production, have your secret manager produce secrets of the same shape so credentials never exist as plain Kubernetes secrets you created by hand. Every name and key is listed in [Secrets](./../configure/secrets.mdx). ## Settings you should never enable Two environment variables exist and both disable a security control. Neither is set by the charts, and neither appears anywhere in the install procedure. | Setting | What it does | When it is acceptable | |---|---|---| | `TLS_INSECURE_SKIP_VERIFY` | Stops the relay verifying the upstream certificate | Never in production. It defeats the protection on your only outbound connection | | `MIRROR_TLS_VERIFY=false` | Stops the bundler verifying your registry's certificate when pushing | Only for a plain-HTTP registry on a trusted network, and preferably not then | If you find either one set in a running deployment, treat it as a finding. ## Verifying the posture Confirm no pod runs as root or privileged: ```bash kubectl -n gen0sec get pods -o json | jq -r ' .items[] | .metadata.name + " runAsNonRoot=" + ((.spec.securityContext.runAsNonRoot // false) | tostring)' ``` Confirm nothing holds cluster-admin: ```bash kubectl get clusterrolebinding -o json \ | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | .metadata.name' ``` Confirm only the relay can reach the internet, once you have added egress policy: ```bash kubectl -n gen0sec run netcheck --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/busybox:1.36.1-glibc -- wget -T5 -qO- https://example.com ; echo "exit=$?" ``` A non-zero exit is the result you want. --- # Privileges and RBAC Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/security/privileges # Privileges and RBAC ## The short answer `cluster-admin` is required **to install**, not to run. It is needed for two things that are cluster-scoped by nature: 1. Installing custom resource definitions. 2. Creating the cluster roles the three operators need. Once installed, no component in the platform holds cluster-admin. ## What is cluster-scoped ### Custom resource definitions From the bundled operators, not written by us. | CRD group | Installed by | Kinds | |---|---|---| | `acid.zalan.do` | Postgres operator | `postgresqls`, `operatorconfigurations`, `postgresteams` | | `kafka.strimzi.io` | Kafka operator | `kafkas`, `kafkatopics`, `kafkanodepools`, `strimzipodsets`, and the Connect and Bridge kinds we do not use | CRDs are cluster-scoped in Kubernetes. There is no namespaced alternative. :::note[Deleting a CRD deletes every resource of that kind, cluster-wide] Including any outside these two namespaces. If anything else in your cluster runs a Postgres or Kafka operator, removing these CRDs would take its databases and clusters with them. [Uninstall](./../operate/uninstall.mdx) removes them by default, but checks first: if any of those kinds is in use outside the two namespaces it is removing, it skips the CRDs and says so. `--keep-crds` skips them unconditionally. ::: ### Cluster roles | Cluster role | Belongs to | What it is for | |---|---|---| | `g0s-postgres-operator` | Postgres operator | Managing Postgres clusters and their pods | | `g0s-postgres-operator-endpoints` | Postgres operator | The `endpoints` permission its own upstream role omits, while still creating a master Endpoints object per cluster | | `postgres-pod` | Postgres pods | What a database pod itself needs | | `strimzi-cluster-operator-global` | Kafka operator | Cluster-scoped reconciliation | | `strimzi-cluster-operator-namespaced` | Kafka operator | Namespaced resources, bound per namespace | | `strimzi-cluster-operator-leader-election` | Kafka operator | Leader election | | `strimzi-cluster-operator-watched` | Kafka operator | Watching the namespaces it manages | | `strimzi-kafka-broker` | Kafka brokers | What a broker pod needs | | `strimzi-entity-operator` | Topic and user operator | Managing topics and users | | `strimzi-kafka-client` | Kafka clients | Client-side permissions | | `g0s-synapse-operator` | Ingress operator | Watching Ingress objects and services | Eleven, and nine of them come straight from the two upstream operators. :::note[The one we added, and why] `g0s-postgres-operator-endpoints` exists because the Postgres operator ships a cluster role without `endpoints`, yet creates a master Endpoints object for every cluster it manages. Without that rule, `postgresql/core` fails with `endpoints is forbidden` and the database never starts. See [Postgres will not start](./../troubleshoot/symptoms.mdx#postgres-will-not-start). ::: ## What the application itself holds The application services in `gen0sec` are not operators and do not watch the cluster. | | | |---|---| | Cluster-scoped permissions | None | | Namespaced permissions | None, except the dashboard (see below) | | Service accounts | One per service. Tokens are not mounted, except the dashboard's | | Node access | None | | Host mounts | None | | Privileged containers | None | :::note[The one exception: the dashboard reads two ConfigMaps] The `ui` service account (`g0s-ui` in `gen0sec`) is bound to the Role `g0s-ui-read-installed-versions` in `gen0sec-system`, which the infra chart creates. It allows `get` on two ConfigMaps by name, `g0s-infra-installed-versions` and `g0s-data-installed-versions`, and nothing else. There is no `list` or `watch`, so it cannot enumerate other ConfigMaps. The dashboard uses it for the [Components](./../configure/components.mdx) tab, and is the only pod with its service account token mounted. ::: See [Hardening](./hardening.mdx) for the container security context. ## Can I install without cluster-admin? Partly, if you split the install between two roles. This is a supported pattern but it is not the documented golden path, so plan for one extra iteration. | Step | Needs | Who | |---|---|---| | Apply CRDs | Cluster-scoped write on `customresourcedefinitions` | Your platform team, once per release | | Create cluster roles and bindings | Cluster-scoped RBAC write | Your platform team, once | | Install the three charts | Namespace-admin on both namespaces | The installer | Render what a chart would create, and hand that to whoever holds the privilege: ```bash helm template g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system -f $B/values/gen0sec-infra-values-onprem.yaml \ | yq eval-all 'select(.kind == "ClusterRole" or .kind == "ClusterRoleBinding")' ``` ## Verifying after install Confirm nothing holds more than it should: ```bash kubectl get clusterrolebinding -o json \ | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | .metadata.name + " -> " + ([.subjects[]?.name] | join(","))' ``` No entry in that output should reference a `gen0sec` service account. If one does, tell us: that is a defect, not a configuration choice. ## What an uninstall leaves behind | | Removed | Reason | |---|---|---| | Namespaced resources | Yes | Helm owns them | | Objects the operators created outside the releases | Yes | The script removes them; `helm uninstall` alone does not | | CRDs | Yes, unless in use elsewhere or `--keep-crds` | Removal is cluster-wide destructive, so the script checks before it acts | | Cluster roles and bindings | Depends on the chart that created them | Check what remains and remove by hand if nothing else uses the operators | [Uninstall](./../operate/uninstall.mdx) reports what it removed and what it left. --- # Provenance, signatures and SBOM Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/security/supply-chain # Provenance, signatures and SBOM Everything we publish is signed, and every signature is verifiable by you without asking us for a key. This page also includes the one case where verification cannot follow the artifacts. ## What is signed, and how | Artifact | Signed | Attested | |---|---|---| | Install kit tarball | Detached cosign bundle, published beside it | | | Container images | `cosign sign --recursive`, so the index **and** each per-architecture manifest | SPDX JSON SBOM, one per architecture | | Helm charts | `cosign sign` on the OCI chart artifact | SPDX JSON SBOM | **Keyless, with GitHub OIDC.** There is no long-lived private key to steal, and no public key for you to fetch and trust. Each signature is bound to a workflow identity recorded in a public transparency log, so verification asserts *who built it*, not *who holds a key*. SBOMs are generated with syft and attached as SPDX JSON attestations. :::note[SLSA provenance is deliberately off] BuildKit's provenance attestations add manifests to the multi-architecture index carrying a placeholder platform. That breaks tools which walk an index by platform, including our own mirroring. We ship the SPDX SBOM attestation instead, which is the thing customers actually verify. If you require SLSA provenance, tell us: it is a build-flag change plus mirroring work, not a redesign. ::: ## Verify the install kit Do this before you unpack it. It is step 3 of [Prepare the artifacts](./../04-prepare-artifacts.mdx). ```bash cosign verify-blob \ --bundle cerebellum-install-kit.tar.gz.cosign.bundle \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/release\.yaml@refs/tags/' \ cerebellum-install-kit.tar.gz ``` **Both certificate flags are required.** `cosign` refuses to verify without an expected issuer and an expected identity, so a shortened command fails rather than passing weakly. The identity is pinned to **one workflow, on a tag reference**. A looser pattern would accept a signature produced by any workflow, on any branch, in our organization. That is the difference between "built by our release pipeline from a tag" and "built by somebody at Gen0Sec". ## Verify an image or a chart Against our registry, or against yours after mirroring: ```bash cosign verify \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/' \ registry.gen0sec.com/gen0sec/auth-api:0.2.1 ``` ## Read the SBOM ```bash cosign download attestation \ --predicate-type https://spdx.dev/Document \ registry.gen0sec.com/gen0sec/auth-api:0.2.1 \ | jq -r '.payload | @base64d | fromjson | .predicate' ``` The predicate is an SPDX JSON document. Feed it to whatever consumes SBOMs in your pipeline. ## Verification across the air gap This is the part nobody else documents, so it gets its own section. ### What crosses, and what does not Signature verification needs the public transparency log, and an isolated network does not have one. So the bundler verifies **once, on the connected staging host, before packaging**, and protects everything downstream with checksums. ``` connected staging host isolated network ────────────────────── ──────────────── pull every artifact by digest cosign verify each one (cannot re-verify: no transparency log) write SHA256SUMS -> verify-package re-checks SHA256SUMS print the bundle fingerprint -> mirror re-checks it before pushing package ``` | Stage | Guarantee | |---|---| | Before packaging | Full cryptographic signature verification, every image and every chart | | The tarball | `SHA256SUMS` over every file, plus a **fingerprint**, which is a sha256 of that list | | On the isolated side | `verify-package` and `mirror` both re-check `SHA256SUMS` and refuse to proceed on a mismatch | **Record the fingerprint** that verification prints on the connected side, out of band. Comparing it on the isolated side is what closes the loop between the two networks. :::note[Signature re-verification inside the isolated network is not supported] It would need the transparency log, or a pinned trust bundle carried across and kept current. We do not pretend otherwise. If your policy requires cryptographic verification at the destination rather than checksum verification, tell us. That is a different design, and we would rather discuss it now than have you find the gap during an audit. ::: ### The one case where attestations are lost `--arch amd64` or `--arch arm64` halves the transfer. It also extracts **one manifest out of each multi-architecture index**, which changes every image digest by definition. | | Multi-architecture bundle (default) | Single-architecture bundle | |---|---|---| | Image signatures | Verify | **Verify.** Each per-architecture manifest is signed individually | | SBOM attestations | Verify | **Lost.** They hang off the index, and the index is what is left behind | | Digests match the release manifest | Yes | No, by definition | The bundler warns at the time and annotates the bundle rather than letting this be a silent difference. **If your acceptance process checks attestations, stay on the default.** ## Digest pinning `release-manifest.yaml` in the install kit pins **every image and chart to a digest**, not to a tag. Tags move. Digests do not. That is what makes a release reproducible: the bundler resolves nothing at build time, it fetches exactly the digests the manifest names, and it fails if one is missing. ```bash yq '.spec.images[] | [.image, .imageTag, .digest] | @tsv' $KIT/release-manifest.yaml ``` ## If verification fails | Symptom | Cause | |---|---| | `no matching signatures` | Wrong identity pattern, or the artifact is not ours | | Certificate fetch errors | The Sigstore hosts are blocked. See [Network and connectivity](./../03-network.mdx#egress-list) | | The image verifies, the SBOM does not | Single-architecture bundle. Expected, see above | | `SHA256SUMS` mismatch on the isolated side | The transfer altered the bundle. Re-transfer, do not proceed | **A failed verification is not a step to skip.** The bundler has a flag that drops the check, and a bundle built with it is annotated as unverified. Use it only if cosign genuinely cannot be installed on your staging host, and record that you did. --- # Collect diagnostics Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/troubleshoot/diagnostics # Collect diagnostics Collect this before you contact us. Everything here is something we would otherwise ask for, so sending it means the first reply is an answer rather than a request for more output. There is no single collection command at this release. This page is the list. ## Set up Run the commands from a directory you can archive afterwards. ```bash NS_SYS=gen0sec-system NS_APP=gen0sec OUT=gen0sec-diagnostics mkdir -p $OUT && cd $OUT ``` Cluster and storage: ```bash kubectl version -o yaml > cluster-version.yaml kubectl get nodes -o wide > nodes.txt kubectl describe nodes > nodes-describe.txt kubectl get storageclass -o yaml > storageclasses.yaml kubectl get pvc -A -o wide > pvc.txt ``` Releases and workloads: ```bash helm list -A > helm-releases.txt for ns in $NS_SYS $NS_APP; do kubectl -n $ns get all -o wide > workloads-$ns.txt kubectl -n $ns get events --sort-by=.lastTimestamp > events-$ns.txt kubectl -n $ns get pods -o yaml > pods-$ns.yaml done ``` Logs from anything not ready: ```bash for ns in $NS_SYS $NS_APP; do for p in $(kubectl -n $ns get pods -o json \ | jq -r '.items[] | select(any(.status.containerStatuses[]?; .ready == false)) | .metadata.name'); do kubectl -n $ns logs $p --all-containers --tail=2000 > logs-$ns-$p.txt 2>&1 kubectl -n $ns logs $p --all-containers --previous --tail=2000 > logs-$ns-$p-previous.txt 2>&1 done done ``` Operators and custom resources: ```bash kubectl -n $NS_SYS logs deploy/g0s-infra-postgres-operator --tail=2000 > logs-postgres-operator.txt kubectl -n $NS_SYS logs deploy/strimzi-cluster-operator --tail=2000 > logs-kafka-operator.txt kubectl -n $NS_SYS logs deploy/g0s-infra-synapse-operator --tail=2000 > logs-ingress-operator.txt kubectl -n $NS_SYS get postgresql,kafka,kafkanodepool,kafkatopic -o yaml > custom-resources.yaml ``` The migration job, which is the most common single cause: ```bash kubectl -n $NS_APP logs job/g0s-db-migrate --tail=2000 > logs-db-migrate.txt 2>&1 ``` Images actually in use, including init containers: ```bash for ns in $NS_SYS $NS_APP; do kubectl -n $ns get pods -o yaml | grep -E '^\s+image:' | sort -u > images-$ns.txt done ``` Routing and the relay: ```bash kubectl -n $NS_APP get ingress -o yaml > ingress.yaml kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' > relay-healthz.json 2>&1 ``` Then pack it: ```bash cd .. && tar -czf gen0sec-diagnostics-$(date +%Y%m%d-%H%M%S).tar.gz gen0sec-diagnostics ``` :::danger[Read the archive before you send it] It is plain text and it is yours. Nothing here redacts anything, so check it, and remove any sensitive data it might have, particularly if you added `helm get values`, which prints whatever you set. None of the commands above read secret contents, database rows or object store objects. Logs can contain request paths, key prefixes, up to 400 bytes of a telemetry body that failed to decode, and the full content of any block event the pipeline parked. ::: :::note[There should be one command for this] A single `collect-diagnostics.sh` in the install kit would replace this page with two lines, and the list above is the specification for it. A single diagnostics tool can accidentally collect data that's sensitive and may fail to redact that information, so that's why we ask you to assemble your own report. Once we're 100% sure we can provide a diagnostics tool that doesn't leak any information of your sensitive data, we'll do so. ::: ## What to tell us alongside it Four lines, and they save a round trip: 1. Which install method you used. 2. The release version. 3. The last command you ran, verbatim. 4. What you expected to happen. --- # Known limitations in this release Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/troubleshoot/known-limitations # Known limitations If you hit something that is not on this page, we'd like to hear about it! ## No zero-egress deployment **What it means.** Your cluster must reach `api.gen0sec.com` on port 443. That connection carries threat intelligence, GeoIP, ML models and IDS rules, which are licensed upstream data that is in no artifact we ship. **What you can remove.** All dependency on Gen0Sec-hosted registries. The [offline bundle method](./../04-prepare-artifacts.mdx) mirrors every image and chart into your own registry, so your cluster pulls nothing from us. **What you cannot remove.** The runtime connection. There is no supported configuration with zero cluster egress. **If your policy forbids all egress**, please contact us. See [Network and connectivity](./../03-network.mdx). ## Our registry challenges pulls that send no user agent **By design on our side, and it can decide which install method you use.** **Symptom.** Every image pull from `registry.gen0sec.com` returns **403 with a CAPTCHA challenge**, and pods sit in `ImagePullBackOff`. A manual `docker pull` or `skopeo copy` from the same node **succeeds**. That discrepancy is the whole diagnosis. If the CLI works and the kubelet does not, this is it. **Cause.** On some Kubernetes distributions, the component that services the kubelet's pull sets no `User-Agent` header. Go's `net/http` then sends its own default, `Go-http-client/1.1`, and our edge treats that as a scanner signature and issues a challenge. A kubelet cannot solve a CAPTCHA, so the pull never succeeds and never will. It is not your credentials, not the pull secret, not TLS, and not an allowlist. ### What passes and what does not Measured against our registry: | User-Agent | Result | |---|---| | `containerd/1.7.0` | 200 | | `containerd/v2.1.4` | 200 | | `docker/29.4.0` | 200 | | `curl/8.7.1` | 200 | | `skopeo/1.24.0` | 200 | | `Go-http-client/1.1` | **403, challenged** | | `Go-http-client/2.0` | **403, challenged** | **containerd's own user agent is fine.** Any real one is. The failure is specifically the absence of a user agent, which Go turns into `Go-http-client`. ### Confirming it The blocked response identifies what it matched: ``` "user_agent": "Go-http-client/1.1", "ja4_fingerprint": null, "captcha_validated": false ``` Compare a manual pull against what the kubelet gets: ```bash # on the node: succeeds, because the CLI sends its own user agent skopeo inspect docker://registry.gen0sec.com/gen0sec/auth-api:0.2.1 ``` ```bash # what the kubelet sees kubectl -n gen0sec describe pod | grep -A3 Failed ``` Success from the CLI plus 403 from the kubelet confirms it. ### Two ways forward **Option 1: use the offline bundle method.** Nothing to configure on your nodes, and your own registry does not inspect user agents. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). **Option 2: make the pull path send a user agent**, on every node. :::note[Settle this before you choose an install method] If you cannot change container runtime configuration on your nodes, whether by policy or because your distribution manages it, use the offline bundle method. This constraint forces that choice, and it is better made before you start than at the first failed pull. ::: ## The relay cannot use an explicit egress proxy **Symptom.** You set `HTTPS_PROXY` on the relay. Nothing changes, and nothing errors. **Cause.** Every HTTP client in the platform builds its own transport without proxy support, so `HTTP_PROXY` and `HTTPS_PROXY` are ignored inside the cluster. This is by design. **What works instead.** | Option | Notes | |---|---| | Allow `api.gen0sec.com:443` directly | One firewall exception that bypasses the proxy | | Redirect transparently at the network layer | The client does not need to be proxy-aware | A TLS-intercepting proxy also cannot work: the charts expose no way to mount a CA bundle into the platform containers, and the relay rejects the intercepted certificate. ## The object store bucket is not created for you **Symptom.** Services start and then fail to read or write the object store. `download-api` answers `500 NoSuchBucket`. **Cause.** Nothing in the charts creates the bucket. **What to do.** [Install](./../05-install.mdx#5-create-the-object-store-bucket) step 5 creates it. It is a first-class step, not an optional one. ## Three services report readiness from a weaker probe `download-api`, `ids-rules-api` and `service-graph-api` serve a richer `/status/healthz` and, in doing so, never register `/status/readyz`. Requesting it returns 404, so a readiness probe pointed at it fails forever while liveness stays healthy, and the pod sits at `0/1` with zero restarts. **What that means for you.** Do not repoint those services at `/status/readyz`. It will fail. ## Helm never upgrades CRDs Not our defect, but it will bite you on the first upgrade that contains CRDs, so it is here. Helm installs a chart's CRDs on first install and ignores them from then on. An upgrade that ships a new CRD schema fails with a server-side-apply error, or silently rejects new fields. [Upgrade](./../operate/upgrade.mdx) applies them by hand as step 1. Do not skip it. ## Kafka volumes survive an uninstall `deleteClaim: false` is deliberate, so `helm uninstall` leaves the Kafka volumes behind. Reinstalling against them produces `Invalid cluster.id ... Expected X, but read Y`, because the new cluster generates a new identity. Delete the volume claims first, or restore the original cluster. See [Kafka will not start](./symptoms.mdx#kafka-will-not-start). ## The audit log needs new partitions after two years The upgrade to v0.2.0 creates monthly audit log partitions for the previous month and the next 24. Nothing creates later ones yet. Past that, audited actions fail with `no partition of relation "audit_log" found for row`, and rule changes made through the API are lost. Create partitions ahead of time, as the database superuser: ```sql SELECT create_audit_log_partition(date '2028-11-01'); ``` ## The audit log is never pruned It is append-only and kept indefinitely, and the application cannot delete from it. To remove old history, back up and then drop a monthly partition (`audit_log_YYYY_MM`) as the superuser. ## Emails are sent as "My App" Invitation and password-reset emails carry the sender name `My App`, whatever you configure. Only the address, `HOST_ALIAS_EMAIL`, is yours. ## New WAF rule functions need current agents `regex_replace()` and `http.request.raw_path` validate from v0.2.0. The rule language is generated from Synapse 0.8.5, and agents on an older version may not evaluate a rule that uses them. Update agents before you rely on either. ## Connection-state firewall rules match every packet A firewall rule with a connection-state condition, such as `allow established,related`, reaches agents without that condition. An allow rule then allows every packet in its direction, ahead of any deny it was written to protect, and a drop rule drops every packet. Nothing reports it. Do not use connection-state conditions on this version. Remove any rule that has one. ## arm64 nodes cannot run Postgres or the ingress operator The Postgres image is published for `linux/amd64` only, and the ingress operator's `linux/arm64` image carries an x86-64 binary. On an arm64 node one fails to pull and the other cannot start. Run both on `linux/amd64` nodes. ## Migration `force` means the opposite of what people expect `migrate force ` marks that version **applied** and continues with the next one. To re-run a migration that failed, force the version **before** it. Migrations are not idempotent. Forcing to the wrong version re-runs DDL that already exists and fails with `already exists`. Example: if `20261004150000` fails, run `force 20261004120000`. --- # Symptom index Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/troubleshoot/symptoms # Symptom index Everything here was reproduced on a real cluster. ## Images ### ImagePullBackOff Three different causes, and the first thing to establish is which. ```bash kubectl -n get pod -o jsonpath='{.spec.containers[*].image}' kubectl -n describe pod | tail -20 ``` | The image reference is | Cause | Fix | |---|---|---| | Correct, and the error mentions authorization | The `gen0sec-registry` secret is missing from that namespace | It is needed in **both** namespaces. See [Install step 2](./../05-install.mdx#the-image-pull-secret) | | Pointing at the wrong registry | A values file was not passed with `-f` | Re-run with `-f $B/values/-values-onprem.yaml` | | A path that does not exist in your registry | Mirror path mismatch, on the offline bundle method | See the next entry | ### 403 with a CAPTCHA challenge from our registry All images, not one, and only on the registry install method. **The tell:** a manual `docker pull` or `skopeo copy` from the same node succeeds while the kubelet's pull returns 403. ```bash # succeeds: the CLI sends its own user agent skopeo inspect docker://registry.gen0sec.com/gen0sec/auth-api:0.2.1 ``` ```bash # 403: what the kubelet gets kubectl -n gen0sec describe pod | grep -A3 Failed ``` On some distributions the component servicing the kubelet's pull sets no `User-Agent`, so Go sends `Go-http-client/1.1`, which our edge challenges as a scanner signature. A kubelet cannot solve a CAPTCHA, so it never recovers. Not credentials, not TLS, not the pull secret. Fix: set a user agent in the container runtime on every node, or use the offline bundle method. Both in [Known limitations](./known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ### ImagePullBackOff on a pod you did not configure Operator-launched pods (Postgres, the connection pooler, Kafka brokers, the entity operator) get their image reference from the **operator**, not from the chart. If one of those is stuck, the mirror path is wrong, not the pull secret. Confirm the path exists in your registry: ```bash curl -su : https://$REGISTRY/v2/_catalog?n=200 | tr ',' '\n' ``` Two paths look surprising and are not negotiable: | Image | Must be mirrored to | |---|---| | Kafka broker | `/gen0sec/strimzi/kafka:1.0.0-kafka-4.1.1` | | Entity operator | `/gen0sec/strimzi/operator:1.0.0` | Everything is under `gen0sec/`. Kafka additionally keeps the upstream `strimzi/` path inside it, because the operator's `defaultImageRegistry` replaces only the registry host and keeps the repository path. A Kafka pod asking for `/strimzi/kafka` with no `gen0sec/` means that setting lost the namespace. ### 401 Unauthorized pulling into a Postgres or pooler pod The Postgres operator does not copy `imagePullSecrets` onto the database pods it creates. They inherit them from their ServiceAccount, which `values-onprem.yaml` configures through `configKubernetes.pod_service_account_definition`. If you overrode `configKubernetes` wholesale, you dropped it. Merge instead of replacing. ### manifest unknown A chart is asking for a tag that was never published. Check what it rendered: ```bash helm get manifest -n | grep -E '^\s+image:' | sort -u ``` Then compare against [Component versions](./../configure/components.mdx). ## Postgres ### Postgres will not start `postgresql/core` sits in `CreateFailed`. ```bash kubectl -n gen0sec-system logs deploy/g0s-infra-postgres-operator | grep -i 'could not create' ``` | Log line contains | Cause | Fix | |---|---|---| | `endpoints is forbidden` | The operator's own ClusterRole omits `endpoints`, but it creates a master Endpoints object per cluster | The infra chart adds the missing rule. Confirm: `kubectl get clusterrole g0s-infra-postgres-operator-endpoints` | | `Unsupported value: "18"` | The CRDs in the cluster are older than the operator chart | Helm never upgrades CRDs. Apply them first. See [Upgrade](./../operate/upgrade.mdx) | | Nothing, and no volume is bound | No default StorageClass, or not enough capacity | See [Sizing](./../02-requirements.mdx#sizing-a-production-install) | After fixing the cause the operator does not retry promptly. Force it: ```bash kubectl -n gen0sec-system rollout restart deploy/g0s-infra-postgres-operator ``` ### Migration failures The migration job runs as a Helm hook. On failure Helm aborts and deploys nothing, so the cluster is in a safe state. ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` | Log line contains | Cause | Fix | |---|---|---| | `relation "..." does not exist` | The chain is running against a database that is not at the version it expects. Usually a forced version, or a `db-migrate` image older than the chart | Read the version table in the next entry before you force anything, then [collect diagnostics](./diagnostics.mdx) | | `role "..." does not exist` | A database role is missing. Older migrations `GRANT` to roles by name and roll back if one is absent. From v0.2.0 a missing role skips the grant with a `NOTICE`, and the service fails later with `permission denied` | The data chart creates them. Confirm: `kubectl -n gen0sec-system get secret \| grep credentials` | | `already exists` | A migration was forced to the wrong version and re-ran DDL | See the next entry | | `canceling statement due to lock timeout` | `20261004150000` could not lock the block events table within 10 seconds. Nothing changed | See [Upgrade](./../operate/upgrade.mdx#5-upgrade-the-platform) | | `no migration found for version` | The `db-migrate` image is older than the schema, typically after a rollback | Use the image of the newest release that ran | ### The schema is marked dirty The migration tool marks the schema dirty when a migration fails halfway, and refuses to continue until you say what happened. ```bash kubectl -n gen0sec-system exec -it core-0 -c postgres -- \ psql -U postgres -d ax_prod -c 'select * from schema_migrations' ``` `force ` means "**this version is applied**, continue with the next one". So to re-run a failed migration, force the version **before** it. Migrations are not idempotent. ## Kafka ### Kafka will not start Broker crash loops with `Invalid cluster.id ... Expected X, but read Y`. The volume survived a previous `helm uninstall`, which is deliberate, and the recreated cluster generated a new identity. Either restore the original cluster or wipe the volumes: ```bash kubectl -n gen0sec-system delete kafkanodepool kafka kubectl -n gen0sec-system delete pvc -l strimzi.io/cluster=core helm upgrade g0s-data ... ``` **This deletes the message data on those volumes.** ### No KafkaTopic resources exist although the chart rendered them Helm can skip re-applying objects that were deleted out of band. Re-apply the stored manifest: ```bash helm get manifest g0s-data -n gen0sec-system | kubectl apply -f - ``` ### Rule changes made through the signal API never appear The request is accepted, but the rule is not created. The signal worker writes an audit record in the same transaction as the rule, and drops the change if that fails. ```bash kubectl -n gen0sec logs deploy/g0s-signal | grep 'Failed to process v2 signal' ``` `permission denied for table audit_log` means the grant was skipped during the migration. `no partition of relation "audit_log"` means the audit log ran out of partitions. See [Known limitations](./known-limitations.mdx#the-audit-log-needs-new-partitions-after-two-years). ### Block events are missing from the dashboard Rows the database refused are parked on the `block_events_dlq` Kafka topic, with the reason in a `dlq_reason` header. Only the refused row is parked, not the batch it arrived in. telemetry-api also logs what it dropped or stored without a field, once a minute per organization: ```bash kubectl -n gen0sec logs deploy/g0s-telemetry-api | grep 'OTLP block events rejected' ``` ## Object store and cache ### Services cannot reach the object store The object store Service is `rustfs-svc`, with the suffix. `S3_ENDPOINT` and `ARXIGNIS_DATA_URL` must match it. ```bash kubectl -n gen0sec-system get svc | grep rustfs kubectl -n gen0sec get deploy g0s-download-api \ -o jsonpath='{.spec.template.spec.containers[0].env}' | tr ',' '\n' | grep -E 'S3_ENDPOINT|DATA_URL' ``` See [Install step 7](./../05-install.mdx#7-install-the-platform). ### 500 NoSuchBucket The bucket does not exist. [Install step 5](./../05-install.mdx#5-create-the-object-store-bucket) creates it. Nothing in the charts does. ### Everything works but reads are slow The shared cache is never fatal. Missing or unreachable, each service logs a warning once and runs on its in-process cache only. Confirm from the service side, not the pod list: ```bash kubectl -n gen0sec logs deploy/g0s-auth-api | grep -i "L2 cache" ``` `L2 cache (Redis) connected to server` means the cache is in use. `falling back to L1-only cache` means `REDIS_URL` is wrong or unreachable. ```bash kubectl -n gen0sec-system get svc dragonfly ``` The Service is deliberately not prefixed with the release name, because `REDIS_URL` names it directly. ### The cache pod crash loops ```bash kubectl -n gen0sec-system logs deploy/dragonfly --previous ``` | Log shows | Cause | Fix | |---|---|---| | `io_uring` errors | Blocked by the host kernel or a seccomp profile | Add `--force_epoll` to `dragonfly.extraArgs` in the infra values | | Read-only filesystem errors | The pod runs with a read-only root, which is why the chart disables snapshotting | Remove any flag you added that re-enables it | ### The cache is being evicted or OOM-killed `maxMemory` is the eviction threshold and the memory limit is the kill threshold. Keep the first comfortably below the second, or the kubelet kills the pod before it gets a chance to evict. ## Install and workstation ### The bundler fails with a yq syntax error You have the wrong `yq`. Two unrelated tools share the name. ```bash yq --version ``` The output must contain `mikefarah`. If not, install the Go version and point the bundler at it with `MIKEFARAH_YQ_PATH`. See [Requirements](./../02-requirements.mdx#check-your-yq). ### zsh: no matches found, before helm runs `zsh` expands `[0]` in a `--set` argument as a glob. Single-quote the whole argument: ```bash --set 'services.ui.ingress.hosts[0].host=example.internal' ``` ### The namespace hangs in Terminating Strimzi puts a finalizer on all 23 KafkaTopics, and once the operator is gone nothing clears it. ```bash kubectl get kafkatopics.kafka.strimzi.io -n gen0sec-system -o name \ | xargs -r -n1 kubectl patch -n gen0sec-system --type=merge \ -p '{"metadata":{"finalizers":null}}' ``` `uninstall.sh` does this for you, in the right order. See [Uninstall](./../operate/uninstall.mdx). ### A pod reports Running but nothing works A pod in `CrashLoopBackOff` still reports `status.phase: Running`. Read the `READY` column, never a field selector on phase. ```bash kubectl -n gen0sec get pods ``` ### Agents get 401 on every relayed artifact Your Gen0Sec API key is wrong, or the secret holds a trailing newline. ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` ``` artifact relayed from upstream path=/v1/ids-rules/version status=401 ``` Recreate the secret with `printf`, not `echo`. `--from-file` stores the bytes exactly, so a newline becomes part of the key. See [Data relay](./../configure/relay.mdx#getting-the-key-right). A missing secret does not produce this: the relay pod fails to start instead. See the next entry. ### CreateContainerConfigError on the relay pod The pod references a secret that does not exist. Almost always `gen0sec-download-proxy`: `downloadProxy.apiKeySecret.name` is set but the secret was never created. ```bash kubectl -n gen0sec describe pod -l app.kubernetes.io/name=download-api | tail -20 kubectl -n gen0sec get secret gen0sec-download-proxy ``` Create it. See [Install step 2](./../05-install.mdx#your-gen0sec-api-key). This failure is the visible one. The dangerous case is the opposite: leaving `apiKeySecret.name` unset renders no secret reference at all, the pod starts, and the failure is the silent 401 above. ### The relay reports 502 on every relayed route The upstream is unreachable. A 502 is deliberate rather than a 404 or 500: a polling agent reads it as "ask again later" and keeps the artifact it already holds. ```bash kubectl -n gen0sec exec deploy/g0s-download-api -- curl -fsS localhost:9999/status/healthz ``` Check that `api.gen0sec.com:443` is allowed. See [Ports and egress](./../security/egress.mdx). If you route egress through an explicit proxy, that is the cause: proxies are not honoured in-cluster. See [Known limitations](./known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy). ## Routing ### An agent gets HTML, or a redirect, from an API The request reached the dashboard's catch-all instead of the intended service. Routing is by longest matching prefix and the dashboard owns `/`, so any path missing from the route table lands there. ```bash kubectl -n gen0sec get ingress -o wide ``` Compare against [Publish the endpoints](./../06-publish.mdx#the-route-table). The failures this causes are quiet by design of HTTP, not by ours: | Missing route | What the agent sees | |---|---| | `/v1/authcheck` | A redirect and HTML. The agent cannot distinguish a bad key from a routing mistake and continues unverified | | `/v1/agents/events` | `SSE connect returned HTTP 404`, then silent fallback to polling, which looks like a slow platform | | `/v1/ban-ledger` | A redirect the agent reads as success. The contribution is lost silently | ### kubectl get ingress shows an empty HOSTS column The ingress controller ignores host-less rules: the object is created, nothing is logged, and no traffic is served. Set `entrypointHost`. See [Publish the endpoints](./../06-publish.mdx). ## Dashboard | Symptom | Cause | |---|---| | Components says it could not read infrastructure versions | The infra chart's `components.ui.*` values do not match the platform release, or a NetworkPolicy blocks the dashboard from the API server. See [Component versions](./../configure/components.mdx#verifying-what-you-are-actually-running) | | Components says data versions are unavailable | The relay or `ids-rules-api` is unreachable from the dashboard | | The online update check cannot reach `releases.gen0sec.com` | Expected when the cluster has no route out. Turn the check off | | No **Forgot your password?** link | SMTP is not configured. See [Invitations and email](./../configure/identity.mdx#invitations-and-email) | | Invitations show a link instead of being emailed | SMTP is not configured, or sending failed. Check `kubectl -n gen0sec logs deploy/g0s-ui` | ## Air gap and mirroring ### Confirming the cluster really is isolated From a pod, not from a node. ```bash kubectl -n gen0sec run netcheck --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/busybox:1.36.1-glibc -- wget -T5 -qO- https://ghcr.io/ ; echo "exit=$?" ``` A non-zero exit is the expected result. If it succeeds, the cluster still has egress and an install that appears to work may be pulling from the internet. ### An image is missing for one architecture You built the bundle with `--arch`. A single-architecture bundle fails when the kubelet tries to pull on a node of the other architecture, which on an isolated cluster is after the transfer. Rebuild without `--arch`, or with the architecture your nodes actually run. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). ## Still stuck [Collect diagnostics](./diagnostics.mdx), then send us the archive. --- # Verify the install Source: https://docs.gen0sec.com/docs/cerebellum/0.2.1/verify # Verify the install Running pods does not mean a working platform. A worker with no Service, a misconfigured credential, or a missing route all leave every pod `Ready` while no data moves. This page defines what "installed successfully" means. ## Before you start - [ ] [First sign-in](./07-first-signin.mdx) complete - [ ] An agent API key was created ## 1. Every pod is ready ```bash kubectl -n gen0sec-system get pods kubectl -n gen0sec get pods ``` Every pod reports `Running` and `n/n` in the `READY` column. On a correct install there are no exceptions: nothing is expected to sit unready. :::note[Check readiness, not phase] A pod in `CrashLoopBackOff` still reports `status.phase: Running`. A `--field-selector` on phase reports a broken install as clean, which is why the command above reads the `READY` column instead. ::: ## 2. The migration job is gone ```bash kubectl -n gen0sec get job g0s-db-migrate ``` The output is similar to this: ``` Error from server (NotFound): jobs.batch "g0s-db-migrate" not found ``` `NotFound` is the pass. Helm deletes the job the moment it succeeds. If it is still there, it failed: `kubectl -n gen0sec logs job/g0s-db-migrate`. ## 3. The relay is connected ```bash kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' ``` The output is similar to this: ```json { "status": "ready", "mode": "proxy", "upstream": "https://api.gen0sec.com", "external_relays": ["geoip", "ids-rules"] } ``` `"mode": "proxy"` confirms the relay came up. If the upstream is unreachable, relayed routes answer 502, which a polling agent reads as "ask again later" rather than "the artifact is gone". Then confirm it is actually authorized upstream, which the mode alone does not tell you: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means your Gen0Sec API key is wrong. ## 4. The platform serves an agent This is the check that matters. It makes the same requests an agent makes, from outside the cluster, over your published address. ```bash export GEN0SEC_BASE_URL=https://$ENTRYPOINT_HOST export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) $KIT/scripts/verify-deployment.sh ``` It checks four things: 1. The API refuses unauthenticated callers. 2. Configuration is served. 3. Threat intelligence and IDS rule bundles are published. 4. A downloaded bundle matches the SHA-256 its manifest advertises. It is read-only and safe to run against production. It exits non-zero and names the failing check. **Run it after every install and every upgrade.** It is the cheapest way to catch a deployment that looks healthy and serves nothing. ## Offline bundle only: confirm nothing pulls from outside ```bash kubectl -n gen0sec-system get pods -o yaml | grep -E '^\s+image:' | sort -u kubectl -n gen0sec get pods -o yaml | grep -E '^\s+image:' | sort -u ``` Every image listed is under your registry. Anything else means a values file was not passed with `-f`. :::note[Why both namespaces, and why read the YAML] The application runs in `gen0sec`; only the operators, object store and cache run in `gen0sec-system`. Checking one namespace covers well under half the images. Reading the YAML rather than a container template also catches init containers, which is where the object store's init image is. ::: ## Definition of installed successfully All five are true: 1. Every pod in both namespaces reports `Running` and `n/n` ready. 2. `kubectl -n gen0sec get job g0s-db-migrate` returns `NotFound` or appears as `Completed`. 3. `download-api` reports `"mode": "proxy"` with your upstream, and its log shows no `status=401` or `status=402` on relayed paths. 4. `verify-deployment.sh` exits zero. 5. On the offline bundle method, every image reference is under your registry. If all five hold, the install is done. If any fail, start at the [symptom index](./troubleshoot/symptoms.mdx). ## You may be interested in... - [Upgrade](./operate/upgrade.mdx), so you know how to move before you need to - [Roll back](./operate/rollback.mdx), and where the data boundary is - [Known limitations in this release](./troubleshoot/known-limitations.mdx) --- # Overview Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/ # Overview ## What is Cerebellum? Cerebellum is the enterprise version of the Gen0Sec platform that you can host in your own Kubernetes cluster. ## Charts | Installation order | Chart | Namespace | What it installs | |---|---|---|---| | 1. | `gen0sec-infra` | `gen0sec-system` | Three operators (Postgres, Kafka, ingress), the object store, and two caches: one shared by the application, one for the ingress data plane | | 2. | `gen0sec-data` | `gen0sec-system` | The Postgres cluster, the Kafka cluster, topics and database roles | | 3. | `gen0sec-platform` | `gen0sec` | Schema migrations, then the application services and the dashboard | The installation order is mandatory, as each chart creates resources the next one depends on. ## Installation methods ### Installation from registry (Recommended) Choose it when, - your cluster can access our registry, - you don't mind depending on an external registry ### Install via creating an offline bundle Choose it when, - your cluster don't have access to our registry, - you'd rather host mirrored images yourself, - you're facing issues pulling images from our registry :::note[Same content] Worry not, the offline bundle builds the same release and installs the same charts. ::: If you are unsure, go with **Install from registry**. You can always switch later. :::info[One thing that can decide this for you] Our registry challenges pulls that arrive with no `User-Agent`, and on some Kubernetes distributions the kubelet's pull path sends none. That is deliberate policy on our side, and the result is a 403 the kubelet cannot recover from. If you cannot change container runtime configuration on your nodes, use **Create an offline bundle**. Your own registry does not inspect user agents. See [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: You will need to decide the installation method in the [Prepare the artifacts](./04-prepare-artifacts.mdx) step. ## Connectivity :::warning[What 'offline' means] The 'offline' part in the offline bundler is used to describe the installation method, not the resulting product. ::: It's important to note that even by creating an offline bundle, doesn't mean you can operate the platform in an airgapped cluster that is entirely isolated from reaching the public internet. Cerebellum needs exactly one outbound destination from your cluster: - `api.gen0sec.com` on port `443`. That single connection carries **threat intelligence, GeoIP databases, ML models and IDS rules**. All four are licensed by their upstream providers and are not included in any of the artifacts we ship. Every agent in your fleet reaches it through one in-cluster service, so it is one destination to allow and one to audit. Nothing else in either namespace needs a route out. The dashboard can optionally check for new versions and send email; see [Network and connectivity](./03-network.mdx). Read [Network and connectivity](./03-network.mdx) before you start. It is the page to send to whoever runs your firewall. ## How long it takes Set aside **half a day for a first install**, and about an hour once you have done one. The install is 26 numbered steps across five pages, and most of the elapsed time is spent waiting on image pulls and on two clusters becoming ready, not on typing. The waits are gated explicitly, so the ceiling is knowable. These are the timeouts the commands themselves carry — the upper bound before a step is treated as failed, not a prediction: | Stage | You are waiting on | Gate | |---|---|---| | [Prepare the artifacts](./04-prepare-artifacts.mdx) | Downloads and signature verification. From the registry this is minutes; building an offline bundle copies every image and is the longest single stage | None | | [Install](./05-install.mdx) infra | Three operators and the object store | None | | [Install](./05-install.mdx) data | Kafka ready, then Postgres `Running` | `15m` each | | [Install](./05-install.mdx) platform | Schema migrations, then every pod ready | `20m` + `10m` | | [Publish](./06-publish.mdx) | The ingress rollout | `20m` | | [First sign-in](./07-first-signin.mdx) | You: MFA enrolment, a forced password change, the onboarding wizard | None | | [Verify](./08-verify.mdx) | Four checks you run | None | What actually decides your number is how fast your nodes pull images. Everything else is bounded by the table above. :::note[The Quickstart is not a shortcut] It is the same procedure with one extra values file per chart, on one node. It is quicker because a single-node cluster pulls and schedules less — not because it skips steps. ::: ## Before you install 1. [Requirements](./02-requirements.mdx). Cluster, tools, and your access token. 2. [Network and connectivity](./03-network.mdx). What to allow, and why. 3. [Prepare the artifacts](./04-prepare-artifacts.mdx). Prepare for the installation. If you want a working instance to look at before you plan a real install, use the [Quickstart](./quickstart.mdx). It builds a throwaway single-node install and tears it down again. --- # Component versions Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/configure/components # Component versions :::note[Read this from your release, not from this page] Every release pins its own versions, and the authoritative list is in the install kit you downloaded. ```bash yq '.spec.images[] | [.image, .imageTag, .digest] | @tsv' $KIT/release-manifest.yaml yq '.spec.charts[] | [.chart, .version, .digest] | @tsv' $KIT/release-manifest.yaml ``` This page describes what those versions mean and which of them have surprising requirements. ::: {/* This page should be generated from release/components.yaml and release-manifest.yaml in CI, per the structure proposal section 7. Hand-maintained, it drifts within one release and is then worse than nothing, because operators trust it. */} ## Third-party components | Component | Version | Notes | |---|---|---| | Postgres operator | 2.0.1 | Zalando. **1.15.x cannot be used**: its schema rejects PostgreSQL 18 | | PostgreSQL | 18, with extensions | Includes AGE, TimescaleDB, `pg_cron`, `pg_net`, `pglogical`, `pgsodium` | | Connection pooler | 2.0.1 | Operator-launched. Services connect through it, never directly | | Logical backup | 2.0.1 | Only used when logical backups are enabled | | Kafka operator | 1.0.0 | Strimzi. v1 CRD API only | | Apache Kafka | 4.1.1 | **KRaft, no ZooKeeper.** Operator-launched | | Object store | Chart 0.10.0 | S3-compatible. Its Service has a `-svc` suffix | | Shared cache | 1.38.1 | Redis protocol. Application services degrade to in-process caching without it | | Ingress cache | Valkey | Deployed by the ingress chart and used only by it. Kept separate on purpose, so a shared-cache outage cannot take the ingress data plane with it. One replica: a restart costs a cold start, not data | | Ingress operator | 0.1.13 | `ingressClassName: synapse`. Built and versioned in its own repository | | Platform services | The release version | Resolved from the registry host, namespace and image tag | ## Mirror paths, for the offline bundle method Every mirrored image lands under `gen0sec/`. Two paths look wrong and are not negotiable. | Image | Must be at | |---|---| | Kafka broker | `/gen0sec/strimzi/kafka:1.0.0-kafka-4.1.1` | | Kafka entity operator | `/gen0sec/strimzi/operator:1.0.0` | **Why Kafka keeps an extra path segment.** The Kafka operator's image-registry setting replaces only the registry *host* and preserves the repository path. So the chart points it at `/gen0sec` rather than ``, and the upstream `strimzi/` segment survives inside our namespace. :::note[This exact thing shipped wrong once] With the namespace omitted, the chart asked for `/strimzi/operator` while the release mirrored `/gen0sec/strimzi/operator`. One path segment apart, and the entire Kafka stack failed to pull on a cluster with no fallback. Every chart is now rendered in CI and the build fails on any image reference not backed by the release manifest, so the two cannot drift again. That gate is also why the table above is trustworthy: it catches a chart asking for a tag nobody publishes, which is how the ingress operator once requested a version that was never released. ::: `bundler.sh mirror` reproduces these paths exactly. You do not compute them. ## Not everything a chart renders is mirrored The vendored Kafka chart hardcodes an image map covering four Kafka versions plus Connect and Bridge build tooling. This edition ships one Kafka version and no Connect clusters, so the rest are deliberately absent from the bundle. That is roughly 1.5 GB of images nothing ever pulls. **If you enable Kafka Exporter, Cruise Control, Kafka Bridge or Connect builds, mirror their images first.** The release does not carry them. ## Kubernetes | | | |---|---| | Supported | 1.23 or later | See [Requirements](./../02-requirements.mdx#kubernetes-versions). ## Support policy | | | |---|---| | Supported releases | The latest two minor trains | | Upgrade path | Sequential minors. **Skipping a minor is not supported.** Patches in place | | Upgrade order | Always infra, then data, then platform | | CRDs | Helm never upgrades them. Apply them before the chart. See [Upgrade](./../operate/upgrade.mdx) | | Downgrade | Not supported across a minor that ran a schema migration. Within a train, use rollback. See [Roll back](./../operate/rollback.mdx) | | Kubernetes upgrades | Independent of ours. Stay within the supported range | ## Verifying what you are actually running The dashboard shows it under **Organization settings**, **Components**: what each component is running, and the latest version known to this release. Each chart records its versions in a ConfigMap, which you can also read directly: ```bash kubectl -n gen0sec get cm g0s-installed-versions -o jsonpath='{.data.versions\.json}' kubectl -n gen0sec-system get cm g0s-infra-installed-versions -o jsonpath='{.data.versions\.json}' kubectl -n gen0sec-system get cm g0s-data-installed-versions -o jsonpath='{.data.versions\.json}' ``` Two things about that page: - **Check for updates online** is off by default and needs `org:write`. When on, the dashboard fetches the latest release's version list from `releases.gen0sec.com` at most once an hour. Nothing about your install is sent. Air-gapped installs leave it off. - Opening it creates an API key named **Components tab (internal)**, which it uses to read data versions from the relay. Deleting it is safe; it is recreated on the next visit. If the page says it could not read infrastructure versions, the infra chart's `components.ui.namespace` and `components.ui.serviceAccountName` do not match the platform release. See [Values reference](./values.mdx#gen0sec-infra). The manifest says what a release pins. This says what your cluster pulled: ```bash kubectl -n gen0sec-system get pods -o yaml | grep -E '^\s+image:' | sort -u kubectl -n gen0sec get pods -o yaml | grep -E '^\s+image:' | sort -u ``` Reading the YAML rather than a container template also catches init containers, which is where the object store's init image is. On the offline bundle method, every reference should be under your own registry. Anything else means a values file was not passed. --- # Sign-in and identity Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/configure/identity # Sign-in and identity ## What is available | Method | Status | Where credentials live | |---|---|---| | Email and password | **Available** | Your Postgres | | Passkeys | **Available** | Your Postgres holds the public key. The private key never leaves the user's device | | Multi-factor authentication | **Available, and required during onboarding** | Your Postgres | | Password reset by email | **Available when [SMTP is configured](#invitations-and-email)** | Your mail server sends the link | | Microsoft Entra ID | **In development, not supported at the moment ** | | **Nothing outside your cluster is involved in sign-in.** Accounts, password hashes, passkeys and sessions are all in your own database. If you configure SMTP, invitation and password-reset emails go out through your mail server. :::note[Single sign-on is not available yet] Microsoft Entra ID is in development. No generic OIDC provider, no SAML and no LDAP integration exists either. If single sign-on is a requirement for you, tell us which provider, because that shapes what ships first. ::: ## The first account Registration is invite-only and invites need an existing member, so a fresh install seeds one account with a **published password**. Signing in, completing onboarding and changing that password is a required install step, not an optional one. See [First sign-in](./../07-first-signin.mdx). ## The published address decides whether sign-in works The dashboard derives its sign-in base URL from the host you publish it on. If that does not match the address people actually type in the browser, every sign-in is rejected as an invalid origin. Two rules: 1. **Use the address people type.** Not an internal service name, not a load balancer's own hostname. 2. **If TLS terminates upstream of the ingress**, set the scheme explicitly, because the chart cannot infer it and guesses `http` while the browser is on `https`. ```bash --set externalScheme=https ``` Leave that unset when TLS terminates at the ingress. See [Publish the endpoints](./../06-publish.mdx#if-tls-terminates-upstream). ## Passkeys Users can register with a passkey instead of a password, and add, rename or remove passkeys under **Profile**. Signing in with a passkey does not ask for the authenticator-app code: the passkey's own PIN or biometric check stands in for it. Enrolling multi-factor is still required during onboarding. A passkey is bound to the host name the dashboard is published on, taken from its sign-in URL. So: - **They need HTTPS and a DNS name.** A plain-HTTP address or a bare IP cannot register one. - **Changing `entrypointHost` makes every registered passkey stop working.** Users then sign in with a password, or register a new passkey on the new address. - **A user who registered with a passkey only has no password to fall back on.** Ask them to set one under **Profile**, especially when SMTP is not configured and nobody can reset it by email. ## Invitations and email Invitations grant one or more workspaces, each with a role no higher than the inviter's own. Pending invitations are listed on the **Team** tab, where they can be revoked. A link expires after seven days, and inviting the same address again replaces the previous link. With SMTP configured, the invitation is emailed and users can reset a forgotten password. Without it, the dashboard shows the invitation link for you to copy and send yourself, and the sign-in page offers no password reset. SMTP is configured through the dashboard's environment. The chart has no dedicated keys for it, so set them under `services.ui` in your overrides file, with the credentials in a Secret you create: ```bash kubectl -n gen0sec create secret generic g0s-ui-smtp \ --from-literal=user='' --from-literal=password='' ``` ```yaml title="my-overrides.yaml" services: ui: env: HOST_EMAIL: smtp.example.com HOST_PORT: "587" HOST_ALIAS_EMAIL: no-reply@example.com HOST_APP_URL: https://cerebellum.example.com envSecrets: HOST_AUTH_USER: { name: g0s-ui-smtp, key: user } HOST_AUTH_PASS: { name: g0s-ui-smtp, key: password } ``` | Variable | Notes | |---|---| | `HOST_EMAIL` | The SMTP server's host name | | `HOST_PORT` | `587` by default, with STARTTLS. `465` switches to implicit TLS | | `HOST_AUTH_USER`, `HOST_AUTH_PASS` | **Required.** SMTP is only used when the host and both credentials are set. A relay without authentication is not supported | | `HOST_ALIAS_EMAIL` | **Required.** The sender address | | `HOST_APP_URL` | **Required.** The address people type, with its scheme. Links in emails are built from it, and it is not derived from `entrypointHost` | Pass the file to every later `helm upgrade` as well. The dashboard's outbound connection to your mail server is listed on [Ports and egress](./../security/egress.mdx). ## Sessions The signing secret lives in `g0s-db-ui`, created with all the other database secrets in [Install step 6](./../05-install.mdx#6-create-the-application-secrets). | Action | Effect on sessions | |---|---| | Re-running `make-db-secrets.sh` | None. The existing signing secret is preserved deliberately | | Rotating the signing secret | Every session ends and every outstanding invite link stops working. Passwords and passkeys are unaffected | | Changing `entrypointHost` | Registered passkeys stop working | | Upgrading | None. Upgrading from v0.1.x to v0.2.0 invalidates invite links issued before the upgrade | ## Turning the dashboard off entirely If your operators use the API only: ```bash --set services.ui.enabled=false ``` The agent-facing APIs are unaffected. Note that agent API keys are issued through the dashboard, so turn it off only once you have the keys you need. ## If it fails | Symptom | Cause | |---|---| | Sign-in rejected as an invalid origin | The published host does not match the address typed, or the scheme is wrong. See above | | Sessions drop after a re-run of the secret script | The signing secret changed. It should not; check whether the secret was deleted first | | Every sign-in fails after an upgrade | Check that `externalScheme` and the host survived the upgrade. `helm get values g0s -n gen0sec` | | The passkey prompt fails or offers no passkey | The address typed differs from `entrypointHost`, the page is plain HTTP, or the host changed after the passkey was registered | | Emailed links start with `undefined/` | `HOST_APP_URL` is not set | | Invitations show a link instead of being emailed | SMTP is not configured, or sending failed. Check `kubectl -n gen0sec logs deploy/g0s-ui` | --- # Reference configurations Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/configure/reference-configs # Reference configurations Two named configurations rather than a blank values file. Each ships in the install kit, and each is reproduced in full on this page, so you can read it before you download anything. | Configuration | For | Redundancy | Storage | |---|---|---|---| | [Single node](#single-node) | Evaluation, training, upgrade rehearsal | **None** | Smallest | | [Production](#production) | Anything you depend on | Full | Smallest production grade | ## Single node **Not for production.** Everything at one replica on one node. Losing the node loses the data. Ships as three overlays in the kit: ``` $KIT/examples/single-node/infra-values.yaml $KIT/examples/single-node/data-values.yaml $KIT/examples/single-node/platform-values.yaml ``` Layer each one **after** the production values file, so it wins. Full procedure: [Quickstart](./../quickstart.mdx). | | Production | Single node | |---|---|---| | Object store | 3 replicas, erasure coded, StatefulSet | 1 replica, standalone, **Deployment** | | Postgres | 3 instances, both poolers on | 1 instance, replica pooler off | | Kafka | 3 brokers, replication factor 3 | 1 broker, every factor 1 | | Services | 2 or 3 replicas | 1 each | | StorageClass | Yours, set explicitly | The cluster's default (empty in the overlay) | ### What the overlays set Verbatim from the v0.2.2 kit, so you can read them without unpacking it first.
infra-values.yaml — object store and cache ```yaml title="$KIT/examples/single-node/infra-values.yaml" # ============================================================================ # gen0sec-infra — SINGLE-NODE EVALUATION overlay # ============================================================================ # Layered ON TOP of values-onprem.yaml (pass both, in this order): # helm install g0s-infra charts/gen0sec-infra -n gen0sec-system \ # -f values/gen0sec-infra-values-onprem.yaml \ # -f examples/single-node/infra-values.yaml # # Shrinks the footprint to fit one worker node. NOT for production: the object # store loses erasure coding and every component becomes a single point of # failure. Use it for evaluation, training, and dry-running an upgrade. # ============================================================================ rustfs: # Distributed mode needs >= 2 nodes for erasure coding; on one node the pods # would schedule but the cluster is not fault-tolerant anyway, so run the # single-process server and be explicit about it. mode: standalone: enabled: true distributed: enabled: false replicaCount: 1 storageclass: # Empty = the cluster's default StorageClass, matching values-onprem.yaml. On # k3s that resolves to local-path, which this overlay used to hardcode — but a # literal `local-path` exists on no other distribution, so the PVCs bound # nowhere and only surfaced as pods Pending past the rollout timeout. name: "" dataStorageSize: 20Gi logStorageSize: 5Gi dragonfly: # Cache only — shrink it so the node's RAM goes to Postgres and Kafka. Keep # maxMemory under resources.limits.memory so it evicts instead of being killed. maxMemory: 256mb proactorThreads: 1 resources: requests: { cpu: 100m, memory: 128Mi } limits: { cpu: 500m, memory: 512Mi } ```
data-values.yaml — Postgres and Kafka ```yaml title="$KIT/examples/single-node/data-values.yaml" # ============================================================================ # gen0sec-data — SINGLE-NODE EVALUATION overlay # ============================================================================ # helm install g0s-data charts/gen0sec-data -n gen0sec-system \ # -f values/gen0sec-data-values-onprem.yaml \ # -f examples/single-node/data-values.yaml # # One Postgres instance, one Kafka node, replication factor 1. Any node loss is # data loss — evaluation only. # ============================================================================ postgres: numberOfInstances: 1 # With a single instance there is no replica to pool. enableReplicaConnectionPooler: false volume: size: 20Gi # Empty = the cluster's default StorageClass (see infra-values.yaml). storageClass: "" resources: requests: { cpu: 500m, memory: 1Gi } limits: { cpu: "4", memory: 6Gi } parameters: shared_buffers: 512MB effective_cache_size: 1536MB max_connections: "400" max_worker_processes: "16" max_parallel_workers: "8" timescaledb.max_background_workers: "8" kafka: replicas: 1 storage: size: 20Gi # Empty = the cluster's default StorageClass (see infra-values.yaml). class: "" # min.insync.replicas must stay < replicas, so every factor drops to 1 # together with the node count. config: offsets.topic.replication.factor: 1 transaction.state.log.replication.factor: 1 transaction.state.log.min.isr: 1 default.replication.factor: 1 min.insync.replicas: 1 log.compression.type: zstd topicDefaults: partitions: 1 replicas: 1 ```
platform-values.yaml — one replica per service ```yaml title="$KIT/examples/single-node/platform-values.yaml" # ============================================================================ # gen0sec-platform — SINGLE-NODE EVALUATION overlay # ============================================================================ # helm install g0s charts/gen0sec-platform -n gen0sec \ # -f values/gen0sec-platform-values-onprem.yaml \ # -f examples/single-node/platform-values.yaml \ # --set global.imageTag=$VERSION --timeout 20m # # One replica of everything: no HA, and a rolling update is a short outage per # service. Resource requests are unchanged — they are already small. # ============================================================================ services: auth-api: { replicaCount: 1 } ban-ledger-api: { replicaCount: 1 } config-api: { replicaCount: 1 } data-api: { replicaCount: 1 } download-api: { replicaCount: 1 } feeds-api: { replicaCount: 1 } ids-rules-api: { replicaCount: 1 } rules-validator: { replicaCount: 1 } service-graph-api: { replicaCount: 1 } signal-api: { replicaCount: 1 } telemetry-api: { replicaCount: 1 } threat-api: { replicaCount: 1 } agent-status: { replicaCount: 1 } ui: { replicaCount: 1 } ```
## Production The default. `values-onprem.yaml` in the bundle **is** the production configuration, so there is no overlay to apply: what you install in [Install](./../05-install.mdx) is this. What you set yourself: ```yaml # gen0sec-infra rustfs: storageclass: name: your-block-storage-class # gen0sec-data postgres: volume: storageClass: your-block-storage-class kafka: storage: class: your-block-storage-class ``` For requirements see [Sizing](./../02-requirements.mdx#sizing-a-production-install). ## Extending retention The shipped sizes are deliberately modest, so growing is the normal direction. Which key you grow depends on what you are retaining. See [Sizing storage for your retention](./../02-requirements.mdx#sizing-storage-for-your-retention). eg. ```yaml # gen0sec-data postgres: volume: size: 200Gi storageClass: your-block-storage-class kafka: storage: size: 200Gi class: your-block-storage-class ``` ```yaml # gen0sec-infra rustfs: storageclass: name: your-block-storage-class dataStorageSize: 100Gi ``` :::note[You can also grow later] Volume expansion grows a volume if your StorageClass has `allowVolumeExpansion: true`. Starting at the defaults and expanding when you need to is a reasonable plan, and it is the direction that works: nothing shrinks a volume. ::: ## Publishing endpoints Independent of the three above, and needed by all of them: ``` $KIT/examples/single-entrypoint/ingress-values.yaml ``` One address for the dashboard and every agent-facing API, routed by path prefix. See [Publish the endpoints](./../06-publish.mdx) for the route table and the two values you set.
The header of that file, with both customer inputs and why they exist The rest of the file is the per-service route list. It is reproduced as a table on [Publish the endpoints](./../06-publish.mdx#the-route-table) rather than twice here. ```yaml title="$KIT/examples/single-entrypoint/ingress-values.yaml (head)" # ============================================================================ # gen0sec-platform — SINGLE-ENTRYPOINT INGRESS overlay # ============================================================================ # Publishes the dashboard and the agent-facing APIs through ONE address, served # by the Synapse proxy that gen0sec-infra deploys for the `synapse` IngressClass. # # helm upgrade --install g0s charts/gen0sec-platform -n gen0sec \ # -f values/gen0sec-platform-values-onprem.yaml \ # -f examples/single-entrypoint/ingress-values.yaml # # Layered ON TOP of values-onprem.yaml, which leaves every ingress disabled. # That default is deliberate — turning this on makes the APIs reachable from # wherever the proxy Service is reachable — so this is opt-in, not the default. # # A HOST IS REQUIRED — set `entrypointHost`, either below or with # `--set entrypointHost=...`. The chart reads it and publishes every rule under # it, so it is named once and nowhere else. # # It used to be a YAML anchor aliased into each rule. That could only ever be # changed by editing this file: anchors are expanded by the YAML parser before # Helm merges `--set`, so `--set entrypointHost=...` rewrote the top-level key # while every aliased copy kept `gen0sec.internal`, and the install published # on a host nobody could reach. The chart consumes the value now, so the # documented `--set` works. # # Leaving it empty FAILS THE RENDER rather than publishing anything. Host-less # rules do not work with this controller: synapse's upstreams.yaml is keyed by # host, and the operator skips any rule with an empty host # (controllers/ingress_controller.go, "if host == \"\" || rule.HTTP == nil # { continue }"). Such a rule produces an Ingress object that its own controller # silently ignores, with no event and no log line, so the chart refuses to # render one at all. # # The dashboard needs this host to be the address people actually type: # BETTER_AUTH_URL is derived from it (services.ui.urlFromIngress), and sign-in # breaks otherwise. # # ROUTING IS BY PATH PREFIX, because one address serves every backend. The # prefixes below are the real route roots, read off each service's router -- # they are not a naming convention and do not survive being guessed at. # # One overlap to keep in mind, resolved by longest-prefix matching: # /v1/agents/events config-api serves this; data-api owns /v1/agents # If a request lands on the wrong backend, that is the first thing to check. # # /v1/ids-rules goes to download-api, NOT the in-cluster ids-rules-api: in proxy # mode IDS rules are relayed from the platform, and ids-rules-api serves a local # ET ingest that this edition does not run. # ============================================================================ # CUSTOMER-INPUT: the single address every rule below is published under. entrypointHost: "" # CUSTOMER-INPUT: the scheme users actually reach that address on. # # Only needed when TLS terminates UPSTREAM of the Ingress -- a cloud load # balancer, a CDN, a tunnel. The chart otherwise infers the scheme from the # Ingress's own `tls:` block, which is absent in that setup, so it guesses http # while the browser is on https. The dashboard hands this to Better Auth as its # base URL and every sign-in is then rejected with "Invalid origin". # # Leave it unset when TLS terminates at the Ingress (i.e. you set ingress.tls), # and the chart reads the scheme off that TLS block instead. # # A single service can still override this with # `services..ingress.externalScheme`, which wins over this value. externalScheme: https ```
## Keeping your overrides Put them in a file under version control, not in `--set` arguments. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ -f my-site/platform.yaml \ --set global.imageTag=$VERSION \ --timeout 20m ``` `helm upgrade` uses only what you pass it. Anything set with `--set` on a previous install and omitted now reverts to the chart default, silently. A file makes that impossible to forget. Keep `entrypointHost` in that file. The ingress overlay ships it empty, and the render fails without it. Pass your file after the overlay so your value wins. Two things that are awkward to keep in a file, because they are per-release or per-secret: | Keep as `--set` | Why | |---|---| | `global.imageTag` | Changes every release | | `downloadProxy.apiKeySecret.name` | Or put it in your file. Just never omit it. See [Data relay](./relay.mdx) | ## What the kit contains ```bash ls $KIT/examples/*/ ``` Everything under `examples/` is a starting point you are meant to read and edit, not a black box. Each file carries comments explaining which values are yours to set. --- # Threat intelligence and the data relay Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/configure/relay # Threat intelligence and the data relay The relay is one in-cluster service, `download-api`. It is the only component with a route out, and it is how your fleet gets threat intelligence. This is not a choice you make. It is a requirement. See [Known limitations](./../troubleshoot/known-limitations.mdx#no-zero-egress-deployment). ## What it relays Read-only, upstream-produced content only. | Class | Routes | Why it is not shipped | |---|---|---| | CTI indicators | `/v1/indicators/version`, `/v1/indicators/download` | Built by the upstream intelligence pipeline | | ML models | `/v1/models/*` and the per-model aliases | Trained and published by Gen0Sec | | GeoIP | `/v1/geoip/manifest`, `/v1/geoip/{file}/download` | Licensed by MaxMind and IPInfo. The IPInfo Lite bundle is the default source | | IDS rules | `/v1/ids-rules/*` | Licensed as part of the Emerging Threats ruleset | **It is a relay, not a cache.** Nothing is stored on the way through. There is no local artifact store to size, no synchronization job to supervise, and no staleness window. An agent that polls gets whatever upstream has at that moment. **The relayed URLs are identical to the direct ones**, so your agents need no configuration change. ## What it never relays | Class | Routes | Why | |---|---|---| | Workload identity | `/v1/identity/*` | Produced in your cluster, describes your workloads | | Policy edges | `/v1/policy-edges/*` | Pod IPs, namespaces, NetworkPolicy edges | | Fleet bans | `/v1/fleet-bans/*` | Your own agents' ban decisions | These three are tenant-scoped: they describe your infrastructure or your decisions. Relaying them would push that to a service you do not run. They keep the local object store path behind the local authorization check, in relay mode exactly as without it. Uploads are never relayed. The relay is read-only, so `PUT` and `POST` on the download surface are served locally or not at all. ## How authorization works Two hops, two credentials, checked in two places. | Hop | Credential | Checked by | |---|---|---| | Agent to relay | The agent's own key, issued by your dashboard | Your cluster. Relayed routes sit behind the local auth middleware and each carries the same permission it has in direct mode | | Relay to Gen0Sec | Your Gen0Sec API key | Gen0Sec | **A caller's key is never forwarded upstream on a self-hosted deployment.** A locally-issued key cannot authenticate to gen0sec. Forwarding it would leak a local credential to a third party without making the request succeed. With no Gen0Sec API key configured, the relay sends no credential at all. ## The setting that must be right ```yaml downloadProxy: enabled: true upstream: "https://api.gen0sec.com" apiKeySecret: name: gen0sec-download-proxy # named by default key: DOWNLOAD_PROXY_API_KEY ``` ### Why the key is required Your agents' keys are issued by **your** dashboard and exist only in **your** database, so they cannot authenticate to gen0sec. The relay presents your own Gen0Sec API key instead, and that is the only credential the upstream will accept from your installation. Without it, the relay sends no credential, Gen0Sec answers 401, and no relayed artifact reaches your fleet. :::note[The chart names the secret by default, so this fails loudly now] `apiKeySecret.name` defaults to `gen0sec-download-proxy`. If that secret does not exist, the relay pod never starts: `CreateContainerConfigError`, visible immediately. That is deliberate. An empty name renders no environment variable at all, the relay starts, and the only symptom is every relayed class answering 401 while agents report empty rulesets rather than errors. A failed rollout at install time is the cheaper failure. Blank the name only if you relay through a gateway that supplies its own credential. ::: ### Getting the key right ```bash printf '%s' '' > ~/.gen0sec/relay-key kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-file=DOWNLOAD_PROXY_API_KEY=$HOME/.gen0sec/relay-key ``` The value renders as a `secretKeyRef`, so it never appears in a manifest or in a values file. ## All settings | Key | Default | Meaning | |---|---|---| | `downloadProxy.enabled` | `true` | Turning it off is not supported | | `downloadProxy.upstream` | `https://api.gen0sec.com` | The upstream base URL. A malformed value **fails startup** rather than degrading | | `downloadProxy.external` | `geoip,ids-rules` | Which external classes to relay. An unknown name fails startup | | `downloadProxy.timeout` | `60s` | Upstream request timeout, matching the router's own | | `downloadProxy.apiKeySecret.name` | `""` | **Set this.** See above | | `downloadProxy.apiKeySecret.key` | `DOWNLOAD_PROXY_API_KEY` | The key within the secret | :::note[Failing startup is deliberate] A malformed upstream is not treated as "fall back to reading from the local object store". That would leave every agent poll erroring against a store this deployment does not populate. Failing fast is the safer wrong answer. ::: ## Verifying ```bash kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' ``` The output is similar to this: ```json { "status": "ready", "mode": "proxy", "upstream": "https://api.gen0sec.com", "external_relays": ["geoip", "ids-rules"] } ``` **That tells you the relay started, not that it is authorized.** For that, read the log: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means your Gen0Sec API key is missing or wrong. ## Response codes worth knowing | Code | Means | What an agent does | |---|---|---| | 200 | Relayed successfully | Uses the artifact | | **401** | The upstream rejected the credential | **Reports an empty ruleset, not an error.** This is the silent failure | | **502** | The upstream is unreachable | Reads it as "ask again later" and keeps what it holds. Correct degradation | | 404 | The artifact does not exist upstream | Treats it as withdrawn | The 401 and 502 rows are the reason this page exists. ## Preview a change before applying it ```bash helm template g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set downloadProxy.apiKeySecret.name=gen0sec-download-proxy \ | grep -A3 DOWNLOAD_PROXY ``` --- # Secrets Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/configure/secrets # Secrets **No chart contains a credential.** Every one is referenced by name. This page is explains what must exist, with which keys, before the platform starts. ## The secrets you create | Secret | Namespace | Keys | Created in | |---|---|---|---| | `gen0sec-registry` | **Both** | `.dockerconfigjson` | [Install step 2](./../05-install.mdx#the-image-pull-secret) | | `gen0sec-object-store` | `gen0sec-system` | `RUSTFS_ACCESS_KEY`, `RUSTFS_SECRET_KEY` | [Install step 2](./../05-install.mdx#the-object-store-credentials) | | `gen0sec-download-proxy` | `gen0sec` | `DOWNLOAD_PROXY_API_KEY` | [Install step 2](./../05-install.mdx#your-gen0sec-api-key) | The pull secret is needed in **both** namespaces. The application runs in `gen0sec`; the operators, object store and cache run in `gen0sec-system`. ## The twenty the script creates `make-db-secrets.sh` composes one secret per service from the passwords the Postgres operator generated. | Secret | Contains | Role it uses | Endpoint | |---|---|---|---| | `g0s-db-` | `DATABASE_URL`, plus object store keys for services that use it | Its own per-service role, read-only where it only reads | The connection pooler | | `gen0sec-db-migrate` | `DATABASE_URL` | `postgres`, superuser | The primary, unpooled | | `g0s-db-ui` | `DATABASE_URL`, `BETTER_AUTH_SECRET`, object store keys | `frontend` | The connection pooler | One more is optional and not created by the script: SMTP credentials for invitation and password-reset email, referenced from `services.ui.envSecrets`. See [Invitations and email](./identity.mdx#invitations-and-email). A referenced Secret that does not exist leaves the dashboard in `CreateContainerConfigError`. Every service connects as its own role. The migration job needs a superuser on the primary because it runs `CREATE EXTENSION`, which an owner role cannot do. :::note[The session signing key is preserved] `g0s-db-ui` carries the dashboard's session signing secret. Re-running the script keeps the existing value, so sessions and invite links stay valid across a re-run. Nothing else regenerates it. ::: Four services need no database connection and get no `DATABASE_URL`: the GeoIP updater, the GeoIP database updater, the model publisher and the rules validator. ## Where the database passwords come from The Postgres operator creates a credentials secret per role it provisions, named after the role: ``` .core.credentials.postgresql.acid.zalan.do ``` Underscores in the role become dashes in the secret name. Compose each connection string from that role's password: ``` postgres://:@core-pooler.gen0sec-system.svc.cluster.local:5432/ax_prod?sslmode=require ``` | Consumer | Role | Endpoint | |---|---|---| | Each service | Its own per-service role | `core-pooler` | | Migration job | `postgres` | `core`, the primary, with the on-prem marker appended | | Dashboard | `frontend` | `core-pooler` | The migration job's connection string goes to the primary and carries the on-prem marker, `options=-c%20gen0sec.deployment_mode%3Donprem`: ``` postgres://postgres:@core.gen0sec-system.svc.cluster.local:5432/ax_prod?sslmode=require&options=-c%20gen0sec.deployment_mode%3Donprem ``` Migrations read the marker to tell your database from our hosted one. Without it the seeded admin account is not created, and from v0.2.1 rule changes stop reaching agents. :::note[One deliberate exception] The dashboard's connection string uses `sslmode=no-verify`. Its client library verifies the certificate chain for `sslmode=require` and rejects the pooler's self-signed certificate; the Go library every other service uses reads the same value as encrypt-without-verify. **Do not copy `no-verify` onto the other services.** Their driver rejects it outright. ::: ## For production: use your secret manager `make-db-secrets.sh` is fine for evaluation. In production, treat its output as a **specification** rather than a tool, and have your secret manager produce secrets of the same shape. Two reasons: 1. The script writes plain Kubernetes secrets from operator-generated passwords. Your manager should be the source of truth, with the cluster holding copies. 2. Rotation is then a manager operation rather than a re-run plus a rollout. An external secret can template the connection string from the operator-generated password, so nothing is copied by hand: ```yaml # shape, not a working manifest: adapt to your operator data: - secretKey: DATABASE_URL remoteRef: key: g0s-auth-api-ro.core.credentials.postgresql.acid.zalan.do property: password # then template: postgres://:{{ .password }}@core-pooler...:5432/ax_prod?sslmode=require ``` To see the exact list your install needs, read it out of the values file rather than from this page: ```bash grep -oE 'secretRef:\s*\{?\s*name:\s*[a-z0-9-]+' \ $B/values/gen0sec-platform-values-onprem.yaml | sort -u ``` Or inspect your live installation inside your cluster. ## Rotation | Credential | How | |---|---| | Database role passwords | Rotate through the Postgres operator, then restart the consumers | | Object store keys | Update **both** `gen0sec-object-store` and every `g0s-db-` that carries them, then restart the consumers | | Registry pull credentials | Recreate `gen0sec-registry` in both namespaces. Existing pods keep running; new pulls use the new value | | Your Gen0Sec API key | Recreate `gen0sec-download-proxy`, then restart the relay | | Dashboard session key | Rotating invalidates every session and every outstanding invite link. Passwords and passkeys are unaffected | | SMTP credentials | Update the Secret, then `kubectl -n gen0sec rollout restart deploy/g0s-ui` | :::note[One credential pair, surfaced two ways] The object store keys appear as `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` to the object store itself, and as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to the services that read it. Same values, two naming conventions. Rotating means updating both places. ::: ## Verifying Nothing should be missing before you install the platform: ```bash $KIT/scripts/preflight.sh ``` After install, confirm the count: ```bash kubectl -n gen0sec get secret | wc -l # ~20 service secrets plus the ones you created kubectl -n gen0sec-system get secret | grep credentials | wc -l # expect 6 ``` ## What is never in a secret | | Where it is instead | |---|---| | The registry host and namespace | The values file | | The relay upstream URL | The values file | | The cache connection string | The values file, deliberately with no credential in it | If you enable authentication on the cache, remove that line from the values file and put the connection string in each service's secret instead. --- # Values reference Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/configure/values # Values reference :::note[This page is the reference] It covers the keys you actually set, which is a much shorter list than every key a chart accepts, and states which of them are required. For anything not listed here, read the chart's own `values.yaml` in the bundle. There is no generated key-by-key reference. The charts carry no JSON schema yet, so nothing can be generated from one; when that changes this page will link to it. ::: ## How values are layered Three files, applied in order, last one wins. ``` chart values.yaml the chart's own defaults, registry deliberately blank values-onprem.yaml shipped in the bundle, points everything at a registry your overrides what you set ``` **Always pass `-f values-onprem.yaml`.** The default `values.yaml` leaves the registry blank on purpose, so installing without it cannot resolve images. Put your overrides in a file rather than in `--set` arguments, because `helm upgrade` uses only what you pass it. Anything set with `--set` last time and omitted this time reverts to the chart default. ```bash helm get values g0s -n gen0sec # what the running release actually has ``` ## gen0sec-infra Top-level keys: `postgres-operator`, `strimzi-kafka-operator`, `synapse-operator`, `synapse`, `rustfs`, `dragonfly`, `components`. | Key | Default | Set it when | |---|---|---| | `rustfs.storageclass.name` | `""`, meaning the cluster default | You want a specific StorageClass | | `rustfs.storageclass.dataStorageSize` | `50Gi` | Sizing the object store. **Four drives per replica** | | `rustfs.storageclass.logStorageSize` | `10Gi` | Rarely | | `rustfs.replicaCount` | `3` | Fewer nodes. Erasure coding assumes they spread | | `rustfs.clusterDomain` | `cluster.local` | Your cluster uses a custom DNS domain | | `dragonfly.extraArgs` | `[]` | Adding `--force_epoll` when `io_uring` is blocked | | `dragonfly.maxMemory` | Chart default | Keep it comfortably under the memory limit, or the kubelet kills the pod before it can evict | | `components.ui.namespace` | `gen0sec` | You install the platform chart in another namespace. Names the dashboard's service account for the Components tab | | `components.ui.serviceAccountName` | `g0s-ui` | You install the platform chart under a release name other than `g0s`. Must be `-ui` | See [Sizing storage for your retention](./../02-requirements.mdx#sizing-storage-for-your-retention). ## gen0sec-data Top-level keys: `postgres`, `kafka`, `kafka2pg`. | Key | Default | Set it when | |---|---|---| | `postgres.numberOfInstances` | `3` | Never below 3 in production | | `postgres.volume.size` | `50Gi` | Sizing the database. This is the system of record, so it is the one to grow first | | `postgres.volume.storageClass` | `""` | You want a specific StorageClass. **Fast block storage** | | `postgres.resources` | 1 vCPU / 2 GiB requested, 8 vCPU / 12 GiB limit | Tuning. The limit sets the per-node minimum | | `kafka.replicas` | `3` | Fewer nodes. **Drop the replication factors together**, or brokers refuse writes | | `kafka.storage.size` | `50Gi` | Sizing the message log and the ingest buffer | | `kafka.storage.class` | `""` | You want a specific StorageClass | :::danger[Kafka replicas and replication factor move together] `min.insync.replicas` must stay below `replicas`. Reducing brokers without reducing the factors gives you a cluster that schedules and then refuses every write. ::: ## gen0sec-platform Top-level keys: `global`, `migration`, `downloadProxy`, `services`. ### global | Key | Default | Notes | |---|---|---| | `global.registry.host` | `registry.gen0sec.com` | Rewritten by the bundler on the offline path | | `global.registry.namespace` | `gen0sec` | Every image lives under this | | `global.imageTag` | Empty | **Required.** Set it to the release version | | `global.imagePullSecrets` | `gen0sec-registry` | Must exist in both namespaces | | `global.commonEnv.S3_ENDPOINT` | The object store service URL | Must match the object store Service, which has a `-svc` suffix | | `global.commonEnv.ARXIGNIS_DATA_URL` | The same URL plus `/platform-data` | **The bucket name must match the bucket you created** | | `global.commonEnv.REDIS_URL` | The cache service URL | No credential in it, deliberately | | `global.podSecurityContext` | Non-root, uid 65532 | Do not weaken. See [Hardening](./../security/hardening.mdx) | | `global.securityContext` | Read-only root, no capabilities | Do not weaken | | `global.ingress.className` | `synapse` | The ingress controller the infra chart installs | | `global.commonEnv.CACHE_L1_TTL` | Unset, so `30s` | Default lifetime of an in-process cache entry, as a duration such as `45s`. An invalid value is ignored | ### Publishing `entrypointHost` is passed from the first install, and `externalScheme` by the [single-entrypoint overlay](./../06-publish.mdx). `values-onprem.yaml` sets neither and leaves every ingress disabled. Both are top level, and the chart applies both to every rule the overlay writes, which is why they are listed here rather than under `services`. | Key | Default | Notes | |---|---|---| | `entrypointHost` | `""` | **Required from the first install**, published or not. The one address that serves the dashboard and every agent-facing API, and the source of the dashboard's sign-in URL even before its Ingress exists. Empty fails the render and names the value to set, unless the dashboard is disabled or reached without an Ingress (see below). It must also be the address people actually type: the dashboard derives its sign-in URL from it | | `externalScheme` | Unset | Set to `https` **only when TLS terminates upstream** of the Ingress — a cloud load balancer, a CDN, a tunnel. With no `tls:` block to read, the chart otherwise infers `http` while the browser is on `https`, and every sign-in is rejected with `Invalid origin`. Leave it unset when TLS terminates at the Ingress | ### downloadProxy | Key | Default | Notes | |---|---|---| | `downloadProxy.enabled` | `true` | Turning it off is not supported | | `downloadProxy.apiKeySecret.name` | `""` | **Required. Empty is a silently broken deployment** | Full detail on [Data relay](./relay.mdx). This is the single most consequential key in the chart. ### services Twenty-one entries. Each one accepts the same shape, and you rarely need to touch any of it. | Key pattern | Notes | |---|---| | `services..enabled` | `threat-api` is not shipped at this release | | `services..replicaCount` | 2 or 3 in production, 1 in the quickstart overlay | | `services..resources` | Requests and limits. See [Sizing](./../02-requirements.mdx#sizing-a-production-install) | | `services..env` | Per-service environment, including `PORT` | | `services..ingress` | Off by default. Set by the entrypoint overlay | | `services..volumes`, `.volumeMounts` | **Do not remove these.** They exist because the read-only root filesystem needs a writable path where each service expects one | | `services.config-generator.env.DB_READ_TIMEOUT` | Unset, so `30s`. Time limit for one config build's database reads, as a duration. A build that runs out is dropped and the workspace keeps its previous config | | `services.config-generator.env.CAPABILITY_STALE_AFTER_SECONDS` | Unset, so `3600`. How long after its last report an agent stops counting toward the [connection-state check](./../troubleshoot/known-limitations.mdx#connection-state-rules-need-current-agents) | | `services.ui.enabled` | `true`. The dashboard | | `services.ui.env.BETTER_AUTH_URL` | Derived. Set it only when the dashboard is reached without an Ingress, such as through `kubectl port-forward` or your own proxy. It then wins over `entrypointHost` | | `services.ui.serviceAccount.automount` | `true`. The dashboard reads the installed-versions ConfigMaps for the Components tab. Every other service keeps `false` | | `services.ui.entraId.enabled` | `false`. Entra ID is in development and not supported at this release. See [Sign-in and identity](./identity.mdx) | :::danger[Removing a service's volume mount breaks that service] Several services default to writing into a path relative to their working directory, which fails on a read-only root filesystem. Each one has an explicit writable volume mounted where it writes. Those mounts are load-bearing, not leftovers. ::: ## Checking what a change will do Always render before you apply. ```bash helm template g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f my-overrides.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ | less ``` To diff against what is running: ```bash helm get manifest g0s -n gen0sec > running.yaml helm template g0s ... > proposed.yaml diff running.yaml proposed.yaml ``` ## Named starting points Two tested configurations rather than a blank page: [Reference configurations](./reference-configs.mdx). --- # First sign-in Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/first-signin ## Why this page exists Registration is invite-only, and invites can only be created by an existing member. A fresh install can be logged in by a pre-seeded admin account. ## The seeded account | | | |---|---| | Email | `admin@gen0sec.local` | | Password | `Gen0Sec!ChangeMe` | :::danger[This password is published, and it is the same on every install] It is a constant in our migration file, so anyone who has read our documentation knows it. The dashboard will not let you past it: the account is created with a must-change flag, so setting a new password is a step you cannot skip or postpone. What that does not do is protect the account between the moment you publish the address and the moment you first sign in. **Whoever signs in first is who sets the new password.** Sign in before you give the address to anyone else. If you have already shared it, sign in now and treat the interval as exposed. ::: ## 1. Sign in Open your published address and sign in with the details above. ```bash echo https://$ENTRYPOINT_HOST ``` If sign-in is rejected as an invalid origin, the published host or scheme does not match what you typed. See [Sign-in and identity](./configure/identity.mdx#the-published-address-decides-whether-sign-in-works). ## 2. Complete first-run setup The dashboard holds you at each of these in turn, and it will not let you skip ahead. A deep link to any other page returns you to the step you are on. 1. **Enroll multi-factor authentication.** Nothing else opens until you confirm a six-digit code. 2. **Set a new password.** Step 3 below. 3. **Create your organization.** 4. **Finish the remaining onboarding steps.** The wizard creates your first API key, then walks the WAF, access rule and monitoring surfaces. Multi-factor comes first, so the published password stops being enough to sign in before you have even reached the password screen. ## 3. Mandatory password change Because the password is preseeded, after setting up MFA, you'll be asked to change your password. :::note[The change signs out this account everywhere else] Every other session on the seeded account ends when the password changes. On a first install there are none, which is the point: if the address was reachable before you signed in, this is what closes that off. ::: The seed is also idempotent on the email address. Re-running the migration or the secret script neither duplicates the account nor re-flags it, and it does not reset a password you have already changed. ## 4. Invite the people who need access Invite your operators rather than sharing the seeded account. Under **Organization settings**, **Team**, pick the workspaces each person gets and a role for each. If [SMTP is configured](./configure/identity.mdx#invitations-and-email) the invitation is emailed. Otherwise the dashboard shows the link: copy it and send it yourself. Links expire after seven days. Invitees register with a passkey or a password, and enroll multi-factor like you did. ## 5. Mint an agent API key Create an agent API key in the dashboard, then save it where the verification step reads it from: ```bash mkdir -p ~/.gen0sec printf '%s' 'yor-agent-key' > ~/.gen0sec/agent-key ``` ## If it fails | Symptom | Cause | |---|---| | The seeded account does not exist | The migration ran without the on-prem marker. Secrets were provisioned without `make-db-secrets.sh` | | Sign-in rejected as an invalid origin | The published host or scheme does not match the address you typed | | The dashboard loads but onboarding will not advance | Check `kubectl -n gen0sec logs deploy/g0s-ui` | | The password is rejected | It is case-sensitive and contains a `!`. Watch your shell if you pasted it | | The new password is rejected as too weak | 12 characters, one symbol, and three of the four character classes. The server applies the same rules as the form | | The password screen appears again on a later sign-in | The change did not complete. The flag clears only on success | | An invitation link says it is no longer valid | It was revoked, replaced by a newer invitation, already used, expired, or issued before an upgrade from v0.1.x | --- # Install Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/install # Install ## Before you start - [ ] [Prepare the artifacts](./04-prepare-artifacts.mdx) complete - [ ] `$KIT`, `$B`, `$REGISTRY`, `$REGISTRY_USER`, `$REGISTRY_TOKEN` are set in your shell - [ ] `$G0S_DOWNLOAD_API_KEY` is set. Step 2 writes it into a secret, and an unset variable produces an empty one that fails silently - [ ] You are logged in to the registry - [ ] The address the dashboard will be reached on, in `$ENTRYPOINT_HOST`. Step 7 needs it, and [Publish the endpoints](./06-publish.mdx) publishes under it ```bash export ENTRYPOINT_HOST=cerebellum.example.internal ``` ## 1. Check the cluster ```bash $KIT/scripts/preflight.sh ``` This checks the prerequisites the charts assume but do not create: a reachable cluster, a StorageClass, and the namespaces and secrets created in step 2. On a fresh cluster it reports the namespaces and secrets as missing, which is expected at this point. Pass a StorageClass name to pin it instead of accepting the default: ```bash $KIT/scripts/preflight.sh gen0sec-system gen0sec my-block-storage ``` ## 2. Create the namespaces and the two bootstrap secrets ```bash for ns in gen0sec-system gen0sec; do kubectl create namespace $ns --dry-run=client -o yaml | kubectl apply -f - done ``` Written to converge, so re-running step 2 after a failure further down is safe rather than a wall of `AlreadyExists`. ### The image pull secret Both namespaces need it. The application runs in `gen0sec`; the operators, object store and cache run in `gen0sec-system`. ```bash for ns in gen0sec-system gen0sec; do kubectl -n $ns create secret docker-registry gen0sec-registry \ --docker-server=$REGISTRY \ --docker-username=$REGISTRY_USER \ --docker-password=$REGISTRY_TOKEN \ --dry-run=client -o yaml | kubectl apply -f - done ``` ### The object store credentials :::danger[Run this once, and only once] Unlike the rest of step 2 this does **not** converge. It mints new random keys every time it runs, and the object store's existing data was written with the old ones — recreating the secret on a second pass strands that data behind credentials nothing holds any more. The guard below stops before that happens. ```bash kubectl -n gen0sec-system get secret gen0sec-object-store >/dev/null 2>&1 \ && echo "already exists — skip to step 3" \ || echo "not present — run the block below" ``` ::: ```bash umask 077 S=$(mktemp -d) openssl rand -hex 16 | tr -d '\n' > "$S/RUSTFS_ACCESS_KEY" openssl rand -hex 32 | tr -d '\n' > "$S/RUSTFS_SECRET_KEY" kubectl -n gen0sec-system create secret generic gen0sec-object-store --from-file="$S" ``` **Save both files somewhere safe before you delete them.** Nothing regenerates these keys, and the data in the object store is written with them. ```bash mkdir -p ~/gen0sec-object-store-keys cp "$S"/RUSTFS_* ~/gen0sec-object-store-keys/ # or load them into your secret manager rm -rf "$S" ``` Step 5 reads them back from that directory. ### Your Gen0Sec API key The relay presents this key to Gen0Sec when it fetches threat intelligence, GeoIP, models and IDS rules. The chart expects this exact secret name, so the relay pod will not start without it. See [Credentials from gen0sec](./02-requirements.mdx#your-gen0sec-api-key). ```bash # Refuses to run rather than creating an empty secret. An empty key is accepted # by Kubernetes and by the chart, and only surfaces later as every relay fetch # returning 401 across the whole fleet — with nothing in the install log. : "${G0S_DOWNLOAD_API_KEY:?not set — see Prepare the artifacts, step 0}" kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-literal=DOWNLOAD_PROXY_API_KEY=$G0S_DOWNLOAD_API_KEY ``` ## 3. Install infra ```bash helm install g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml ``` This installs three operators (Postgres, Kafka, ingress), the object store and the shared cache. ### Wait for all five The next step creates resources these operators have to reconcile. Do not continue until every one is available. ```bash kubectl -n gen0sec-system rollout status deploy/g0s-infra-postgres-operator kubectl -n gen0sec-system rollout status deploy/strimzi-cluster-operator kubectl -n gen0sec-system rollout status deploy/g0s-infra-synapse-operator kubectl -n gen0sec-system rollout status deploy/dragonfly kubectl -n gen0sec-system rollout status \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" ``` The output for each is similar to this: ``` deployment "g0s-infra-postgres-operator" successfully rolled out ``` :::note[Why the last one resolves its own kind] The object store is a StatefulSet in the production configuration and a Deployment in the single-node configuration. Resolving the kind means the same command works in both. ::: ## 4. Install data ```bash helm install g0s-data $B/charts/gen0sec-data-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml ``` This creates the Postgres cluster, the Kafka cluster, its topics, and the database roles. ### Wait for both clusters Both take several minutes. Nothing after this works until Postgres reports `Running`. ```bash kubectl -n gen0sec-system wait --for=condition=Ready kafka/core --timeout=15m kubectl -n gen0sec-system wait \ --for=jsonpath='{.status.PostgresClusterStatus}'=Running postgresql/core --timeout=15m ``` ## 5. Create the object store bucket The object store starts empty and nothing creates the bucket for you. Create it now, before the platform install. Services that read or write the object store fail until it exists. This step is required on every install. The two datasets your cluster produces about itself, `identity` and `policy-edges`, live here and are never sent anywhere. See [Network and connectivity](./03-network.mdx#what-never-leaves-your-cluster). Run it as a throwaway pod in the cluster. The pod reads the keys straight from the `gen0sec-object-store` secret, so they never reach your shell and never touch your local AWS configuration. ```bash kubectl -n gen0sec-system run s3-init --rm -i --restart=Never \ --image=amazon/aws-cli:2.15.0 \ --overrides='{"spec":{"containers":[{"name":"s3-init","image":"amazon/aws-cli:2.15.0", "env":[{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"name":"gen0sec-object-store","key":"RUSTFS_ACCESS_KEY"}}}, {"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"name":"gen0sec-object-store","key":"RUSTFS_SECRET_KEY"}}}, {"name":"AWS_EC2_METADATA_DISABLED","value":"true"}, {"name":"AWS_REGION","value":"us-east-1"}], "args":["--endpoint-url","http://rustfs-svc.gen0sec-system.svc.cluster.local:9000","s3","mb","s3://platform-data"]}]}}' ``` Expect `make_bucket: platform-data`. `BucketAlreadyOwnedByYou` is also a pass, so this step converges: running it twice is harmless. :::note[Offline installs: mirror this one image first] `amazon/aws-cli` is **not** in the offline bundle. The bundle carries only what `release-manifest.yaml` pins, and its `SHA256SUMS` covers the whole tree — the bundler refuses a package containing anything the checksums do not list — so it cannot be added after the fact. Mirror it alongside the bundle, then point the pod at your copy: ```bash skopeo copy docker://amazon/aws-cli:2.15.0 docker://$REGISTRY/amazon/aws-cli:2.15.0 ``` Substitute `$REGISTRY/amazon/aws-cli:2.15.0` for `amazon/aws-cli:2.15.0` in both the `--image` flag and the `image` field of the overrides above. They must match: the `--image` flag sets the pod's image, and the override object sets the container's, and Kubernetes uses the latter. ::: Verify by running the same command with `"s3","ls"` in place of `"s3","mb","s3://platform-data"`. The bucket should be listed: ``` 2026-09-01 14:02:11 platform-data ``` An empty result means the bucket was not created, and the services in step 7 will fail with `500 NoSuchBucket`. Do not continue. :::note[If you changed the bucket name] `platform-data` is the value of `global.commonEnv.S3_BUCKET`. If you overrode it, use your name instead. ::: :::caution[Do not export the object-store keys into your shell] The in-cluster method above is recommended because the credentials never enter your shell. That matters most on managed Kubernetes. **On EKS, exporting them breaks `kubectl`.** The kubeconfig authenticates through an `aws eks get-token` exec plugin that reads `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` — the same two variables. Setting them to the object-store keys makes every later command in that shell fail with: ``` error: You must be logged in to the server (Unauthorized) ``` The error names kubectl, not the variables, which makes it hard to attribute. The in-cluster method avoids it entirely. If you use the workstation method below, scope the credentials to the one command with `env ...` rather than exporting them. **If the object store returns `401`,** the keys are not the ones this install generated. Keys left over from a previous install of the same cluster do this, and so does a stale copy in `~/gen0sec-object-store-keys/`. Credentials are genuinely validated: wrong keys return `InvalidAccessKeyId`, and none at all returns `403`. :::
Workstation method, for reference This needs no S3 client image in the cluster, which is why it was preferred: on the offline bundle method an in-cluster image has to be mirrored first, so the bucket comes to depend on a step that exists only to create the bucket. Use it if that trade-off suits you — but read the caution above first, because exporting the object-store keys will break `kubectl` on EKS. ### 5a. Open a port forward In a second terminal: ```bash kubectl -n gen0sec-system port-forward svc/rustfs-svc 9000:9000 ``` Leave it running. The output is similar to this: ``` Forwarding from 127.0.0.1:9000 -> 9000 ``` ### 5b. Create the bucket Back in your first terminal, use the two keys you generated in step 2. Scope them to the one command with `env` rather than exporting them, so they cannot clobber the variables your kubeconfig's exec plugin reads: ```bash env AWS_ACCESS_KEY_ID="$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ AWS_SECRET_ACCESS_KEY="$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" \ AWS_REGION=us-east-1 AWS_EC2_METADATA_DISABLED=true \ aws --endpoint-url http://localhost:9000 s3 mb s3://platform-data ``` The output is similar to this: ``` make_bucket: platform-data ``` `BucketAlreadyOwnedByYou` is also a pass. With the MinIO client instead: ```bash mc alias set g0s http://localhost:9000 \ "$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ "$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" mc mb g0s/platform-data mc ls g0s ``` ### 5c. Verify it exists ```bash env AWS_ACCESS_KEY_ID="$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ AWS_SECRET_ACCESS_KEY="$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" \ AWS_REGION=us-east-1 AWS_EC2_METADATA_DISABLED=true \ aws --endpoint-url http://localhost:9000 s3 ls ``` The output is similar to this: ``` 2026-09-01 14:02:11 platform-data ``` An empty result means the bucket was not created, and the services in step 7 will fail with `500 NoSuchBucket`. Do not continue. Then stop the port forward with `Ctrl-C` in the second terminal.
## 6. Create the application secrets ```bash $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml ``` This composes around twenty secrets from the passwords the Postgres operator generated in step 4. That is why it cannot run earlier. Each service gets its own database role through the connection pooler. The migration job gets a superuser connection direct to the primary, because it runs `CREATE EXTENSION`. :::note[In production, treat this script's output as a specification] `make-db-secrets.sh` is fine for evaluation. For production, have your secret manager produce secrets of the same shape. Every name and key is listed in [Secrets](./configure/secrets.mdx). ::: ## 7. Install the platform ```bash helm install g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` **`entrypointHost` is required here**, before anything is published. The dashboard builds its sign-in URL from it, and without it the render fails with `entrypointHost is required`. Use the address people will type, the one you publish under in the next step. :::note[Nothing to set for the relay] The chart already points at `gen0sec-download-proxy` by default. You created that secret in step 2, and if it is missing the relay pod fails to start with `CreateContainerConfigError` rather than coming up half-configured. ::: Schema migrations run first, as a pre-install hook. If they fail, Helm aborts and nothing is deployed. That is deliberate: a partially migrated database is worse than no install. ### Check the migration ```bash kubectl -n gen0sec get job g0s-db-migrate ``` The output is similar to this: ``` Error from server (NotFound): jobs.batch "g0s-db-migrate" not found ``` **`NotFound` is the success case.** The job is a Helm hook with `hook-delete-policy: hook-succeeded`, so Helm deletes it the moment it passes. If the job is still there, it may have failed: ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` ### Wait for the pods ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m ``` :::note[Check readiness, not phase] A pod in `CrashLoopBackOff` still reports `status.phase: Running`. A `--field-selector` on phase reports a broken install as clean. ::: :::note[Two values that must match your cluster] Both are set in `values-onprem.yaml` and both point at the in-cluster object store. Check them if anything cannot read its data. | Key | Must be | |---|---| | `global.commonEnv.S3_ENDPOINT` | `http://rustfs-svc.gen0sec-system.svc.cluster.local:9000` | | `global.commonEnv.ARXIGNIS_DATA_URL` | The same endpoint, plus the bucket: `.../platform-data` | `ARXIGNIS_DATA_URL` is where in-cluster services read their datasets from. It points at your object store, not at Gen0Sec. **The bucket name at the end of it must match the bucket you created in step 5.** If you used a different name there, change it here too. The Service is `rustfs-svc`, with the `-svc` suffix. That suffix is the single most common cause of services that come up and then cannot read anything. ::: ## If it fails | Symptom | Start here | |---|---| | `helm install` fails with `entrypointHost is required` | `entrypointHost` was not passed in step 7. Set it and re-run | | `ImagePullBackOff` | [ImagePullBackOff](./troubleshoot/symptoms.mdx#imagepullbackoff) | | Install aborts at `g0s-db-migrate` | [Migration failures](./troubleshoot/symptoms.mdx#migration-failures) | | `postgresql/core` never reaches `Running` | [Postgres will not start](./troubleshoot/symptoms.mdx#postgres-will-not-start) | | Kafka broker crash loops | [Kafka will not start](./troubleshoot/symptoms.mdx#kafka-will-not-start) | --- # Network and connectivity Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/network # Network and connectivity ## Why the platform must reach Gen0Sec No platform is useful without data, neither is Gen0Sec. Because the datasets are licensed per customer, they are not in the install kit, not in the offline bundle, and not in any container image. The platform fetches them at runtime from `api.gen0sec.com`. | Dataset | Licensed by | |---|---| | CTI indicators | Upstream threat intelligence providers | | GeoIP databases | MaxMind, IPInfo | | IDS rules | Emerging Threats ruleset | | ML models | Trained and published by Gen0Sec | One in-cluster service, `download-api`, does the fetching. It relays each request upstream and streams the response straight back. It is a relay, not a cache, so there is no artifact bucket to size, no sync job to supervise and no staleness window. An agent that polls gets whatever upstream has at that moment. This also means you get the most recent data we can provide. ## One hop, not many Your agents never talk to the internet. They talk to `download-api`, and `download-api` is the only service in either namespace that needs a route out. The dashboard makes optional calls of its own, listed in the [egress list](#egress-list); none of them carries agent data. ``` agents -> download-api -> api.gen0sec.com:443 ^ | the only egress point in the deployment ``` That means one destination to allow, one to audit. It also means the relayed URLs are identical to the direct ones, so agents need no configuration change. ## What never leaves your cluster Two datasets describe your own infrastructure, and they are never relayed: | Dataset | What it contains | Where it stays | |---|---|---| | `identity` | Workload identity produced in your cluster | Your object store | | `policy-edges` | Pod IPs, namespaces, NetworkPolicy edges | Your object store | No hidden switches, no configuration options, it is not implemented. Your data is yours and yours only. Uploads are not relayed either. The relay is read-only, so `PUT` and `POST` on the download surface are served locally or not at all. ## How authorization works **Your agents' API keys never leave your cluster.** Agents authenticate to your installation with keys your own dashboard issued, held in your own database. On a correct configuration the relay does not forward those keys upstream. It presents **your own Gen0Sec API key**, the one issued with your subscription, on the second hop only. ``` agent --[your agent's key]--> relay --[your Gen0Sec API key]--> api.gen0sec.com agent key stops here ``` | | | |---|---| | What Gen0Sec sees | One credential, belonging to your installation | | What Gen0Sec never sees | Any individual agent key, or how many you have issued | | Revoking an agent | Entirely local, in your dashboard | | Revoking your installation's access | Revoke or rotate your Gen0Sec API key | :::danger[This key is required, and omitting it fails silently] With no Gen0Sec API key configured, the relay falls back to forwarding the caller's own key. That is correct for the hosted product, where the same platform issued it. Here it is always wrong: your keys exist only in your cluster, so Gen0Sec rejects every one with a 401, and **the agent reports an empty ruleset rather than an error**. The install may look healthy, but the fleet quietly stops receiving threat intelligence, GeoIP, models and IDS rules. Set up in [Install](./05-install.mdx) step 2, enabled in step 7, and checked by [Verify the install](./08-verify.mdx). ::: Routes that are not relayed, including `identity` and `policy-edges`, keep the local authorization check against your own database. ## Egress list All connections are outbound, TCP, TLS. Nothing needs an inbound rule from the internet. | Destination | Port | Needed by | Purpose | Required | |---|---|---|---|---| | `api.gen0sec.com` | 443 | Cluster pods (`download-api` only) | Threat intelligence, GeoIP, ML models, IDS rules | **Always** | | `releases.gen0sec.com` | 443 | Your workstation | Download the install kit and its signature | **Always** | | `registry.gen0sec.com` | 443 | Your workstation | `helm pull` for charts, `skopeo copy` for images | **Always** | | `registry.gen0sec.com` | 443 | Cluster nodes (kubelet) | Image pulls at install time | Registry install only | | `fulcio.sigstore.dev` | 443 | Your workstation | `cosign` certificate verification | **Always** | | `rekor.sigstore.dev` | 443 | Your workstation | `cosign` transparency log verification | **Always** | | `tuf-repo-cdn.sigstore.dev` | 443 | Your workstation | `cosign` trust root refresh | **Always** | | Your own registry | 443 or 80 | Cluster nodes, and the host that runs `bundler.sh apply` | Image pulls at install time | Offline bundle only | | `helm.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest agent version, shown on the Clients and Components pages | No. Always attempted. Blocking it only hides the latest version | | `releases.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest component versions | No. Only if an administrator turns on **Check for updates online** | | Your SMTP server | 587 or 465 | Cluster pods, the dashboard only | Invitation and password-reset email | No. Only if you [configure SMTP](./configure/identity.mdx#invitations-and-email) | {/* ASSUMPTION: this list is derived from the release workflow, the bundler and the chart values. Confirm it is complete, and confirm the Sigstore host set, which changes with cosign versions. */} :::note[The Sigstore hosts are the ones people forget] `cosign verify-blob` checks a signature against a public transparency log. Without those three destinations it fails, and it fails on the first command of the install. Allow them on the workstation, not on the cluster. ::: ### Inside the cluster No NetworkPolicy is required for the install. If you enforce a default-deny policy, the two namespaces need to reach each other: the application in `gen0sec` talks to Postgres, Kafka, the object store and the cache in `gen0sec-system`. ## What this rules out `api.gen0sec.com` on port 443 is required for the platform to serve data. An agent fleet with the platform hosted on a cluster that is permitted no outbound connections at all cannot function correctly. The offline bundle method removes every dependency on gen0sec-hosted registries: your cluster pulls images only from your own. It does not remove the runtime connection, because the mentioned datasets above are not ours to ship. If your policy allows no cluster egress, talk to us. We would rather hear it now than during your evaluation. > **Air-gapped.** Often used to mean "no connection of any kind". Under that definition Cerebellum > is not deployable. If it means "no dependency on vendor-hosted registries and one audited > outbound destination", the offline bundle method fits, and this page is the list your firewall team > needs. --- # Back up and restore Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/operate/backup # Back up and restore ## What to back up Not everything in the deployment is equally valuable. Backing up the wrong things is how backup jobs become expensive and untested. | Component | Back up | If you lose it | |---|---|---| | **Postgres** | **Yes, always** | Everything. Configuration, agents, policy, history, the audit log. This is the system of record | | **Object store** | **Yes** | The datasets and the artifacts your cluster produced about itself. Datasets are re-fetchable, `identity` and `policy-edges` are not | | Kafka | No | In-flight messages only. It is a transport, not a store. Producers re-send | | Shared cache | No | Nothing. It repopulates | | Secrets | **Yes** | Access to your own volumes. See the warning below | :::danger[Back up the volumes along with the secrets, or the volumes are useless] The database passwords are generated by the Postgres operator and stored as Kubernetes secrets. Data is written with those credentials. A volume backup without the matching secret backup restores data you cannot open. ::: ## Postgres The Postgres operator ships a logical backup image, already mirrored as part of the release. Enable it in the infra values rather than scripting `pg_dump` yourself. ```yaml # gen0sec-infra values postgres-operator: configLogicalBackup: logical_backup_schedule: "0 2 * * *" logical_backup_provider: s3 ``` Verify a backup ran: ```bash kubectl -n gen0sec-system get cronjob | grep logical-backup kubectl -n gen0sec-system get job | grep logical-backup ``` :::note[A backup you have not restored is not a backup] Restore into a scratch namespace once, before you need to. The [Quickstart](./../quickstart.mdx) cluster is a reasonable place to rehearse it, and rehearsing a restore is also a rehearsal of an upgrade rollback. ::: ## Object store Two prefixes matter, and they matter differently. | Prefix | Recoverable without a backup? | |---|---| | `identity/`, `policy-edges/` | **No.** Produced by your cluster about your cluster | | Dataset prefixes | Yes. They are fetched from Gen0Sec | Back up the whole bucket with any S3-compatible tool, with an in-cluster job or manually, over a port forward as in [Install step 5](./../05-install.mdx#5-create-the-object-store-bucket): ```bash kubectl -n gen0sec-system port-forward svc/rustfs-svc 9000:9000 ``` ```bash aws --endpoint-url http://localhost:9000 s3 sync s3://platform-data ./platform-data-backup ``` Restore is the same command with the arguments reversed. ## Secrets ```bash kubectl -n gen0sec-system get secret -o yaml > secrets-gen0sec-system.yaml kubectl -n gen0sec get secret -o yaml > secrets-gen0sec.yaml ``` **These files contain live credentials in base64.** Treat them exactly as you treat the database itself: encrypted at rest, access controlled, and out of any shared drive. If you run an external secret manager, you do not need this: your manager is already the source of truth and the cluster holds copies. See [Secrets](./../configure/secrets.mdx). ## Recovery point and recovery time | | Value | Determined by | |---|---|---| | **Recovery point** | Your backup interval | The logical backup schedule. Nightly gives up to 24 hours of loss | | **Recovery time** | Measure it | Database size and your storage throughput. Rehearse it and record your own figure | We do not publish an RTO as it depends entirely on your configuration specifics and the data size. Time your own rehearsal and use that number in your own runbook. ## Restoring Order matters, and it is the install order. 1. Install infra and data as on a [first install](./../05-install.mdx), so the operators and empty clusters exist. 2. Restore the secrets, before anything tries to use the volumes. 3. Restore Postgres from the logical backup. 4. Restore the object store bucket. 5. Install the platform. **The migration hook runs against restored data**, so the restored schema must match the release you are installing. Restore into the version it came from, then upgrade. 6. Run the smoke test: `$KIT/scripts/verify-deployment.sh`. :::note[Do not restore an old database into a new release] The migration chain expects to run forward from the schema version in the dump. Restoring a v0.1 backup and installing v0.3 skips the intermediate migrations. Restore v0.1, then upgrade one minor at a time. See [Upgrade](./upgrade.mdx). ::: --- # Monitor Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/operate/monitor # Monitor ## What the platform exposes today | Signal | Where | Notes | |---|---|---| | Health and readiness | `/status/healthz` and `/status/readyz` on each service | Three services serve only `healthz`. See [Known limitations](./../troubleshoot/known-limitations.mdx#three-services-report-readiness-from-a-weaker-probe) | | Relay mode and upstream | `/status/healthz` on the relay | Returns the mode, the upstream and the active relay classes | | Prometheus metrics | Worker port 8080 | Exposed, but no Service is created for workers, so scraping needs a direct pod target | | Kubernetes state | Pods, Deployments, custom resources | The primary signal | | Block events refused by the database | Kafka topic `block_events_dlq`, 14 days retention | Each message carries the reason in its `dlq_reason` header. Steady growth means events are being lost | | Block events dropped at ingest | telemetry-api log line `OTLP block events rejected, or stored without a field they carried` | At most once a minute per organization, with counts per reason | | Installed and latest versions | Dashboard, **Organization settings**, **Components** | See [Component versions](./../configure/components.mdx#verifying-what-you-are-actually-running) | | Who changed what | Dashboard, **Organization settings**, **Audit Log** | Owner and Admin roles only | **No ServiceMonitor, alert rules or dashboards ship with the charts at the moment.** If you run Prometheus, you build the scrape configuration. ## The four checks worth alerting on If you set up nothing else, set up these. ### 1. Pods not ready ```bash kubectl -n gen0sec get pods --no-headers \ | awk '{split($2,r,"/"); if (r[1] != r[2]) print}' ``` Any output is a problem. **Read the ready column, never a field selector on phase:** a pod in `CrashLoopBackOff` still reports `status.phase: Running`. ### 2. The database cluster is not Running ```bash kubectl -n gen0sec-system get postgresql core \ -o jsonpath='{.status.PostgresClusterStatus}' ``` Expect `Running`. Anything else and nothing downstream works. ### 3. The relay lost its upstream ```bash kubectl -n gen0sec exec deploy/g0s-download-api -- \ curl -fsS localhost:9999/status/healthz ``` Expect `"status": "ready"` and `"mode": "proxy"`. An unreachable upstream answers **502** on relayed routes rather than 404 or 500. That is deliberate: a polling agent reads 502 as "ask again later" and keeps the artifact it already holds, instead of treating it as withdrawn. So a broken upstream degrades your fleet slowly rather than immediately, which is safer and also easier to miss. Alert on it. ### 4. The platform stopped serving agents ```bash $KIT/scripts/verify-deployment.sh ``` Read-only, safe against production, exits non-zero and names the failing check. Running it on a schedule is the single most useful synthetic check available, because it is the only one that proves data actually moves. ## Signals that are noisy on purpose Do not page on these. | Signal | Why it is not an incident | |---|---| | `falling back to L1-only cache` | The shared cache is never fatal. Services run on their in-process cache. Reads get slower, nothing breaks | | A 502 from a relayed route, once | Upstream hiccup. Agents retry and keep what they have | | The migration job absent | That is success. Helm deletes it on completion | | `unknown_source` in a telemetry-api warning | An agent newer than the platform. The event is stored, without its source | ## Logs ```bash kubectl -n gen0sec logs deploy/ --tail=200 kubectl -n gen0sec-system logs deploy/g0s-infra-postgres-operator --tail=200 ``` All services log to standard output as structured JSON, so any collector that reads container stdout works without configuration. Log level is set per service through the values file. See [Values reference](./../configure/values.mdx). ## What a healthy install looks like Five statements, the same five as [Verify the install](./../08-verify.mdx#definition-of-installed-successfully). If you build a dashboard, build it to answer these: 1. Every pod in both namespaces is `Running` and fully ready. 2. `postgresql/core` reports `Running`. 3. `kafka/core` reports `Ready`. 4. The relay reports `"mode": "proxy"` with your upstream. 5. The smoke test exits zero. --- # Roll back Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/operate/rollback # Roll back Read the boundary section first. It decides whether a rollback is safe or whether you need a restore. ## The boundary, stated plainly **`helm rollback` reverts manifests. It does not revert your database.** That single sentence is the whole risk. Helm restores the previous Deployments, images and configuration. Any schema change the upgrade's migration hook applied is still applied. If the old application cannot read the new schema, rolling back the manifests gives you old code against a new database. | Situation | Rollback is | |---|---| | Patch upgrade within a train, no migration ran | Safe. Just do it | | Minor upgrade whose migrations only added things | Safe. Old code ignores what it does not know | | Minor upgrade whose migrations removed or renamed something the old code reads | **Not safe.** Restore from backup instead | | You are not sure which | Check the release notes. If they do not say, ask us before rolling back | :::note[v0.2.2 back to v0.2.1] Rolling back the platform alone is safe: v0.2.1 works against the v0.2.2 schema. **Do not run the down migrations.** They delete the CVE-2026-109618 and CVE-2026-109619 managed WAF rules and every workspace's switch for them. Rolling back brings the firewall defect back: rules with a connection-state condition again reach agents without it, and match every packet. Remove those rules first. Keep infra on v0.2.2: rolling it back restarts Postgres again and leaves arm64 nodes unable to run it. While rolled back, do not `helm upgrade` the v0.2.1 chart: its migration job does not know the v0.2.2 schema version and fails with `no migration found for version`. ::: :::note[v0.2.1 back to v0.2.0] Safe, and there is nothing to run: all 17 v0.2.1 down migrations are no-ops, and v0.2.0 works against the v0.2.1 schema. Both v0.2.1 fixes live in the database, so rolling back does not bring either defect back. While rolled back, do not `helm upgrade` the v0.2.0 chart: its migration job does not know the v0.2.1 schema version and fails with `no migration found for version`. ::: :::note[v0.2.0 back to v0.1.x] Rolling back the platform alone is safe: every v0.2.0 migration only adds, so v0.1.x runs against the v0.2.0 schema. **Do not run the down migrations.** There are 18, and they delete the audit log, registered passkeys and pending invitations. While rolled back, do not `helm upgrade` the v0.1.x chart: its migration job does not know the v0.2.0 schema version and fails with `no migration found for version`. ::: :::note[Why we prefer expand and contract] Migrations that add in one train and remove in a later one mean a one-release rollback never needs a schema change. That is the design intent, and it is what makes most rollbacks safe. It is not a guarantee for every release, which is why this page exists rather than a one-line reassurance. ::: ## Before you start - [ ] You know which revision you are going back to - [ ] You have read the release notes for the version you are leaving - [ ] The previous release's images are still in your registry **Offline bundle method:** mirroring is additive, so the old tags are still there unless you deleted them. Do not prune your registry between an upgrade and the point at which you are confident. ## 1. Find the revision ```bash helm history g0s -n gen0sec ``` The output is similar to this: ``` REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Mon Sep 1 10:14:02 2026 superseded gen0sec-platform-0.1.0 0.1.0 Install complete 2 Mon Sep 1 15:41:18 2026 deployed gen0sec-platform-0.2.0 0.2.0 Upgrade complete ``` ## 2. Roll back the platform Start here. In most cases this is the only step you need, because the application is what broke. ```bash helm rollback g0s 1 -n gen0sec ``` ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` If the smoke test passes, stop. Do not roll back infra or data unless they are the problem. ## 3. Roll back the schema, only if you must Needed when the upgrade applied a change the old code cannot work with. Check the release notes first. ```bash kubectl -n gen0sec run db-migrate-down --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/db-migrate: \ --env DATABASE_URL="$OWNER_DSN" -- down ``` Use the `db-migrate` image of the release you are **leaving**, not the one you are returning to. Only it contains the down migrations for its own schema; an older image fails with `no migration found for version `. `down` reverts one migration per run, so run it once per migration the release added. **Never run `down all`**: it reverts the entire schema. `$OWNER_DSN` is the superuser connection string from the `gen0sec-db-migrate` secret. See [Secrets](./../configure/secrets.mdx). :::danger[A down migration can lose data] Reversing a migration that dropped a column recreates the column, not its contents. If the upgrade removed data, only a restore brings it back. See [Back up and restore](./backup.mdx). The v0.2.0 down migrations drop the audit log, passkeys and pending invitations, and clear the `fleet_ban` source from stored block events. One of them also decompresses recent block events and locks that table while it runs. ::: ## 4. Roll back infra and data, only if they are the problem Same order as the upgrade, in reverse: platform, then data, then infra. ```bash helm rollback g0s-data 1 -n gen0sec-system helm rollback g0s-infra 1 -n gen0sec-system ``` :::note[CRDs do not roll back] Helm never manages CRDs after the first install, in either direction. A rolled-back operator runs against the newer CRD schema. That works when the change was additive, which is the usual case, and it is another reason to roll back the platform alone where possible. ::: ## When a restore is the right answer instead Rollback is for "the new version misbehaves". Restore is for "the data is wrong". | You want to | Use | |---|---| | Undo a bad release | Rollback | | Undo a destructive migration | Restore | | Recover from data corruption | Restore | | Return to a point in time | Restore | [Back up and restore](./backup.mdx). ## If it fails | Symptom | Cause | |---|---| | `ImagePullBackOff` after rollback | The previous version's images were pruned from your registry | | Old pods start, then error on database queries | Schema mismatch. Go to step 3, or restore | | `helm rollback` reports no such revision | `helm history` was truncated. `helm get manifest g0s --revision N` to confirm what exists | ## Next [Uninstall](./uninstall.mdx), or [Back up and restore](./backup.mdx). --- # Rotate credentials Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/operate/rotate # Rotate credentials Credentials rotates differently, and some of them have a consequence worth knowing before you start. | Credential | Rotating it interrupts | Reversible | |---|---|---| | [Database role passwords](#database-role-passwords) | The services that use them, briefly | Yes | | [Object store keys](#object-store-keys) | Everything that reads the object store | Yes, **if you kept the old values** | | [Registry pull credentials](#registry-pull-credentials) | Nothing running. New pulls only | Yes | | [Your Gen0Sec API key](#your-gen0sec-api-key) | Artifact delivery, briefly | Yes | | [Dashboard session signing key](#dashboard-session-signing-key) | **Every session and every invite link** | No | If you run an external secret manager, rotation happens there and the cluster picks up the new value. This page is for installs holding plain Kubernetes secrets. See [Secrets](./../configure/secrets.mdx). ## Database role passwords Rotate through the Postgres operator, not by editing the secret. The operator owns these values, and an edited secret is overwritten on its next reconciliation. After the operator has regenerated the credentials, recompose the application secrets and restart the consumers: ```bash $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy ``` The script preserves the dashboard's session signing key, so re-running it does not sign people out. ## Object store keys These are the ones you generated during install, and **nothing regenerates them**. Rotating means updating two places. ```bash umask 077 S=$(mktemp -d) openssl rand -hex 16 | tr -d '\n' > "$S/RUSTFS_ACCESS_KEY" openssl rand -hex 32 | tr -d '\n' > "$S/RUSTFS_SECRET_KEY" kubectl -n gen0sec-system delete secret gen0sec-object-store kubectl -n gen0sec-system create secret generic gen0sec-object-store --from-file="$S" ``` Then restart the object store, and recompose the application secrets, because the services receive the same values under different names: ```bash kubectl -n gen0sec-system rollout restart \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy ``` Save the new files where you kept the old ones, then remove the temporary directory. :::danger[Do not lose these] Data in the object store was written with them. A retained volume without the matching keys is data you cannot open. See [Back up and restore](./backup.mdx). ::: ## Registry pull credentials Recreate the secret in **both** namespaces. Running pods keep running; the new value applies to the next pull. ```bash umask 077 AUTH=$(printf '%s:%s' "$REG_USER" "$(cat $REG_PASS_FILE)" | base64 | tr -d '\n') printf '{"auths":{"%s":{"auth":"%s"}}}' "$REGISTRY" "$AUTH" > dockerconfig.json unset AUTH for ns in gen0sec-system gen0sec; do kubectl -n $ns delete secret gen0sec-registry kubectl -n $ns create secret generic gen0sec-registry \ --type=kubernetes.io/dockerconfigjson \ --from-file=.dockerconfigjson=dockerconfig.json done rm -f dockerconfig.json ``` Rotate before the old credential expires, not after. An expired pull secret is invisible until something reschedules. ## Your Gen0Sec API key ```bash printf '%s' '' > $RELAY_KEY_FILE kubectl -n gen0sec delete secret gen0sec-download-proxy kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-file=DOWNLOAD_PROXY_API_KEY=$RELAY_KEY_FILE kubectl -n gen0sec rollout restart deploy/g0s-download-api ``` Confirm it worked, because a wrong key fails silently: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means the new key is wrong, or has a trailing newline. Use `printf`, not `echo`. See [Data relay](./../configure/relay.mdx). ## Dashboard session signing key :::danger[This signs everyone out] Rotating it invalidates every active session and every outstanding invite link. There is no partial rotation and no grace period. ::: Only do this if you believe the key is compromised. Delete it and let the secret script regenerate it: ```bash kubectl -n gen0sec delete secret g0s-db-ui $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy/g0s-ui ``` The script preserves an existing key, which is why deleting the secret first is the step that actually rotates it. ## Certificates | Certificate | Owned by | |---|---| | Ingress TLS | You. Rotate in your ingress or through cert-manager | | Postgres pooler | The Postgres operator, self-signed, rotated by it | | Relay to Gen0Sec | Public certificate on our side. Nothing to do | The platform presents no certificate of its own outside the ingress you configure. ## After any rotation ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` The smoke test is the only check that proves the platform still serves an agent rather than merely starting. --- # Uninstall Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/operate/uninstall # Uninstall ## What you are about to delete | | Default | `--keep-data` | `--keep-crds` | |---|---|---|---| | The three Helm releases | Removed | Removed | Removed | | Objects the operators created outside the releases | Removed | Removed | Removed | | Kafka topic finalizers | Cleared | Cleared | Cleared | | Namespaces `gen0sec` and `gen0sec-system` | **Deleted** | Kept | Deleted | | Volumes: Postgres, Kafka, object store | **Deleted** | Kept | Deleted | | Secrets, including the database passwords | **Deleted** | Kept | Deleted | | Operator CRDs | **Removed** | Removed | Kept | :::danger[The default deletes your data, and it asks you to prove you mean this cluster] Without `--keep-data` the databases and the object store go, and nothing can undo that. So that run also **asks you to type the current context name back**, and refuses to run at all with no terminal attached. `--keep-crds` does not trigger that prompt, because a schema the next install reinstalls anyway is no reason to skip asking about the databases. ::: ## Remove everything ```bash $KIT/scripts/uninstall.sh --yes ``` It prints the cluster context, the release list and the volume count, then waits for you to type the context name. Read what it prints before you type. ## Keep the data ```bash $KIT/scripts/uninstall.sh --yes --keep-data ``` Stops after clearing the topic finalizers. The namespaces stay, and so do the volumes and the secrets in them. No context confirmation, because nothing irreversible happens. :::note[Why the secrets have to stay with the volumes] A reinstall over retained volumes needs the original database passwords: the data was written with them and a fresh Postgres operator generates different ones. Keeping the volumes without the secrets leaves you with data you cannot open. ::: ## Keep the CRDs ```bash $KIT/scripts/uninstall.sh --yes --keep-crds ``` Use this when something else in the cluster runs the same operators, or when you want the next install to skip reinstalling schemas. ## How CRD removal is made safe CRDs are cluster-scoped, and **deleting one deletes every resource of that kind in every namespace**. Another Postgres or Kafka cluster run by the same operators would go with them. So the script checks first: if any of those kinds is in use outside the two namespaces it is removing, **it skips the CRDs** and tells you. `--keep-crds` skips them unconditionally. That check is why removing CRDs by default is safe. It is not a blanket delete. If you want to see what would be affected before you run anything: ```bash kubectl get postgresqls,kafkas,kafkatopics,kafkanodepools -A 2>/dev/null | grep -v 'gen0sec' ``` Any output there means the script will leave the CRDs alone. ## The Kafka finalizer, and why order matters Strimzi puts a `strimzi.io/topic-operator` finalizer on all 23 KafkaTopics. Once the operator is gone, nothing clears it, and deleting the namespace leaves it in `Terminating` indefinitely. The script uninstalls the releases first, then clears the finalizers, then deletes the namespaces. That order is the whole reason the script exists. If you already deleted the namespace and it is stuck, this is the fix: ```bash NS=gen0sec-system kubectl get kafkatopics.kafka.strimzi.io -n $NS -o name \ | xargs -r -n1 kubectl patch -n $NS --type=merge \ -p '{"metadata":{"finalizers":null}}' ``` ## Reinstalling over retained volumes Two things bite here. **Kafka.** The volumes survive on purpose, and a recreated cluster generates a new identity. Brokers then crash loop with `Invalid cluster.id`. See [Kafka will not start](./../troubleshoot/symptoms.mdx#kafka-will-not-start). **Postgres.** You need the original role passwords. If you kept the secrets, the operator adopts the existing data. If you did not, the volumes are unreadable. ## Uninstalling by hand The script does more than these commands: it also removes objects the operators created outside the releases, and it handles the CRD safety check. Use it if you can. ```bash helm uninstall g0s -n gen0sec helm uninstall g0s-data -n gen0sec-system helm uninstall g0s-infra -n gen0sec-system # clear the finalizers before touching the namespaces kubectl get kafkatopics.kafka.strimzi.io -n gen0sec-system -o name \ | xargs -r -n1 kubectl patch -n gen0sec-system --type=merge \ -p '{"metadata":{"finalizers":null}}' kubectl delete namespace gen0sec kubectl delete namespace gen0sec-system ``` Do not pass `--wait` to `helm uninstall`. The Kafka topics cannot finish deleting until the finalizers above are cleared, so waiting only burns the timeout. --- # Upgrade Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/operate/upgrade # Upgrade When this page is done you are running the new release, with the same data. Read [Roll back](./rollback.mdx) before you start. The rollback boundary changes what a safe upgrade looks like. ## Before you start - [ ] Read the release notes for every version between yours and the target - [ ] A current backup. See [Back up and restore](./backup.mdx) - [ ] The new release's install kit, verified - [ ] A maintenance window. The application is briefly unavailable while pods roll ## Rules that are not negotiable | Rule | Why | |---|---| | One minor at a time | Skipping a minor is not supported. Each train's migrations assume the previous train ran | | Patches in place | Within a train, upgrade directly | | Same order as install: infra, then data, then platform | Each chart depends on the previous one | | Apply CRDs by hand first | Helm never upgrades CRDs. See step 1 | | Back up first | The migration hook changes your schema, and `helm rollback` does not undo that | ## 1. Apply the new CRDs **Do this before anything else.** Helm installs a chart's CRDs on first install and ignores them from then on. An upgrade that ships a new CRD schema fails with a server-side-apply error, or worse, silently rejects fields it does not know. ```bash cd $B/charts tar xzf gen0sec-infra-*.tgz kubectl apply --server-side --force-conflicts \ -f gen0sec-infra/charts/postgres-operator/crds/ kubectl apply --server-side --force-conflicts \ -f gen0sec-infra/charts/strimzi-kafka-operator/crds/ ``` The output is similar to this: ``` customresourcedefinition.apiextensions.k8s.io/postgresqls.acid.zalan.do serverside-applied ``` :::note[This is not hypothetical] The Postgres operator 1.15 to 2.0 change adds PostgreSQL 18 to the `postgresqls` schema. Without the CRD apply, the `postgresql/core` resource is rejected outright and the database never starts. ::: ## 2. Get the new artifacts Follow [Prepare the artifacts](./../04-prepare-artifacts.mdx) for the new release. Both methods work the same as on a first install, with one addition for the offline bundle. **Offline bundle only.** Mirroring is additive: the new tags land beside the old ones. Your running install keeps working until you upgrade, and a rollback still has its images. Do not reuse the previous bundle's directory. The bundler refuses to write into an existing one, and each bundle's values files are targeted at the version they shipped with. ## 3. Upgrade infra Run every command from the **new** bundle directory, so `charts/` and `values/` are the new release's. ```bash helm upgrade g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml ``` If you installed the platform under a release name or namespace other than `g0s` in `gen0sec`, add `--set components.ui.serviceAccountName=-ui,components.ui.namespace=` here. The infra chart binds the dashboard's service account by name, and a wrong name fails silently: the Components tab then cannot show infrastructure versions. Wait for the operators to finish rolling before continuing. ```bash kubectl -n gen0sec-system rollout status deploy/g0s-infra-postgres-operator kubectl -n gen0sec-system rollout status deploy/strimzi-cluster-operator kubectl -n gen0sec-system rollout status deploy/g0s-infra-synapse-operator kubectl -n gen0sec-system rollout status deploy/dragonfly kubectl -n gen0sec-system rollout status \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" ``` :::note[v0.2.2: Postgres restarts here] v0.2.2 changes the Postgres image, so the Postgres operator rolls every database pod onto it. With one database instance, as on the quickstart, the platform is unavailable until that pod is back. Step 4 waits for it. ::: ## 4. Upgrade data ```bash helm upgrade g0s-data $B/charts/gen0sec-data-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml ``` ```bash kubectl -n gen0sec-system wait --for=condition=Ready kafka/core --timeout=15m kubectl -n gen0sec-system wait \ --for=jsonpath='{.status.PostgresClusterStatus}'=Running postgresql/core --timeout=15m ``` ## 5. Upgrade the platform The schema migration runs first, as a **pre-upgrade** hook, before any new pod rolls. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` Upgrading from v0.1.x, `entrypointHost` is required even if you never published the endpoints: the dashboard now builds its sign-in URL from it, and the render fails without it. `$KIT` is the **new** release's kit. Pass every file you layered at install time, in the same order: `helm upgrade` applies only what you give it. Leaving out the ingress overlay removes every Ingress, and leaving out `single-node/platform-values.yaml` on a quickstart install scales every service back to its production replica count. See [Publish the endpoints](./../06-publish.mdx). **If the migration fails, Helm aborts and the old application keeps running.** That is the designed behavior and it is why the hook runs before the rollout. ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` Fix the cause, then re-run the same command. See [Migration failures](./../troubleshoot/symptoms.mdx#migration-failures). :::note[v0.2.0: one migration needs a quiet moment] `20261004150000` changes a constraint on the block events table and waits at most 10 seconds for its lock. On a busy install the job fails with `canceling statement due to lock timeout` and leaves the schema dirty. Nothing was changed. Pause the block events pipeline, mark the previous version and re-run this step: ```bash kubectl -n gen0sec-system scale deploy/g0s-data-kafka2pg-block-events --replicas=0 kubectl -n gen0sec run db-migrate-force --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/db-migrate:$VERSION \ --env DATABASE_URL="$OWNER_DSN" -- force 20261004120000 ``` Scale the pipeline back to 1 once the upgrade completes. Events wait in Kafka meanwhile. ::: :::note[v0.2.1: check the on-prem marker first] The migration job's connection string must carry the on-prem marker, or v0.2.1 stops rule changes reaching agents. `make-db-secrets.sh` always adds it. If your secret manager produces `gen0sec-db-migrate`, confirm before you run this step: ```bash kubectl -n gen0sec get secret gen0sec-db-migrate -o jsonpath='{.data.DATABASE_URL}' \ | base64 -d | grep -c 'gen0sec.deployment_mode%3Donprem' ``` `1` means it is there. See [Secrets](./../configure/secrets.mdx#where-the-database-passwords-come-from). A `WARNING: ... left NOT VALID` in the migration log is not a failure. It means existing rows break a check that new writes now have to pass, and the migration continues. ::: :::note[v0.2.2: check connection-state rules first] If a workspace has a firewall rule with a connection-state condition, its config stops updating after this step until every agent reporting to it runs Synapse 0.8.0 or later with firewall rules enabled. Upgrade those agents first, or remove the rule. See [Known limitations](./../troubleshoot/known-limitations.mdx#connection-state-rules-need-current-agents). The migration rebuilds the signal event chart views from every stored signal event, so it takes longer the more of them you keep. ::: ## 6. Verify ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` Run the smoke test after every upgrade, not just after installs. It is the only check that proves the platform still serves an agent rather than merely starting. Full checklist: [Verify the install](./../08-verify.mdx). ## If you changed values along the way `helm upgrade` uses the values files you pass, not the ones from last time. Anything you set with `--set` on the previous install must be set again, or it reverts to the chart default. Check what the running release actually has: ```bash helm get values g0s -n gen0sec ``` Keep your overrides in a file rather than in shell history. See [Reference configurations](./../configure/reference-configs.mdx). ## If it fails | Symptom | Start here | |---|---| | CRD schema error on `helm upgrade` | You skipped step 1 | | `Unsupported value` from the Postgres operator | You skipped step 1 | | Migration job fails | [Migration failures](./../troubleshoot/symptoms.mdx#migration-failures) | | Migration job fails with `lock timeout` | A busy block events table. See the note in step 5 | | Pods roll but the smoke test fails | [Roll back](./rollback.mdx) | | `ImagePullBackOff` on new tags | The new images were not mirrored. Offline bundle method, step 2 | ## Next [Roll back](./rollback.mdx). --- # Prepare the artifacts Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/prepare-artifacts import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; # Prepare the artifacts ## Before you start - [ ] [Requirements](./02-requirements.mdx) met, tools installed, `yq` version checked - [ ] [Network and connectivity](./03-network.mdx) configured by your network team - [ ] Your [registry token](./02-requirements.mdx#the-registry-token), created on the dashboard under **Repository Access** - [ ] Your [Gen0Sec API key with the required scopes](./02-requirements.mdx#your-gen0sec-api-key) ## 0. Set your keys Create your token files. ```bash mkdir -p ~/.gen0sec cd ~/.gen0sec/ && touch .registry-token .download-api-key ``` Then open up an editor of your choice to fill each file with their respective content. Then run ```bash export REGISTRY_TOKEN=$(cat ~/.gen0sec/.registry-token) export G0S_DOWNLOAD_API_KEY=$(cat ~/.gen0sec/.download-api-key) ``` :::tip[Manage your keys securely] By storing keys in files, you make sure they don't get etched into your shell history. If you have history turned off or you know what you're doing, you can also go with ```bash export REGISTRY_TOKEN="your-registry-token" export G0S_DOWNLOAD_API_KEY="your-gen0sec-api-key" ``` ::: ## 1. Set the release Pin the release explicitly. A pinned tag installs the same bytes in six months; a resolved one does not. You can find available release versions on the [Releases page](https://releases.gen0sec.com/cerebellum/). ```bash export TAG=v0.2.2 export VERSION=${TAG#v} ``` To always use the latest version instead: ```bash export TAG=$(curl -fsSL https://releases.gen0sec.com/api/repos/cerebellum/latest | jq -r .tag_name) export VERSION=${TAG#v} ``` :::note[Release candidates are not published here] Only stable releases are mirrored, so an `-rc.N` tag returns 404 no matter how it is spelled. The [release index](https://releases.gen0sec.com/cerebellum/) is the list of what exists. ::: ## 2. Download the install kit ```bash curl -fsSLO https://releases.gen0sec.com/cerebellum/$TAG/cerebellum-install-kit.tar.gz curl -fsSLO https://releases.gen0sec.com/cerebellum/$TAG/cerebellum-install-kit.tar.gz.cosign.bundle ``` ## 3. Verify the signature Run this before you unpack anything. ```bash cosign verify-blob \ --bundle cerebellum-install-kit.tar.gz.cosign.bundle \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/release\.yaml@refs/tags/' \ cerebellum-install-kit.tar.gz ``` The output is similar to this: ``` Verified OK ``` Anything else means stop. Do not unpack the archive. :::note[Why both certificate flags are required] `cosign` refuses to verify without an expected issuer and an expected identity, so a shortened command fails rather than passing weakly. The identity is pinned to one workflow on a tag reference. A broader pattern would accept a signature produced by any workflow in our organization. The signature covers the file's contents, so it proves the bytes are ours regardless of how they reached you. ::: ## 4. Unpack it ```bash tar -xzf cerebellum-install-kit.tar.gz export KIT=$PWD/cerebellum-install-kit-$VERSION cd $KIT ``` The kit contains the bundler, the release manifest, the helper scripts used later in this install, and the sizing overlays. It does not contain images or charts. ## 5. Log in to our registry ```bash export REGISTRY=registry.gen0sec.com export REGISTRY_USER=any printf '%s' "$REGISTRY_TOKEN" | helm registry login $REGISTRY -u "$REGISTRY_USER" --password-stdin printf '%s' "$REGISTRY_TOKEN" | skopeo login $REGISTRY -u "$REGISTRY_USER" --password-stdin ``` Both are needed. `helm` and `skopeo` keep separate credential stores, and the next step uses each of them. `cosign` is a third store again, and it needs no login of its own. The bundler lends it the credential `skopeo` holds. ## 6. Get the charts Pull the charts and their values files. Images are not pulled: your cluster pulls each one from `registry.gen0sec.com` when it needs it. :::note[Confirm your nodes can pull from us before you go further] Our registry challenges pulls that send no `User-Agent`, which on some distributions is what the kubelet's pull path does. This step succeeds either way, because it runs from your workstation. The failure appears at step 3 of [Install](./05-install.mdx), as a 403 on every image. Check it now: [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: ```bash ./bundler.sh assemble-charts release-manifest.yaml export B=$PWD/gen0sec-cerebellum-multiarch-$VERSION ``` The output ends with a per-chart digest confirmation. :::note[What "digest-verified" means here] `assemble-charts` checks two digests per chart: the registry's manifest against the digest this release pins, and the chart archive against the layer named in that manifest. What you install provably came out of the chart this release published. ::: Each chart carries its own `values-onprem.yaml`, extracted into `$B/values/`. That file is what points the install at a registry. The default `values.yaml` leaves the registry blank on purpose, so installing without `-f` will not resolve images. ### 6a. Build the bundle on your workstation ```bash ./bundler.sh create-package release-manifest.yaml ``` This pulls every image and chart the release pins by digest, verifies every signature, then packages the result as `gen0sec-cerebellum-multiarch-$VERSION.tar.gz`. It carries both `linux/amd64` and `linux/arm64`, so it installs on any node in your cluster. Signature verification happens here, on the connected side, because it needs the public transparency log. Everything downstream is protected by checksums instead. :::note[Before you use --arch to halve the transfer] `--arch amd64` or `--arch arm64` extracts one manifest out of each multi-architecture index. That changes every image digest by definition. Per-architecture signatures still verify, because each one is signed individually. **SBOM attestations do not**, because they are attached to the index, and the index is what a single-architecture bundle leaves behind. The bundler warns at the time and annotates the bundle. If your acceptance process checks attestations, stay on the default. If your nodes are mixed, or you are unsure what they run, stay on the default: a missing architecture fails when the kubelet tries to pull, on an isolated cluster, after the transfer. A single-architecture bundle also renames the tarball and the directory inside it, so substitute the name in every command that follows. ::: ### 6b. Move it across In this step you take your bundle onto an intermediary machine inside the network where your internal registry and the target cluster is located you'll install the bundle onto. After transferring the bundle, you can clean up all artifacts it has created. eg. ```bash scp gen0sec-cerebellum-multiarch-$VERSION.tar.gz :~/ # `create-package` ran inside the kit directory, so the bundle is sitting in it. Copying the kit first # would transfer tens of gigabytes twice. `cleanup` removes only what the bundler created. ./bundler.sh cleanup cd .. && scp -r cerebellum-install-kit-$VERSION :~/ # You can even remove the installation kit if you don't intend to keep it. # cd $KIT/.. && rm -rf cerebellum-install-kit.tar.gz $KIT ``` ### 6c. Load it into your registry Log in to your intermediary machine and use the bundler to load the images into your internal registry. ```bash export REGISTRY=registry.internal.example export REGISTRY_USER=your-registry-user export REGISTRY_TOKEN=$(cat path/to/your-registry-token) # Carried across because this is a different machine and Install step 2 needs # it. Unset, the secret it creates is empty and every relay fetch 401s. export G0S_DOWNLOAD_API_KEY=$(cat path/to/your-gen0sec-api-key) export KIT=~/cerebellum-install-kit-$VERSION export VERSION=$(cat $KIT/release-manifest.yaml | yq ".spec.version") tar -xzf gen0sec-cerebellum-multiarch-$VERSION.tar.gz export B=~/gen0sec-cerebellum-multiarch-$VERSION cd $B printf '%s' "$REGISTRY_TOKEN" | skopeo login $REGISTRY -u "$REGISTRY_USER" --password-stdin ./bundler.sh apply release-manifest.yaml -r $REGISTRY ``` `apply` pushes every bundled image into your registry, then rewrites the bundled values files to point at it. The charts are not pushed: you install them from `$B/charts/*.tgz`. The bundle is checked against its own `SHA256SUMS` before anything is pushed, so a corrupted transfer fails here rather than at install time. :::note[For a plain-HTTP registry] Set `MIRROR_TLS_VERIFY=false` before `apply`. This affects the push only, and it is the one place in this install where TLS verification can be disabled. Prefer a certificate from your internal CA. ::: ## Checkpoint Whichever method you used, you should now have the following: | Variable | Points at | |---|---| | `$KIT` | The unpacked install kit | | `$B` | A directory containing `charts/` and `values/` | | `$REGISTRY` | The registry your cluster will pull from | | `$REGISTRY_USER` | The username for that registry (`any` for ours) | | `$REGISTRY_TOKEN` | The password or token for that registry | | `$G0S_DOWNLOAD_API_KEY` | Your Gen0Sec API key, set in step 0. [Install](./05-install.mdx#your-gen0sec-api-key) step 2 puts it in a secret | You are logged in to `$REGISTRY`. Every page after this one assumes exactly that and nothing else, which is why nothing after this point branches. --- # Publish the endpoints Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/publish # Publish the endpoints The ingress controller is installed and running. However it's not configured to serve anything by default. That is deliberate, because creating those routes makes the APIs reachable from wherever the controller is reachable, so it is your decision, not ours. The data plane's LoadBalancer Service already has an external address at this point. Until you complete this page it serves nothing, because no rule names a host. :::note[Ingress and TLS are yours to configure] The platform ships the routing rules. Terminating TLS, issuing the certificate and exposing the LoadBalancer are your cluster's responsibility. For the configuration options, see the [Kubernetes ingress reference](https://docs.gen0sec.com/docs/client/kubernetes-ingress). ::: ## Before you start - [ ] [Install](./05-install.mdx) complete, all pods running - [ ] A DNS name you control, resolving to your ingress controller's address - [ ] A LoadBalancer address for the ingress data plane, or a Service type your cluster can expose - [ ] A TLS certificate for that name, or cert-manager configured ## 1. Choose the address One address serves the dashboard and every agent-facing API. Requests are routed by path prefix. Use the address you set as `entrypointHost` in [Install step 7](./05-install.mdx#7-install-the-platform). It is already in `$ENTRYPOINT_HOST` if you followed the install in this shell: ```bash export ENTRYPOINT_HOST=cerebellum.example.internal ``` **The host is required, and the chart enforces it.** A rule with no host is silently ignored by the ingress controller: the Ingress object is created, no event is logged, and nothing serves traffic. Rather than publish one, the chart refuses to render and names the value to set, so this fails at `helm upgrade` instead of in production. **Use the address people will actually type.** The dashboard derives its sign-in URL from this host, and sign-in fails if the two do not match. ## 2. Publish The kit ships the overlay. Layer it on top of the values file you have been using. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` :::warning[Re-pass every values file you installed with] `helm upgrade` applies only the files you pass it. Anything layered at install time and omitted here reverts to the chart default. On the [Quickstart](./quickstart.mdx) you also passed `-f $KIT/examples/single-node/platform-values.yaml`, so include it in this upgrade too (before the ingress overlay), or every service scales back to its production replica count: ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-node/platform-values.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` ::: :::note[Quote any --set argument containing brackets] `zsh` treats `[0]` as a glob and fails before `helm` runs. Single-quote the whole argument: `--set 'services.ui.ingress.hosts[0].host=...'`. ::: ## 3. Verify the routes exist ```bash kubectl -n gen0sec get ingress ``` The output is similar to this: ``` NAME CLASS HOSTS ADDRESS PORTS AGE g0s-platform synapse cerebellum.example.internal 10.0.0.42 80 30s ``` An empty `HOSTS` column means the host was not applied, and nothing will serve traffic. The upgrade fails before it can leave an Ingress in that state, so if you see one it was created by an earlier install. Then check from outside the cluster. The dashboard is published at `/` and returns a redirect to its sign-in page, so follow it with `-L`: ```bash curl -fsSL -o /dev/null -w '%{http_code}\n' https://$ENTRYPOINT_HOST/ ``` The output is similar to this: ``` 200 ``` :::note[There is no health endpoint on the ingress] `/status/healthz` is served per pod inside the cluster and is deliberately **not** published through the ingress. A request to it from outside falls through to the dashboard catch-all and returns `404`, so do not use it as an external check. The dashboard root above confirms reachability; the [smoke test](./08-verify.mdx#4-the-platform-serves-an-agent) confirms the platform serves agents. ::: ## Find the address to point DNS at The ingress data plane is a `LoadBalancer` Service. ```bash kubectl -n gen0sec-system get svc g0s-infra-synapse -o wide ``` Point your DNS name at the external address. If `EXTERNAL-IP` stays ``, your cluster has no LoadBalancer implementation: change the Service type to something it can expose. It cannot stay cluster-internal. ## The route table You do not configure these. The overlay sets them, read off each service's own router. They are here because they are the answer to "a request landed on the wrong backend", and because a missing route fails quietly rather than loudly. | Path prefix | Served by | |---|---| | `/` | Dashboard (catch-all, shortest prefix, so it is last resort) | | `/v1/indicators`, `/v1/models`, `/v1/geoip`, `/v1/ids-rules` | Relay | | `/v1/penalty-model`, `/v1/traffic-model`, `/v1/ja4-classifier`, `/v1/flow-classifier`, `/v1/jepa-embedder` | Relay, per-model aliases | | `/v1/identity`, `/v1/policy-edges`, `/v1/fleet-bans` | Relay, served locally | | `/v1/authcheck`, `/v1/auth` | Authentication | | `/v1/events`, `/v1/agents`, `/v1/log` | Agent ingest | | `/v1/agents/events` | Configuration push (longer prefix than `/v1/agents`, so it wins) | | `/v1/config` | Configuration | | `/v1/telemetry`, `/v1/logs` | Telemetry | | `/v1/signal`, `/v1/rules` | Signal | | `/v1/ban-ledger` | Ban ledger | | `/v1/threat` | Threat feeds | | `/v1/service-graph` | Service graph | Routing is by **longest matching prefix**, because one address serves every backend. The configuration push stream on `/v1/config` is long-lived, so the chart sets read, write and idle timeouts of 3600 seconds on that route. Do not lower them: agents fall back to polling with no push, which looks like a slow platform rather than a timeout. :::note[Why a missing route is worse than a broken one] The dashboard owns `/`, so any path absent from this table falls through to it and gets a redirect or HTML. HTTP has no way for the caller to tell that apart from a real answer, so the failures are silent: | Missing route | What the agent actually does | |---|---| | `/v1/authcheck` | Receives HTML, cannot distinguish a bad key from a routing mistake, and continues unverified | | `/v1/agents/events` | Logs `SSE connect returned HTTP 404`, then falls back to polling. Looks like a slow platform, not a routing bug | | `/v1/ban-ledger` | Reads the redirect as success. The contribution is lost with no error anywhere | This is why the overlay exists rather than a list of prefixes for you to type. ::: :::note[/v1/ids-rules goes to the relay, not to the in-cluster rules service] IDS rules are relayed from gen0sec. The in-cluster rules service builds rules from a local ingest pipeline that this edition does not run. Route this prefix there and you reach a service with no rules, and the relay is never used. ::: ### If TLS terminates upstream If a load balancer, CDN or tunnel terminates TLS in front of the ingress, the chart cannot infer the external scheme and guesses `http` while the browser is on `https`. The dashboard then builds its sign-in URL with the wrong scheme and rejects every sign-in with `Invalid origin`. Set it explicitly: ```bash --set externalScheme=https ``` Leave it unset when TLS terminates at the ingress, because then the chart reads it from that service's ingress TLS block. One service can differ from the rest with `services..ingress.externalScheme`, which wins over the install-wide value above. Only a service that derives its own external URL reads it, and the dashboard is the only one that does. ## The dashboard The dashboard is enabled by default and is published at the root of your address. Sign-in is email and password, held in your own Postgres, so nothing outside the cluster is involved. Its secret, `g0s-db-ui`, was created in [Install](./05-install.mdx) step 6 with all the others. Re-running that script keeps the existing signing secret, so sessions and invite links stay valid. To leave the dashboard out: ```bash --set services.ui.enabled=false ``` ## If it fails | Symptom | Cause | |---|---| | `helm upgrade` fails with `entrypointHost is required` | Expected, and the point: `entrypointHost` was not passed. Set it and re-run | | `kubectl get ingress` shows an empty `HOSTS` column | An Ingress left by an earlier install, created when a host-less rule could still render. Re-run the upgrade with `entrypointHost` set | | Sign-in is rejected with `Invalid origin` | The dashboard's URL does not match the address people type. Check `entrypointHost`, and `externalScheme` if TLS terminates upstream | | `curl` times out | DNS does not resolve to your ingress controller, or a firewall blocks the port | | The dashboard loads but sign-in redirects to the wrong address | The sign-in URL is derived from `entrypointHost`. It must be the address people type | | A request reaches the wrong service | Longest-prefix routing. Check the `/v1/agents` overlap above | --- # Quickstart on one node Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/quickstart # Quickstart on one node :::danger[Not for production] This runs every component at a single replica on a single node. There is no redundancy anywhere: **losing the node loses the data.** Postgres has no standby, Kafka has one broker at replication factor 1, and the object store has no erasure coding. Use it for evaluation, training and upgrade rehearsals. For anything you depend on, follow [Sizing a production install](./02-requirements.mdx#sizing-a-production-install), which runs on three nodes. ::: ## What you get A working instance you can sign in to, on one machine, without planning a cluster. This is not a separate procedure. It is the same install as the rest of this documentation, with one extra values file per chart. ## What you need | | | |---|---| | One node | 12 vCPU, 32 GB RAM, 200 GB disk | | Kubernetes | k3s | | StorageClass | Any default block StorageClass. k3s installs `local-path` as the default, which is enough | | Tools | The same list as [Requirements](./02-requirements.mdx#tools-on-your-workstation) | | Credentials | The same two as [Requirements](./02-requirements.mdx#credentials-from-gen0sec). The relay is required here too | ## How it differs from production The kit ships an overlay per chart that reduces everything to one replica. | | Production | This quickstart | |---|---|---| | Object store | 3 replicas, erasure coded, distributed | 1 replica, **standalone mode**, and a Deployment rather than a StatefulSet | | Postgres | 3 instances | 1 instance, no replica pooler | | Kafka | 3 brokers, replication factor 3 | 1 broker, every factor 1 | | Application services | 2 or 3 replicas each | 1 each | | Cache | Chart default | 256 MB, one thread | | Storage | Your block StorageClass, 500 GiB per node minimum | The cluster's default StorageClass, about 65 GiB of volumes | | Survives losing a node | Yes | No | :::note[The connection pooler stays on] Only the **replica** pooler is disabled, because there is no replica to pool. Every service still connects through `core-pooler`, exactly as in production, so the secrets that `make-db-secrets.sh` composes are the same shape here. ::: :::note[Two things that surprise people] The object store stops being a StatefulSet and becomes a Deployment of the same name. That is why the readiness checks in [Install](./05-install.mdx) resolve the kind instead of naming it. Volume claims from a previous multi-replica install are not reused. If you are converting an existing install, delete them first. ::: ## Run it Follow the spine exactly as written, starting at [Prepare the artifacts](./04-prepare-artifacts.mdx). Three commands change. Nothing else does. | Spine step | On this install | |---|---| | [Prepare the artifacts](./04-prepare-artifacts.mdx) | Unchanged. Either method works, and the registry method is fewer steps | | [Install](./05-install.mdx) steps 1 and 2 | Unchanged. Same namespaces, same three secrets | | [Install](./05-install.mdx) steps 3, 4 and 7 | Add the overlay. See below | | [Install](./05-install.mdx) steps 5 and 6 | Unchanged. You still create the bucket, and you still run the secret script | | [Publish the endpoints](./06-publish.mdx) | Unchanged. See the note on the address below | | [First sign-in](./07-first-signin.mdx) and [Verify the install](./08-verify.mdx) | Unchanged | Pass the overlay **after** the production values file, so it wins. ```bash helm install g0s-infra $B/charts/gen0sec-infra-*.tgz -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml \ -f $KIT/examples/single-node/infra-values.yaml ``` ```bash helm install g0s-data $B/charts/gen0sec-data-*.tgz -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml \ -f $KIT/examples/single-node/data-values.yaml ``` ```bash helm install g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-node/platform-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` The wait gates after each one are the same as on [Install](./05-install.mdx). Read the files before you use them: each overlay carries comments explaining what it trades away. :::caution[Your cluster needs a default StorageClass] The overlays leave the storage class empty in all three places — `rustfs.storageclass.name` in `infra-values.yaml`, `postgres.volume.storageClass` and `kafka.storage.class` in `data-values.yaml` — which means "use the cluster's default", the same as the production values file. On k3s that resolves to `local-path` and there is nothing to do. If your cluster has **no** default, every PVC stays `Pending`. `helm install` still reports success, and the only symptom is the wait gate above timing out without naming a cause. Check first: ```bash kubectl get storageclass # exactly one row must be marked (default) ``` To pin a class explicitly instead of relying on the default, set those three keys, or layer a fourth `-f` file last. ::: ## The address [Publish the endpoints](./06-publish.mdx) is unchanged, including the requirement for a host, which the platform install above already needs in `$ENTRYPOINT_HOST`. On k3s the bundled LoadBalancer implementation gives the ingress data plane the node's own address, so there is nothing further to provision. For an evaluation, a name in `/etc/hosts` on the machine you browse from is enough. It has to be the name you actually type in the browser, because the dashboard builds its sign-in URL from it. :::note[About TLS terminatotion] In case your TLS termination does not happen on a higher level, configure Synapse Ingress to work with cert-manager (see the [Kubernetes ingress reference](https://docs.gen0sec.com/docs/client/kubernetes-ingress)). The shipped ingress overlay sets `externalScheme: https` and configures no certificate at the ingress. A single-node evaluation usually has none. Either provision a certificate, or serve plain HTTP: set `externalScheme` to `http` in the ingress overlay and use an `http://` base URL in the sign-in and verify steps below. ::: ## First sign-in Nothing is different here, and it is worth knowing before you get there: the dashboard makes you enroll multi-factor authentication and replace the seeded password before anything opens. See [First sign-in](./07-first-signin.mdx). ## Verify Run the same checks as a production install, from [Verify the install](./08-verify.mdx). The smoke test is the one that matters, because it proves the platform serves an agent rather than merely starting: ```bash # Save the agent key you minted at first sign-in, then read it back below. mkdir -p ~/.gen0sec printf '%s' '' > ~/.gen0sec/agent-key export GEN0SEC_BASE_URL=https://$ENTRYPOINT_HOST export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) $KIT/scripts/verify-deployment.sh ``` A single-node install that passes it is a good rehearsal for a real one. That is most of the value of building this. ## Tearing it down ```bash $KIT/scripts/uninstall.sh --yes ``` It asks you to type the cluster context name back before deleting anything, so it will not run unattended. See [Uninstall](./operate/uninstall.mdx). ## When you are ready for production [Sizing a production install](./02-requirements.mdx#sizing-a-production-install) has the node counts, per-node capacity and storage figures. There is no in-place conversion from this install to a production one: build the production cluster separately. --- # Bundler CLI Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/reference/bundler # Bundler CLI `bundler.sh` ships in the install kit. It builds an offline bundle on a connected host and loads it into your registry on a host that has no internet access. Its only input is the `release-manifest.yaml` inside the kit. No access to our source is needed, so you can build your own bundle from any published release. For the procedure rather than the reference, see [Prepare the artifacts](./../04-prepare-artifacts.mdx). ## Which commands need internet access This is the table an isolated site actually needs. The pipeline is split on internet access, not on environment. | Command | Internet | Talks to | |---|---|---| | `validate` | **Required** | Parses the manifest, resolves tooling, then fetches one image manifest to confirm registry access | | `assemble` | **Required** | Source registry: images and charts | | `assemble-charts` | **Required** | Source registry: charts only | | `assemble-images` | **Required** | Source registry: images only | | `verify-assembly` | **Required** | Source registry for signature artifacts, plus Sigstore trust material | | `create-package` | **Required** | Inherits `assemble` and `verify-assembly` | | `package` | None | Local `tar` | | `verify-package` | None | Local extract and checksum comparison | | `mirror` | None | Destination registry, over your local network | | `retarget` | None | Local files | | `apply` | None | Destination registry. This is `mirror` plus `retarget` | | `cleanup`, `version`, `help` | None | Local | **Split the pipeline at the tarball, not earlier.** `create-package` cannot run on an isolated host: it fails in preflight with no route to the source registry. ## Commands | Command | What it does | |---|---| | `validate` | Preflight. The manifest exists and its schema is sane, every image and chart is digest-pinned, the tooling resolves, and the source registry is readable. Fetches one image manifest, so a bad host, missing login, TLS mismatch or unpublished digest fails here rather than partway through a multi-gigabyte download | | `assemble` | Creates the bundle directory, pulls every image and chart, extracts each chart's production values file, copies in the manifest and itself. Fails if the directory exists | | `assemble-charts` | The charts half. **What the registry install method uses**, since the cluster pulls its own images | | `assemble-images` | The images half. Separable so a failed multi-hour pull can be retried without re-fetching charts | | `verify-assembly` | Keyless signature verification of every pinned image and chart, then writes `SHA256SUMS` and prints the bundle **fingerprint**. Reports every failure and refuses to continue on any | | `package` | Tars the bundle directory into a tarball **beside** it, never inside | | `verify-package` | Unpacks to a temporary directory and re-checks `SHA256SUMS` | | `create-package` | `assemble`, `verify-assembly`, `package`, `verify-package`, in order | | `mirror` | Verifies the bundle against `SHA256SUMS`, then pushes every image to your registry | | `retarget` | Rewrites the registry host in the bundle's own values files, in place, and updates the affected `SHA256SUMS` entries so the bundle stays self-consistent | | `apply` | `mirror` then `retarget`. The usual customer step | | `cleanup` | Removes every directory and archive the bundler created | | `version` | The bundler's version, stamped at release | | `help` | Usage | Every command runs `validate` first, so any of them is safe to invoke on its own. ## Options | Option | Effect | |---|---| | `--arch amd64` or `--arch arm64` | Single-architecture bundle, roughly half the size. **Loses SBOM attestations.** See below | | `-o ` | A different tarball name | | `-r ` | Destination registry for `mirror`, `retarget` and `apply` | | `--dangerously-skip-cosign-verify` | Drops signature verification and its tooling requirement. The bundle is annotated as unverified | ### The --arch trade-off A single-architecture bundle extracts one manifest out of each multi-architecture index, which changes every image digest by definition. | | Default | With `--arch` | |---|---|---| | Image signatures | Verify | **Verify.** Each per-architecture manifest is signed individually | | SBOM attestations | Verify | **Lost.** They hang off the index, which is what is left behind | | Digests match the manifest | Yes | No | | Tarball and directory name | `multiarch` | Named after the architecture, so substitute it everywhere | If your acceptance process checks attestations, or your nodes are mixed, or you are not certain what they run: take the default. A missing architecture fails when the kubelet tries to pull, on an isolated cluster, after the transfer. ## Environment variables | Variable | Purpose | |---|---| | `MIKEFARAH_YQ_PATH` | Path to the Go `yq`, when the one on `PATH` is the Python one | | `BUNDLE_TLS_VERIFY=false` | Disables TLS verification on the **pull** side | | `MIRROR_TLS_VERIFY=false` | Disables TLS verification on the **push** side, for a plain-HTTP registry | The two TLS variables are the only places in the whole install where verification can be disabled. Prefer a certificate from your internal CA. See [Hardening](./../security/hardening.mdx). ## Credentials, and the three stores Three tools, three credential stores. You log in to two of them. | Tool | Store | Login needed | |---|---|---| | `skopeo` | The containers store | Yes, both source and destination | | `helm` | `HELM_REGISTRY_CONFIG` | Yes, source only | | `cosign` | The Docker config | **No.** The bundler lends it the one `skopeo` holds | If you see the bundler say it is reusing skopeo's credential for cosign, that is the expected path and not a warning. ## Behavior worth knowing | | | |---|---| | `assemble` refuses to write into an existing directory | So a stale bundle cannot be silently mixed with a new one | | `retarget` takes no file arguments | Rewriting arbitrary paths handed in on a command line is not something an isolated install should be able to ask for | | `retarget` verifies integrity **before** editing, never after | That order is what stops a checksum update from re-blessing a bundle that arrived altered | | `retarget` is idempotent | Running `apply` twice is safe | | `mirror` runs before `retarget` inside `apply` | So a registry failure leaves your values files untouched | | Mirroring is additive | New tags land beside old ones, so a running install keeps working and a rollback still has its images | | An egress proxy is fine for the bundler | Provided the host can reach the registry and Sigstore. Unlike the relay, these are ordinary CLI tools that honour proxy variables | ## Bundle contents ``` gen0sec-cerebellum--/ ├── oci/ every image, as an OCI layout ├── charts/ installable chart archives ├── values/ each chart's production values, retargeted by `apply` ├── release-manifest.yaml the digests this bundle was built from ├── bundler.sh a copy └── SHA256SUMS over every file above ``` `assemble` extracts each chart's values file as `values/-values-onprem.yaml`. There is no bare `values-onprem.yaml` in the bundle root, which is why the install commands name the file in full. The bundle does **not** carry the install kit's `scripts/` or `examples/`. Those cross the air gap inside the kit directory. See [Install kit contents](./kit.mdx). --- # Release notes Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/reference/changelog # Release notes ## v0.2.2 A patch release. Upgrade in place from v0.2.1 following [Upgrade](./../operate/upgrade.mdx), including infra: this release changes the Postgres and ingress operator images. From v0.2.0 or v0.1.x, upgrade straight to v0.2.2, and read the notes for every version in between first. It carries 4 schema migrations, which run as the pre-upgrade hook. Two add managed WAF rules, one rebuilds the signal event views, and one grants a read the config generator now needs. Rolling the platform back to v0.2.1 is safe, but it brings back the firewall defect below; see [Roll back](./../operate/rollback.mdx). :::danger[Connection-state firewall rules matched all traffic] On v0.2.1 and earlier, a firewall rule with a connection-state condition, such as `allow established,related`, reached agents **without** that condition. An allow rule then allowed every packet in its direction, ahead of the deny it was written to protect, and a drop rule dropped every packet. Nothing reported it. If you use connection-state conditions, review what your agents let through since you created those rules. ::: :::warning[Connection-state rules now need Synapse 0.8.0 or later] v0.2.2 sends the condition, but an agent that does not understand it would ignore it and fail open as above. So when a workspace has a connection-state rule, the config generator publishes its config only if every agent that reported in the last hour advertises support. Synapse 0.8.0 or later, with firewall rules enabled, does. Otherwise the workspace **keeps its previous config**, and no rule change reaches its agents until you upgrade them or remove the rule. Check before upgrading; see [Known limitations](./../troubleshoot/known-limitations.mdx#connection-state-rules-need-current-agents). ::: :::warning[The infra upgrade restarts Postgres] The new Postgres image makes the Postgres operator roll every database pod. With one database instance, as on the quickstart, the platform is unavailable while that pod restarts. ::: ### Fixed **Connection-state firewall rules.** See the warnings above. **arm64 nodes could not run Postgres or the ingress operator.** The Postgres image was published for `linux/amd64` only, and the ingress operator's `linux/arm64` image carried an x86-64 binary, so on an arm64 node one failed to pull and the other could not start. Both are now built for each architecture. **Agents could stop hearing about new configs.** config-api connected to Kafka once, at startup. If that failed, or the connection later dropped, it stopped telling connected agents a new config was ready until it was restarted. It now retries and reconnects, and after a gap asks connected agents to fetch their config again. **One slow database query could stall every config build.** The config generator builds one workspace at a time, and its database reads had no time limit. Each build now has one, [`DB_READ_TIMEOUT`](./../configure/values.mdx#services), 30 seconds by default, and a build that runs out of time is dropped and the workspace keeps its previous config. Also, when reading the IDS ruleset version failed, configs were published with no ruleset version. The last version read is used instead. **Signal events in the newer format showed no IP, name or action** in the dashboard, and were not counted in its charts. Both formats are now read. **Config builds slowed with every country and ASN access rule.** Each rule scanned the whole GeoIP database again. A workspace with 44 such rules took about 70 seconds per build, and now takes under two. Resolved ranges are kept in memory until the GeoIP database changes. **A changed `DATA_DIR` left the config generator without a GeoIP database.** It downloaded the database to `DATA_DIR` but read it from a fixed path. Nothing to do if you never changed it. ### Added - **Managed WAF rules for CVE-2026-109618 and CVE-2026-109619**, five in all. They block oversized form field names, which exhaust PHP's memory, and a `PHPSESSID` form field or a PHP open tag or serialized object in an upload filename, which reach PHP's session upload progress. They match part names and headers only, never values. **They are switched on for every workspace that has at least one managed WAF rule switched on.** Others are left alone. Switch them off per workspace like any managed rule. - **Repository credentials can carry the SynapseOS updates scope**, so an appliance can fetch its OS updates with them. It is ticked by default with the other scopes. - `DB_READ_TIMEOUT` and `CAPABILITY_STALE_AFTER_SECONDS` on the config generator. See [Values](./../configure/values.mdx#services). ### Changed **The config generator's memory limit is 1 GiB**, up from 512 MiB, for the GeoIP cache. Blocking every country and ASN peaks at about 470 MiB. If you set `services.config-generator.resources` yourself, raise its limit to match. **Component versions.** PostgreSQL 18.4 to 18.6, in Postgres image `spilo:4.1.0-18-age-2`. The ingress operator is `0.1.13`, a version rather than a commit hash. **The bundler needs cosign 2.6.3 or later.** The Postgres and ingress operator images are now signed by cosign v3, which stores signatures where older cosign cannot see them. The bundler stops with a message saying so, rather than reporting every image as failing. **Image signatures come from three repositories.** The Postgres and ingress operator images are built and signed in their own repositories. See [Supply chain](./../security/supply-chain.mdx#verify-an-image-or-a-chart). ### Security - The firewall fix above. - Next.js 16.4.1, for the advisories open against 16.3.7. The dashboard's base image is pinned to Node.js 24.22.1 by digest. - The Postgres image is signed and carries an SBOM. It was unsigned before, which the bundler reported as a warning. ### Documentation - [Requirements](./../02-requirements.mdx#tools-on-your-workstation) asks for cosign 2.6.3 or later. - v0.2.0 and v0.2.1 list the firewall and arm64 defects in their known limitations. ## v0.2.1 A patch release. Upgrade in place from v0.2.0 following [Upgrade](./../operate/upgrade.mdx). From v0.1.x, upgrade straight to v0.2.1, and read the v0.2.0 notes below first. It carries 17 schema migrations, which run as the pre-upgrade hook. They bring our hosted and on-premises databases onto one migration chain, so most of them re-apply schema an on-premises database already has and change nothing there. What they do change on yours is listed below. None removes anything and every down migration is a no-op, so rolling back to v0.2.0 is safe; see [Roll back](./../operate/rollback.mdx). :::warning[Check the on-prem marker before upgrading] Migrations tell an on-premises database from our hosted one by a marker on the migration job's connection string. Without it, v0.2.1 points rule-change notifications at an address that exists only in our hosted deployment, and rule changes stop reaching agents. `make-db-secrets.sh` always adds it. If your secret manager produces `gen0sec-db-migrate`, confirm it is there: ```bash kubectl -n gen0sec get secret gen0sec-db-migrate -o jsonpath='{.data.DATABASE_URL}' \ | base64 -d | grep -c 'gen0sec.deployment_mode%3Donprem' ``` `1` means it is there. On `0`, add it before upgrading; see [Secrets](./../configure/secrets.mdx#where-the-database-passwords-come-from). ::: ### Fixed **Rule changes did not reach agents.** Creating, editing or deleting an access, firewall, WAF, smart firewall or IDS rule, or switching a managed rule on or off, saved the change but did not regenerate the configuration agents fetch. The fleet kept enforcing the previous rules, deleted ones included. A database trigger asks the config generator to regenerate after every rule write, and it needs elevated privileges to make that call. A migration shipped in v0.1.0 replaced the trigger without them, so the call was refused for the dashboard's and the services' database roles. The refusal was logged as a PostgreSQL warning and the rule write still succeeded, so nothing in the dashboard showed it: ``` WARNING: notify_config_generator: POST to http://g0s-config-generator... failed: permission denied for function http_post ``` The trigger runs with those privileges again. Rules saved before the upgrade are picked up by the next rule change in the same workspace, which regenerates that workspace's whole configuration. **Managed WAF rules did not work.** On an on-premises database the column linking a workspace to a managed WAF rule was named `rule_managed_id`, while the dashboard, the config generator and the database trigger all read `waf_rules_managed_id`. So: - The dashboard could not list managed WAF rules or switch them on or off. - Managed WAF rules switched on for a workspace were never sent to its agents. - Changing a managed WAF rule failed with `column s.waf_rules_managed_id does not exist`. v0.2.0 listed the last of these as fixed. That fix was correct for our hosted database, whose column already had the new name, and it is what produced this error on yours. The column is now renamed on upgrade, so every reader finds it. **Creating your organization ended on a page not found.** Right after first-run setup created the organization, the dashboard sent you to `//check-workspace`, which does not exist. You now land on the workspace overview. ### Changed **The overview no longer offers Threats.** The Threats module is not part of the self-hosted release, and its navigation item was already hidden, but the overview still showed its card and counted it. The overview now shows the six modules you have. **Some roles gain permissions.** Where a role, built-in or custom, already holds the first permission, the upgrade grants it the second: | Holds | Gains | |---|---| | `api_keys:read` | `registry_keys:read` | | `api_keys:write` | `registry_keys:write` | | `smart_firewall_rules:read` | `fleet_bans:read` | | `smart_firewall_rules:write` | `fleet_bans:write` | So anyone who could manage API keys can now manage registry tokens under **Repository Access**, and anyone who could manage smart firewall rules can now manage fleet bans. Review custom roles after the upgrade if either should stay separate. Nothing is revoked. ### Security - The dashboard's database role can no longer hard-delete firewall rules. Firewall rules are soft-deleted, so the dashboard never needed it. ### Documentation - [Secrets](./../configure/secrets.mdx#where-the-database-passwords-come-from) now spells out the on-prem marker the migration connection string needs, rather than only naming it. ## v0.2.0 A minor release. Upgrade from v0.1.0 or v0.1.1 following [Upgrade](./../operate/upgrade.mdx). It carries 18 schema migrations, which run as a pre-upgrade hook, so take a backup first. Every one only adds, so rolling the platform back to v0.1.x afterwards is safe; see [Roll back](./../operate/rollback.mdx). :::warning[Fleet bans start being enforced] On v0.1.x the fleet-ban artifact was written in a format agents could not read, so every agent loaded zero fleet bans while the publishing job reported success. From v0.2.0 they load. Agents with `platform.fleet_bans.enforce` on will start refusing traffic they did not refuse before. Review your fleet bans, or turn enforcement off, before you upgrade. ::: :::warning[Invitation links issued on v0.1.x stop working] Invitations are now tracked in the database, and links issued before the upgrade are not in it. Re-invite anyone who has not accepted yet. ::: :::warning[Check `externalScheme` in your values before upgrading] `externalScheme` was ignored on v0.1.x and is read from v0.2.0. If your values carry a top-level `externalScheme` that has never had any effect, **it takes effect on upgrade** and changes the scheme in the dashboard's sign-in URL. A scheme that does not match the address people actually type is rejected as `Invalid origin`. ```bash helm get values g0s -n gen0sec | grep externalScheme ``` Remove it or correct it before upgrading. Nothing to do if you pass `examples/single-entrypoint/ingress-values.yaml` unchanged: it ships `externalScheme: https`, which is the scheme v0.1.x effectively used anyway. ::: ### Added - **Passkeys.** Users can register and sign in with a passkey, and manage them under **Profile**. Passkeys are bound to the published host name and need HTTPS. See [Sign-in and identity](./../configure/identity.mdx#passkeys). - **Invitation and password-reset email**, through your SMTP server. Without SMTP, invitations fall back to a link you copy. See [Invitations and email](./../configure/identity.mdx#invitations-and-email). - **Role-scoped invitations.** An invitation grants specific workspaces, each with a role no higher than the inviter's. Pending invitations are listed and can be revoked, and access to one workspace can be revoked without removing the person. - **Audit log**, under **Organization settings** for the Owner and Admin roles. It records changes made in the dashboard and rule changes made through the signal API, with source IP and user agent. It is kept indefinitely; see [Known limitations](./../troubleshoot/known-limitations.mdx). - **Components**, under **Organization settings**: what each component runs, against the latest version known to the release. An online check for newer versions is off by default. See [Component versions](./../configure/components.mdx#verifying-what-you-are-actually-running). - **WAF rules accept `regex_replace()` and the `http.request.raw_path` field.** See [Known limitations](./../troubleshoot/known-limitations.mdx#new-waf-rule-functions-need-current-agents) for agent versions. - `CACHE_L1_TTL` sets the default lifetime of the in-process cache, as a duration. Default `30s`, up from a fixed `15s`. Set it through `global.commonEnv`. ### Fixed **Fleet bans never reached agents.** See the warning above. **Block events were lost in several ways.** Each of these applied to v0.1.x: - A batch of more than 20,000 records was refused whole, and OTLP senders do not retry that. The first 20,000 are now kept. - One row the database refused sent its whole insert batch, other organizations' rows included, to the dead-letter topic. Only the refused row is parked now, with the reason in a `dlq_reason` header. - A NUL byte in any field, a `source_ip` that was not an IP address, or a non-ASCII country code made the database refuse the row. These are now cleaned or dropped, and the event is kept. - A non-integer hit count stalled the dashboards' aggregates for every organization. It is now dropped, and the event kept. - An agent newer than the platform could report a source the platform did not know, and the event was refused. It is now stored without its source. `fleet_ban` is also accepted as a source. - IDS alerts were rejected as malformed block events and warned about once a minute. Records that are not block events are now skipped quietly. **Block events lacked country and ASN** whenever the GeoIP lookup missed, although the agent had sent them. The country is also stored in upper case, so `us` and `US` no longer count twice. **Drops made by the agent's kernel IP filter were not stored.** The agent reports them as `dropped_ip` records rather than block events, and those were skipped, so for an agent enforcing in the IP filter the access-rules views stayed empty. A drop the access rules caused is now stored as an access-rules block event, with its count as the hits and `bpf:ip_filter` as the rule. **Writing a managed WAF rule failed** with `column s.rule_managed_id does not exist`. **`--set entrypointHost` had no effect.** Following the [Publish the endpoints](./../06-publish.mdx) procedure verbatim published every route on the placeholder `gen0sec.internal` whatever address you passed. The overlay carried the host as a YAML anchor, and anchors are expanded when the file is parsed, before Helm merges `--set`, so the flag rewrote the top-level key while all ten rules kept the value the file shipped with. No template read the value either, so removing the anchor alone would not have helped. The failure was silent. The Ingress objects were created and looked correct, but Synapse keys its upstreams by host and serves nothing for an address no rule names. The chart now resolves the entrypoint itself and publishes every rule under it. **The dashboard's sign-in URL could be absent.** `BETTER_AUTH_URL` was derived from the first entry of a service's `ingress.hosts`. With the address named once in `entrypointHost` that list is empty, so the variable was not wrong but missing, and Better Auth rejected every sign-in with `Invalid origin`. It now follows the same entrypoint the rules do. ### Changed **An empty `entrypointHost` now fails the render** instead of publishing host-less rules. Such a rule matches every `Host` header and Synapse's operator skips it, so it could only ever be a misconfiguration that reported nothing. `helm upgrade` now stops and names the value to set. Rules that carry their own `ingress.hosts[].host` are unaffected and still need no `entrypointHost`. **`entrypointHost` is required from the first install**, before anything is published. The dashboard's sign-in URL now always comes from, in order: `services.ui.env.BETTER_AUTH_URL`, the dashboard's own `ingress.hosts[0].host`, then `entrypointHost`, whether or not its Ingress is enabled. With none of them the render fails and names `entrypointHost`. It was previously derived only once the dashboard's Ingress was enabled. [Install](./../05-install.mdx#7-install-the-platform) and [Upgrade](./../operate/upgrade.mdx#5-upgrade-the-platform) pass it. **`--set externalScheme` had no effect either.** Same defect, same shape: nothing read the top-level value, and the single-entrypoint overlay pinned the dashboard's own scheme on top of it, so the flag was inert twice over. It only ever looked correct because what it pinned was `https`, which is what a right answer produces anyway. The wrong scheme in the dashboard's base URL is rejected as `Invalid origin`, so this mattered to anyone terminating TLS upstream who needed anything else. The chart reads the top-level value now, and the overlay stops pinning the dashboard. `services..ingress.externalScheme` still overrides it for one service. A value that was inert becomes live. See the warning at the top before you upgrade. **The dashboard holds one namespaced permission.** The infra chart adds the Role and RoleBinding `g0s-ui-read-installed-versions` in `gen0sec-system`, allowing `get` on two named ConfigMaps, and the dashboard's service account token is now mounted. Nothing to do on the documented release names. Otherwise set `components.ui.*`; see [Upgrade](./../operate/upgrade.mdx#3-upgrade-infra) and [Privileges and RBAC](./../security/privileges.mdx). **The dashboard makes optional outbound calls**, none carrying agent data. See [Ports and egress](./../security/egress.mdx). **telemetry-api logs what it drops**, once a minute per organization, and its existing decode warnings are rate-limited the same way. See [Monitor](./../operate/monitor.mdx). **Rule changes made through the signal API are audited** in the same transaction. If the audit record cannot be written, the rule change is not applied. See [Symptoms](./../troubleshoot/symptoms.mdx#rule-changes-made-through-the-signal-api-never-appear). **The Agents item is hidden** from the dashboard navigation on self-hosted installs. **One migration waits at most 10 seconds for a lock** on the block events table. On a busy install it can fail and is safe to retry. See the note in [Upgrade](./../operate/upgrade.mdx#5-upgrade-the-platform). ### Security - gRPC 1.83.2, fixing an HTTP/2 memory exhaustion reachable in the APIs (GO-2026-6348), and OpenTelemetry 1.47.0. Other Go modules updated for advisories that were not reachable. - Next.js 16.3.7 and React 19.3.0, fixing CVE-2026-94545 as first shipped in v0.1.1. - better-auth 1.7.7, and the dashboard's dependency overrides enforced so `pnpm audit` is clean. ### Documentation - These docs are now versioned, with a version selector. v0.1.0 and v0.1.1 each keep their own version, with both defects above listed in their known limitations. - [Publish the endpoints](./../06-publish.mdx#if-tls-terminates-upstream) documents `--set externalScheme=https` again, now that it works. - [Upgrade](./../operate/upgrade.mdx) now passes the ingress overlay to the platform upgrade. Without it, the upgrade removes every Ingress. - [Roll back](./../operate/rollback.mdx) uses the `db-migrate` image of the release you are leaving. The previous image cannot run the down migrations. ## v0.1.1 A security patch for the dashboard. It fixes [CVE-2026-94545](https://nvd.nist.gov/vuln/detail/CVE-2026-94545) (medium, CVSS 4.0 score 5.3), in which Satori, the HTML-to-SVG library Next.js bundles for `next/og`, did not escape certain values in the SVG it generates. The dashboard does not render images through `next/og`, but the vulnerable copy shipped in its image and scanners flag it. The fix updates Next.js from 15.5.24 to 16.3.7, and React from 19.2.3 to 19.3.0. It upgrades in place from v0.1.0, with no migration and no values changes. Nothing else changed: the procedure is v0.1.0's, and its known limitations still apply. ## v0.1.0 First on-premises release. ### Installing Two methods, both documented on one procedure that branches once: [install from our registry](./../04-prepare-artifacts.mdx), or [create an offline bundle](./../04-prepare-artifacts.mdx) and mirror it into your own. ### Requires | | | |---|---| | Kubernetes | 1.23 or later | | Nodes | 3 for production, 16 vCPU and 32 GiB each recommended | | Storage | [sized to your retention](./../02-requirements.mdx#sizing-storage-for-your-retention) | | Cluster egress | `api.gen0sec.com:443`, required | ### Credentials you need **Two**, and the second is easy to miss. Both are self-service from the dashboard: 1. A registry token, for pulling charts and images. Created under **Repository Access**. The install kit itself is public and needs no credentials. 2. **Your Gen0Sec API key**, which the relay uses to fetch data from us. Without it the whole fleet receives empty artifacts with no error. See [Credentials from gen0sec](./../02-requirements.mdx#credentials-from-gen0sec). ### Not in this release | | | |---|---| | Zero-egress deployment | [Known limitations](./../troubleshoot/known-limitations.mdx#no-zero-egress-deployment) | | Explicit egress proxy for the relay | [Known limitations](./../troubleshoot/known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy) | | Single sign-on of any kind | Email and password only. Microsoft Entra ID is in development | | Shipped dashboards and alert rules | [Monitor](./../operate/monitor.mdx) lists what to watch meanwhile | | The threat service | Not shipped at this release | --- ## How we version | | | |---|---| | Scheme | Semantic versioning. Each minor is a release train | | Supported | The latest two minor trains | | Upgrades | Sequential minors, patches in place. **Skipping a minor is not supported** | | Release candidates | Not published to `releases.gen0sec.com`. Only stable releases are mirrored | Full policy: [Component versions](./../configure/components.mdx#support-policy). ## Finding a release ```bash curl -fsSL https://releases.gen0sec.com/api/repos/cerebellum/latest | jq -r .tag_name ``` Returns the latest full release. **Pin the tag explicitly for an install you want to reproduce**, so a rebuild in six months fetches the same bytes. ## What every release publishes | Artifact | Where | |---|---| | Install kit and its signature | `https://releases.gen0sec.com/cerebellum//` | | Images and charts | The registry, each pinned to a digest in the kit's release manifest | | Signatures and SBOM attestations | Alongside each image and chart, as OCI referrers | See [Supply chain](./../security/supply-chain.mdx). --- # Install kit contents Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/reference/kit # Install kit contents ## What you download Two files, published per release at `https://releases.gen0sec.com/cerebellum//`. | File | What | |---|---| | `cerebellum-install-kit.tar.gz` | The kit | | `cerebellum-install-kit.tar.gz.cosign.bundle` | Its detached signature | The release also publishes `latest-known-versions.json` there. You do not download it: the dashboard reads it only when its online update check is on. It is display data, not signed and not part of the kit. Verify before unpacking. See [Prepare the artifacts](./../04-prepare-artifacts.mdx#3-verify-the-signature). ## Layout ``` cerebellum-install-kit-/ ├── bundler.sh builds and loads the offline bundle ├── release-manifest.yaml every image and chart, pinned to a digest ├── scripts/ │ ├── preflight.sh cluster checks the charts assume but do not create │ ├── make-db-secrets.sh composes the ~20 application secrets │ ├── uninstall.sh the three releases, finalizers, namespaces │ └── verify-deployment.sh external smoke test, read-only ├── examples/ │ ├── single-node/ three overlays, one replica each │ └── single-entrypoint/ one address for the dashboard and every API ├── offline-data/ │ └── README.txt └── SHA256SUMS over every file above ``` **No images and no charts.** Those come from the registry, or from the offline bundle. ## The scripts | Script | Run it | Documented in | |---|---|---| | `preflight.sh` | Before installing | [Install step 1](./../05-install.mdx#1-check-the-cluster) | | `make-db-secrets.sh` | After the databases are ready, before the platform | [Install step 6](./../05-install.mdx#6-create-the-application-secrets) | | `verify-deployment.sh` | After every install and every upgrade | [Verify the install](./../08-verify.mdx#4-the-platform-serves-an-agent) | | `uninstall.sh` | When removing the installation | [Uninstall](./../operate/uninstall.mdx) | `verify-deployment.sh` reads two environment variables: ```bash export GEN0SEC_BASE_URL=https://cerebellum.example.internal export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) ``` ## The examples Everything under `examples/` is a starting point you are meant to read and edit. Each file carries comments marking which values are yours. | Directory | Purpose | |---|---| | `single-node/` | Three overlays that reduce every component to one node. [Quickstart](./../quickstart.mdx) | | `single-entrypoint/` | One address for the dashboard and every agent-facing API. [Publish the endpoints](./../06-publish.mdx) | ## Verifying the kit's own integrity ```bash cd $KIT && sha256sum -c SHA256SUMS ``` `SHA256SUMS` covers the kit **as delivered**. Adding files makes it no longer match, which is expected once you start using it. ## What the offline bundle method has to transfer Two things, and the order matters. | Transfer | Contains | |---|---| | The bundle tarball | Images, charts, values, the manifest, a copy of the bundler | | The kit directory | `scripts/`, `examples/`, and the manifest | The bundle carries its own copy of `bundler.sh`, so loading it needs no tooling from the kit. **The kit still has to cross**, because the install uses its scripts and its examples. Copy the bundle first, then clean up, then the kit. `create-package` runs inside the kit directory, so the bundle is sitting in it, and copying the kit first transfers tens of gigabytes twice. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). Both archives unpack side by side: ``` ~/gen0sec-cerebellum-multiarch-0.2.2/ the bundle ~/cerebellum-install-kit-0.2.2/ the kit ``` ## The offline-data directory The kit creates `offline-data/` containing a single `README.txt`, and nothing else. It is a placeholder for a data-delivery path that is not part of this release: the four datasets the platform needs are fetched at runtime through the relay, not delivered as files. See [Data relay](./../configure/relay.mdx). If you were told to populate it, check with us first. At this release you should not need to. --- # Requirements Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/requirements # Requirements ## Cluster | | Requirement | |---|---| | Kubernetes | 1.23 or later | | Access | `cluster-admin`, for the install only. See [Privileges and RBAC](./security/privileges.mdx) for what it is used for and what it leaves behind | | StorageClass | A default StorageClass on block storage. **Volume expansion recommended** | | Supported architecture | `linux/amd64` or `linux/arm64`. Mixed-architecture clusters are supported | | LoadBalancer | The ingress data plane is exposed as a `LoadBalancer` type Service by default. See the note below | :::note[The ingress needs to be reachable from outside the cluster] The ingress data plane is a `LoadBalancer` type Service, because nothing it programs can serve traffic if it stays cluster-internal. Your cluster needs a LoadBalancer implementation. If your cluster fronts ingress another way, a NodePort, a hardware balancer or an external proxy, change the Service type to match. ::: Cerebellum brings its own data plane - which contains Postgres, Kafka, object store and cache. You do not connect it to existing instances of any of them. :::info[Maintenance] We've deliberately chosen to use well-known solutions from the software industry in our data layer to ease the burden of maintenance. While Cerebellum ships this data layer by default, to make the setup easier, the data in it is the highest importance to you. So make sure you monitor the dataplane and do backups and maintenances regularly. ::: :::note[Container runtime, for the registry install method only] Our registry challenges pulls that carry no `User-Agent`, and on some distributions the kubelet's pull path sends none. containerd's own user agent passes; the problem is its absence. Either configure one on every node, or use the offline bundle method, which pulls into your own registry instead. Settle this before you choose a method. See [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: ### Kubernetes versions 1.23 is the floor because `autoscaling/v2` is the newest API the charts use, and it reached GA in that release. While anything from 1.23 and up should work, we recommend you use the latest stable version. ## Sizing a production install Production runs on **3 nodes**. Several data plane members each form a three-member quorum, so three is the floor for redundancy, not a suggestion. ### CPU & memory per node | | Resources | |---|---| | **Minimum** | 8 vCPU, 16 GiB | | **Recommended** | 16 vCPU, 32 GiB | ### Storage per node We recommend to start your nodes with `500 GiB` **minimum**. Several of the data plane members use PVCs. While the defaults are kept low - as your platform matures the data will grow and you'll need to expand the storage volumes. ### Sizing storage for your retention The shipped sizes are a starting point, not a capacity plan. They are deliberately modest, so for most production workloads you will be growing them rather than shrinking them. Four keys control everything: | Key | Chart | Default | |---|---|---| | `postgres.volume.size` | `gen0sec-data` | 50 GiB | | `kafka.storage.size` | `gen0sec-data` | 50 GiB | | `rustfs.storageclass.dataStorageSize` | `gen0sec-infra` | 50 GiB | | `rustfs.storageclass.logStorageSize` | `gen0sec-infra` | 10 GiB | Which one to grow depends on what you are retaining: | If you need | Grow | |---|---| | Longer event and telemetry history | `postgres.volume.size`. This is the system of record | | A longer Kafka retention window, or a bigger ingest buffer | `kafka.storage.size` | | More room for datasets and locally-produced artifacts | `rustfs.storageclass.dataStorageSize` | Example for longer retention: ```yaml # gen0sec-data postgres: volume: size: 200Gi kafka: storage: size: 200Gi ``` ```yaml # gen0sec-infra rustfs: storageclass: dataStorageSize: 100Gi ``` :::note[Plan for growth] Volume expansion grows a volume if your StorageClass supports it, and it is straightforward. Nothing shrinks one: reducing a size means recreating the volumes and losing what is on them. Starting at the shipped defaults and growing is therefore the recommended direction. It is the reason for low initial limits. Check that your StorageClass has `allowVolumeExpansion: true` before you install. ::: ### What is not on this page **Throughput.** How many agents, events per second or lookups a given topology sustains is not directly derivable from the charts' parameters. We suggest you scale the platform components and storage sizes according to your topology's specific needs. {/* Four workloads declare no CPU or memory requests: the object store, the Kafka brokers, valkey, and the Postgres operator. They are therefore BestEffort or Burstable and are evicted first under node pressure, which is why the per-node recommendation is driven by limits rather than by the requests total. Raised separately as a chart issue. */} ### Evaluating on one node Use the [Quickstart](./quickstart.mdx). It runs everything at a single replica on one node, for evaluation, training and upgrade rehearsals. ## Tools on your workstation Install these on the machine you run the commands from, not on the cluster. | Tool | Version | Used for | Install | |---|---|---|---| | `helm` | 3.16 or later | Installing the charts, and pulling them from the registry | [helm.sh](https://helm.sh/docs/intro/install/) | | `kubectl` | matching your cluster | Everything cluster-side | [kubernetes.io](https://kubernetes.io/docs/tasks/tools/#kubectl) | | `cosign` | 2.6.3 or later | Verifying signatures on the install kit, images and charts. Older versions cannot see image signatures made by cosign v3 | [docs.sigstore.dev](https://docs.sigstore.dev/cosign/system_config/installation/) | | `skopeo` | any | Copying images between registries | [github.com/containers/skopeo](https://github.com/containers/skopeo/blob/main/install.md) | | `yq` | 4 or later, **mikefarah/yq only** | Reading the release manifest | [github.com/mikefarah/yq](https://github.com/mikefarah/yq#install) | | `jq` | any | Reading the release API | [jqlang.github.io](https://jqlang.github.io/jq/download/) | | `aws` or `mc` | any | Creating the object store bucket, one step during install | [aws-cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) or [MinIO client](https://min.io/docs/minio/linux/reference/minio-mc.html) | | `curl`, `tar`, `openssl`, `sha256sum` or `shasum` | any | Downloading, unpacking, generating secrets, checking checksums | Your OS packages | ### Check your yq There are two separate tools that are called `yq`. The Python one (`kislyuk/yq`) cannot parse the expressions the bundler uses, and it fails with a syntax error partway through. ```bash yq --version ``` The output must contain `mikefarah`. If it does not, install the Go version and point the bundler at it: ```bash export MIKEFARAH_YQ_PATH=/path/to/mikefarah-yq ``` ## Credentials from Gen0Sec You need **two**, and they are different things. Missing the second one produces a working-looking install whose entire fleet silently receives nothing. | Credential | Used by | Used for | |---|---|---| | **Registry token** | The bundler, on your workstation | Pulling charts and images from our registry | | **Your Gen0Sec API key** | The relay pod, inside your cluster | Fetching threat intelligence, GeoIP, models and IDS rules from Gen0Sec | Both are self-service from the dashboard. Neither requires contacting us. :::note[The install kit itself needs no credentials] The kit is published openly. Downloading and verifying it, in [Prepare the artifacts](./04-prepare-artifacts.mdx), happens before you use either credential — so if that download fails, the token is not what to check. The usual cause is a tag that is not published; the [release index](https://releases.gen0sec.com/cerebellum/) lists the ones that are. ::: ### The registry token Create it on the dashboard, under **Repository Access**: ``` https://app.gen0sec.com//repositoryAccess ``` Your workspace id is in the address bar once you are signed in, and the same page is linked from **Clients**. The token is shown **once**, at creation — store it before leaving the page, because it cannot be retrieved afterwards and a lost one has to be replaced. **The username is literally `any`.** Wherever a command asks for one, type `any`. Only the token is checked, so the username can be anything at all — `any` is simply what these docs use so the commands are copy-pasteable. ### Your Gen0Sec API key You can create your API key on the [SaaS Dashboard](https://app.gen0sec.com) Scopes required: - `ids_rules:read`, - `indicators:read`, - `ml_models:read` :::note[Why this one is separate from the registry token] Your agents get their API keys from your own dashboard, and those keys exist only in your cluster's database, so they cannot authenticate to Gen0Sec. The API key created in this step is used by your in-cluster relay to authenticate against the Gen0Sec SaaS, so it can fetch data related to threat intelligence, GeoIP, ML models and IDS rules. The chart expects it as a secret named `gen0sec-download-proxy`, and the relay pod will not start without it, so a missing key is a failed rollout rather than a silently degraded fleet. You will create this secret in the step 2 of [Install](./05-install.mdx). ::: --- # Architecture and data flows Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/security/architecture # Architecture and data flows Written for a security reviewer. Everything here is observable in a running install, and every claim has a page you can check it against. ## What runs where Two namespaces, three charts, one ingress point. ``` internet │ │ api.gen0sec.com:443 (the only required outbound path) │ ┌───────────────────────────┼──────────────────────────────────────────────┐ │ your cluster │ │ │ ▲ │ │ ┌─────────────────────── │ ─────────────────────────────────────────┐ │ │ │ namespace: gen0sec │ │ │ │ │ │ │ │ │ │ dashboard relay(download-api) 13 application services │ │ │ │ │ │ │ └───────────────────────────────┬───────────────────────────────────┘ │ │ │ in-cluster only │ │ ┌───────────────────────────────┼───────────────────────────────────┐ │ │ │ namespace: gen0sec-system │ │ │ │ │ ▼ │ │ │ │ Postgres (3) Kafka (3) object store (4) caches (2) │ │ │ │ Postgres operator Kafka operator ingress operator │ │ │ └───────────────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────┘ ▲ │ one address, TLS, path-prefix routed │ your agents and your users ``` | Namespace | Contains | Reachable from outside | |---|---|---| | `gen0sec` | The application services and the dashboard | Yes, through the ingress you publish | | `gen0sec-system` | Postgres, Kafka, object store, two caches, and the three operators | No | Nothing in `gen0sec-system` is published. The application reaches it by cluster DNS. ## Trust boundaries Four, and they are worth naming separately because they fail differently. | Boundary | Crossed by | Controlled by | |---|---|---| | **Internet to cluster** | The relay, outbound only, one destination | Your firewall. See [Ports and egress](./egress.mdx) | | **Your network to cluster** | Agents and users, inbound, one address | Your ingress and its TLS | | **Namespace to namespace** | The application reading its databases | Cluster DNS. No NetworkPolicy ships. See [Hardening](./hardening.mdx) | | **Cluster to registry** | Image pulls | The image pull secret, one per namespace | ## Data flows ### Inbound: agents and users | Flow | Path | Authentication | |---|---|---| | Agent telemetry and events | Your address, then the ingest services, then Kafka, then Postgres | Agent API key | | Agent configuration pull and push | Your address, then the configuration services | Agent API key | | Agent artifact download | Your address, then the relay, then Gen0Sec | Agent API key, validated locally. The relay uses your installation's own key upstream, so agent keys never leave. See [Data handling](./data-handling.mdx) | | Dashboard sign-in | Your address, then the dashboard, then Postgres | Email and password, or a passkey, held in your Postgres. No external identity provider at this release | ### Outbound: the relay, plus optional dashboard calls The dashboard can also check for new versions and send email. All of it is optional and carries no agent data. See [Data handling](./data-handling.mdx#the-dashboards-optional-connections). The relay has one destination and one purpose: fetching four licensed datasets that are in no artifact we ship. | Dataset | Why it cannot be shipped | |---|---| | CTI indicators | Licensed by upstream providers | | GeoIP databases | Licensed by MaxMind and IPInfo | | ML models | Published by gen0sec, refreshed independently of releases | | IDS rules | Licensed as part of the Emerging Threats ruleset | The relay is a **relay, not a cache**. Nothing is stored on the way through, so there is no local artifact store to size and no staleness window. ### Never outbound | Dataset | What it is | Where it stays | |---|---|---| | `identity` | Workload identity produced in your cluster | Your object store | | `policy-edges` | Pod IPs, namespaces, NetworkPolicy edges | Your object store | Relaying either would push a map of your workload topology to a service you do not run. It is not configurable and there is no flag that enables it. Uploads are not relayed either: the relay is read-only. ## Where data is stored | Store | Contains | Backed up by you | |---|---|---| | Postgres | The system of record: configuration, agents, policy, history | Yes, always | | Object store | Datasets, and the two locally-produced sets above | Yes | | Kafka | In-flight events between ingest and persistence | No, it is transport | | Cache | Nothing durable | No | See [Back up and restore](./../operate/backup.mdx). ## What the platform requires of your cluster | Requirement | Detail | |---|---| | `cluster-admin` during install | For CRDs and cluster-scoped RBAC. See [Privileges and RBAC](./privileges.mdx) | | Custom resource definitions | Postgres and Kafka operator CRDs, cluster-scoped by nature | | Its own ingress controller | Installed by the infra chart, `ingressClassName: synapse` | | One required outbound destination | `api.gen0sec.com:443` | | No node access, no host mounts, no privileged containers | See [Hardening](./hardening.mdx) | ## Questions a reviewer usually asks next | Question | Page | |---|---| | What exactly can leave the cluster? | [Data handling](./data-handling.mdx) | | What ports and destinations do I open? | [Ports and egress](./egress.mdx) | | Why does it need `cluster-admin`? | [Privileges and RBAC](./privileges.mdx) | | How do I verify what you shipped me? | [Supply chain](./supply-chain.mdx) | | What is the container security posture? | [Hardening](./hardening.mdx) | | What does not work yet? | [Known limitations](./../troubleshoot/known-limitations.mdx) | --- # Data handling Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/security/data-handling # Data handling ## What leaves your cluster One required connection, outbound, to one destination. The dashboard can make three optional ones, listed after it. | | | |---|---| | Destination | `api.gen0sec.com:443` | | Made by | The relay pod only | | Direction | Outbound, request and response | | Purpose | Fetching four licensed datasets | **What is in the request.** A path naming the artifact, and your own Gen0Sec API key. Nothing else. The relay does not add a customer identifier, a cluster identifier, an agent identifier, telemetry or usage data. **What is in the response.** The artifact: a CTI indicator database, a GeoIP database, an ML model, or an IDS rule bundle. ### The dashboard's optional connections | Destination | When | What is sent | |---|---|---| | `helm.gen0sec.com:443` | Opening the Clients or Components page | A request for the public chart index. Nothing about your install | | `releases.gen0sec.com:443` | Only if **Check for updates online** is on | A request for the latest release's public version list. Nothing about your install | | Your SMTP server | Only if you configure SMTP | The recipient's address, your organization's name, and an invitation or password-reset link | All three fail without affecting anything else. See [Ports and egress](./egress.mdx). ## What never leaves your cluster | Data | What it is | |---|---| | `identity` | Workload identity produced inside your cluster | | `policy-edges` | Pod IP addresses, namespaces, NetworkPolicy edges | | Agent telemetry and events | Everything your agents report | | Configuration and policy | Everything you configure | | Logs | All of it, to container stdout in your cluster | | Database contents | Postgres never leaves | | Dashboard user accounts, passwords and passkeys | Held in your Postgres | | Audit log | Who changed what, in the dashboard or through the API: the user or API key, the action, the changed fields with secret-looking fields redacted, source IP and user agent | `identity` and `policy-edges` deserve the explicit statement, because they are the two that are technically served by the same service that does the relaying. **They are never relayed.** Doing so would push a map of your workload topology to a service you do not run. It is not configurable, and no flag enables it. The upload surface is not relayed either, because the relay is read-only. ## Who validates agent credentials **Your agents' keys are validated in your cluster, and they never leave it.** Every relayed route sits behind the local authentication middleware, and each one carries the same permission it carries in direct mode. So a key that would be refused when served locally is refused it when relayed. ``` agent --[agent key, authorized locally, permission checked per route]--> relay --[your Gen0Sec API key]--> api.gen0sec.com agent key stops here ``` Two separate concerns, deliberately separated: | Hop | Credential | Checked by | |---|---|---| | Agent to relay | The agent's own key | Your cluster, against your installation, with the artifact's permission | | Relay to Gen0Sec | Your Gen0Sec API key | Gen0Sec | | | | |---|---| | Credentials Gen0Sec receives | One, belonging to your installation | | Credentials Gen0Sec never receives | Your individual agent key | | What Gen0Sec can infer about your fleet | Nothing from the credential. Requests carry no agent or cluster identifier | | Revoking one agent | Local, in your dashboard, immediate | | Revoking your installation's upstream access | Revoke or rotate your Gen0Sec API key | This is the answer to "does our agent inventory leave our perimeter". It does not. :::note[Agent keys are never forwarded upstream, even by accident] On a self-hosted deployment the relay never puts a caller's key in an outbound request, in any configuration. A locally-issued key could not authenticate to Gen0Sec anyway, so forwarding it could only leak a local credential to a third party without making the request succeed. With no Gen0Sec API key configured the relay sends no credential at all rather than sending yours. ::: ## Credentials at rest | Credential | Where it lives | Generated by | |---|---|---| | Database role passwords | Kubernetes secrets | The Postgres operator | | Object store keys | A Kubernetes secret | You, during install | | Registry pull credentials | A Kubernetes secret per namespace | You, during install | | Dashboard session signing key | A Kubernetes secret | Generated once, preserved across regeneration | | SMTP credentials | A Kubernetes secret | You, optional | **No credential appears in a chart, a values file or a rendered manifest.** Every one is referenced by name. See [Secrets](./../configure/secrets.mdx). The install procedure reads every credential from a file or from standard input rather than passing it as a command-line argument, so nothing lands in shell history or in the process list. ## Credentials in transit | Hop | Encrypted | |---|---| | Agents and users to your ingress | Yes, TLS you terminate | | Relay to `api.gen0sec.com` | Yes, TLS | | Services to Postgres | Yes, TLS to the pooler | | Services to the object store | Plain HTTP, in-cluster only | | Services to the cache | Plain, in-cluster only, no credential in the URL | The last two are in-cluster traffic on the pod network. If your threat model includes an attacker on the pod network, add NetworkPolicies and a service mesh. See [Hardening](./hardening.mdx). :::note[One deliberate exception, and why] The dashboard's database connection verifies encryption without verifying the certificate chain, because its client library rejects the pooler's self-signed certificate outright while every other service's library accepts encrypt-without-verify. The traffic is encrypted and in-cluster. Do not copy that setting onto the other services: their driver rejects it. ::: ## Data retention Retention is yours. Nothing expires data on your behalf, and nothing is sent anywhere before it is deleted. | Store | Grows with | Sized by | |---|---|---| | Postgres | Agents, events, history | `postgres.volume.size` | | Postgres audit log | Administrative changes, kept indefinitely | Small. See [Known limitations](./../troubleshoot/known-limitations.mdx#the-audit-log-is-never-pruned) | | Kafka | In-flight events, and its retention window | `kafka.storage.size` | | Object store | Datasets and locally-produced artifacts | `rustfs.storageclass.dataStorageSize` | See [Sizing](./../02-requirements.mdx#sizing-a-production-install). ## Deleting everything [Uninstall](./../operate/uninstall.mdx) without `--keep-data` deletes the namespaces and every persistent volume in them: the database, the message log and the object store. There is no copy anywhere else. --- # Ports and egress Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/security/egress # Ports and egress Same as [Network and connectivity](./../03-network.mdx), reproduced here for better visibility. **One canonical source.** If this table and the install page ever disagree, that is a bug in our publishing, not a judgement call for you. Report it. ## Outbound All TCP, all TLS, all outbound. Nothing needs an inbound rule from the internet. | Destination | Port | Needed by | Purpose | Required | |---|---|---|---|---| | `api.gen0sec.com` | 443 | Cluster pods, the relay only | Threat intelligence, GeoIP, ML models, IDS rules | **Always** | | `releases.gen0sec.com` | 443 | Workstation | Download the install kit and its signature | **Always** | | `registry.gen0sec.com` | 443 | Workstation | Pull charts and images | **Always** | | `registry.gen0sec.com` | 443 | Cluster nodes | Image pulls at install time | Registry install only | | `fulcio.sigstore.dev` | 443 | Workstation | Signature certificate verification | **Always** | | `rekor.sigstore.dev` | 443 | Workstation | Transparency log verification | **Always** | | `tuf-repo-cdn.sigstore.dev` | 443 | Workstation | Signature trust root refresh | **Always** | | Your own registry | 443 or 80 | Cluster nodes, and the host that loads the bundle | Image pulls at install time | Offline bundle only | | `helm.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest agent version, shown on the Clients and Components pages | No. Always attempted. Blocking it only hides the latest version | | `releases.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest component versions | No. Only if an administrator turns on **Check for updates online** | | Your SMTP server | 587 or 465 | Cluster pods, the dashboard only | Invitation and password-reset email | No. Only if you [configure SMTP](./../configure/identity.mdx#invitations-and-email) | **Only one row is required by the running cluster:** `api.gen0sec.com:443`, and only the relay pod uses it. The dashboard rows are optional: blocking them hides version information, or leaves invitations and password resets without email. Everything else is install-time, and most of it is your workstation rather than your cluster. ## Inbound | Source | Destination | Port | Purpose | |---|---|---|---| | Your agents | Your published address | 443 | Telemetry, events, configuration, artifact download | | Your users | Your published address | 443 | Dashboard | One address serves both, routed by path prefix. See [Publish the endpoints](./../06-publish.mdx#the-route-table). ## In-cluster No NetworkPolicy is required, and none ships. If you enforce default-deny, these are the flows to allow. | From | To | Port | |---|---|---| | `gen0sec` services | `core-pooler.gen0sec-system` (Postgres pooler) | 5432 | | Migration job | `core.gen0sec-system` (Postgres primary) | 5432 | | `gen0sec` services | `core-kafka-bootstrap.gen0sec-system` | 9092 | | `gen0sec` services | `rustfs-svc.gen0sec-system` (object store) | 9000 | | `gen0sec` services | `dragonfly.gen0sec-system` (cache) | 6379 | | Ingress controller | `gen0sec` services | Service ports | | `gen0sec` services | `gen0sec` services | Service ports | | Dashboard (`ui`) pods | The Kubernetes API server | 443 or 6443 | The dashboard reads two ConfigMaps from the API server for the Components tab. Find its address with `kubectl get endpoints kubernetes -n default`. Without that flow only the Components tab degrades. See [Hardening](./hardening.mdx) for what we recommend beyond that. ## Notes that save time **The Sigstore hosts are the ones people forget.** Signature verification is the first command of the install, and it fails without them. They are needed on the workstation, not on the cluster. **A proxy does not help for the cluster's one destination.** Explicit forward proxies are not honoured in-cluster. See [Known limitations](./../troubleshoot/known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy). **The offline bundle removes registry egress, not all egress.** The runtime connection stays. See [What this rules out](./../03-network.mdx#what-this-rules-out). --- # Hardening Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/security/hardening # Hardening Two lists: what the charts already do, and what is left to you. The second list is short but it is not empty, and we would rather name it than let you assume otherwise. ## Hardened by default Every application service runs with this security context, and it is not opt-in. ```yaml podSecurityContext: runAsNonRoot: true runAsUser: 65532 runAsGroup: 65532 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: [ALL] ``` | Control | Status | |---|---| | Runs as non-root | Yes, uid 65532 | | Read-only root filesystem | Yes | | Privilege escalation | Disabled | | Linux capabilities | All dropped | | Privileged containers | None | | Host network, PID or IPC | None | | Host path mounts | None | | Node access | None | :::note[Read-only root filesystems were not free] Several services defaulted to writing into a relative path under their working directory, which fails outright on a read-only root. Each one now gets an explicit writable volume mounted where it expects to write. That is why the values file carries per-service volume mounts: they are the cost of this control, not decoration. If you add a service-level override that removes one, that service will fail to start. ::: ## Pod Security Admission The application pods satisfy the **restricted** profile. Label the namespace to enforce it: ```bash kubectl label namespace gen0sec \ pod-security.kubernetes.io/enforce=restricted \ pod-security.kubernetes.io/enforce-version=latest ``` **Do not label `gen0sec-system` restricted yet.** The Postgres and Kafka pods there are created by upstream operators and have not been assessed against that profile. Use `baseline` if you need something, and tell us if you require restricted there. ## What you should add None of these ship, and all of them are worth doing. ### NetworkPolicies **No NetworkPolicy ships with the charts.** On a cluster without default-deny, that means the platform namespaces are as open as everything else. Add default-deny in both namespaces, then allow exactly these: | From | To | Port | |---|---|---| | `gen0sec` services | `core-pooler.gen0sec-system` | 5432 | | Migration job | `core.gen0sec-system` | 5432 | | `gen0sec` services | `core-kafka-bootstrap.gen0sec-system` | 9092 | | `gen0sec` services | `rustfs-svc.gen0sec-system` | 9000 | | `gen0sec` services | `dragonfly.gen0sec-system` | 6379 | | Ingress controller | `gen0sec` services | Service ports | | `gen0sec` services | `gen0sec` services | Service ports | | Dashboard (`ui`) pods | The Kubernetes API server | 443 or 6443 | | Relay pod | `api.gen0sec.com:443` | 443, egress | **Egress default-deny is the high-value one here.** The relay is meant to be the only pod that needs a route out. The dashboard's optional egress is listed on [Ports and egress](./egress.mdx). A NetworkPolicy is what turns that from a design intention into an enforced property, and it is the single strongest control you can add to this deployment. ### TLS everywhere it is not already | Hop | Today | Worth adding | |---|---|---| | Ingress to users and agents | TLS you terminate | Already yours | | Relay to Gen0Sec | TLS | Nothing to do | | Services to Postgres | TLS to the pooler | Nothing to do | | Services to the object store | Plain HTTP, in-cluster | A service mesh, if your threat model includes the pod network | | Services to the cache | Plain, in-cluster, no credential in the URL | Enable cache authentication, then move the connection string into each service's secret | ### Resource limits on the four workloads that have none The object store, the Kafka brokers, the ingress cache and the Postgres operator declare no CPU or memory requests. That makes them BestEffort or Burstable, so the kubelet evicts them first under node pressure. For the object store, holding your data, that is the wrong priority order. Set requests on them, or set a namespace `LimitRange` so nothing lands unbounded. See [Sizing](./../02-requirements.mdx#sizing-a-production-install). ### Secrets from an external manager `make-db-secrets.sh` is fine for evaluation. In production, have your secret manager produce secrets of the same shape so credentials never exist as plain Kubernetes secrets you created by hand. Every name and key is listed in [Secrets](./../configure/secrets.mdx). ## Settings you should never enable Two environment variables exist and both disable a security control. Neither is set by the charts, and neither appears anywhere in the install procedure. | Setting | What it does | When it is acceptable | |---|---|---| | `TLS_INSECURE_SKIP_VERIFY` | Stops the relay verifying the upstream certificate | Never in production. It defeats the protection on your only outbound connection | | `MIRROR_TLS_VERIFY=false` | Stops the bundler verifying your registry's certificate when pushing | Only for a plain-HTTP registry on a trusted network, and preferably not then | If you find either one set in a running deployment, treat it as a finding. ## Verifying the posture Confirm no pod runs as root or privileged: ```bash kubectl -n gen0sec get pods -o json | jq -r ' .items[] | .metadata.name + " runAsNonRoot=" + ((.spec.securityContext.runAsNonRoot // false) | tostring)' ``` Confirm nothing holds cluster-admin: ```bash kubectl get clusterrolebinding -o json \ | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | .metadata.name' ``` Confirm only the relay can reach the internet, once you have added egress policy: ```bash kubectl -n gen0sec run netcheck --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/busybox:1.36.1-glibc -- wget -T5 -qO- https://example.com ; echo "exit=$?" ``` A non-zero exit is the result you want. --- # Privileges and RBAC Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/security/privileges # Privileges and RBAC ## The short answer `cluster-admin` is required **to install**, not to run. It is needed for two things that are cluster-scoped by nature: 1. Installing custom resource definitions. 2. Creating the cluster roles the three operators need. Once installed, no component in the platform holds cluster-admin. ## What is cluster-scoped ### Custom resource definitions From the bundled operators, not written by us. | CRD group | Installed by | Kinds | |---|---|---| | `acid.zalan.do` | Postgres operator | `postgresqls`, `operatorconfigurations`, `postgresteams` | | `kafka.strimzi.io` | Kafka operator | `kafkas`, `kafkatopics`, `kafkanodepools`, `strimzipodsets`, and the Connect and Bridge kinds we do not use | CRDs are cluster-scoped in Kubernetes. There is no namespaced alternative. :::note[Deleting a CRD deletes every resource of that kind, cluster-wide] Including any outside these two namespaces. If anything else in your cluster runs a Postgres or Kafka operator, removing these CRDs would take its databases and clusters with them. [Uninstall](./../operate/uninstall.mdx) removes them by default, but checks first: if any of those kinds is in use outside the two namespaces it is removing, it skips the CRDs and says so. `--keep-crds` skips them unconditionally. ::: ### Cluster roles | Cluster role | Belongs to | What it is for | |---|---|---| | `g0s-postgres-operator` | Postgres operator | Managing Postgres clusters and their pods | | `g0s-postgres-operator-endpoints` | Postgres operator | The `endpoints` permission its own upstream role omits, while still creating a master Endpoints object per cluster | | `postgres-pod` | Postgres pods | What a database pod itself needs | | `strimzi-cluster-operator-global` | Kafka operator | Cluster-scoped reconciliation | | `strimzi-cluster-operator-namespaced` | Kafka operator | Namespaced resources, bound per namespace | | `strimzi-cluster-operator-leader-election` | Kafka operator | Leader election | | `strimzi-cluster-operator-watched` | Kafka operator | Watching the namespaces it manages | | `strimzi-kafka-broker` | Kafka brokers | What a broker pod needs | | `strimzi-entity-operator` | Topic and user operator | Managing topics and users | | `strimzi-kafka-client` | Kafka clients | Client-side permissions | | `g0s-synapse-operator` | Ingress operator | Watching Ingress objects and services | Eleven, and nine of them come straight from the two upstream operators. :::note[The one we added, and why] `g0s-postgres-operator-endpoints` exists because the Postgres operator ships a cluster role without `endpoints`, yet creates a master Endpoints object for every cluster it manages. Without that rule, `postgresql/core` fails with `endpoints is forbidden` and the database never starts. See [Postgres will not start](./../troubleshoot/symptoms.mdx#postgres-will-not-start). ::: ## What the application itself holds The application services in `gen0sec` are not operators and do not watch the cluster. | | | |---|---| | Cluster-scoped permissions | None | | Namespaced permissions | None, except the dashboard (see below) | | Service accounts | One per service. Tokens are not mounted, except the dashboard's | | Node access | None | | Host mounts | None | | Privileged containers | None | :::note[The one exception: the dashboard reads two ConfigMaps] The `ui` service account (`g0s-ui` in `gen0sec`) is bound to the Role `g0s-ui-read-installed-versions` in `gen0sec-system`, which the infra chart creates. It allows `get` on two ConfigMaps by name, `g0s-infra-installed-versions` and `g0s-data-installed-versions`, and nothing else. There is no `list` or `watch`, so it cannot enumerate other ConfigMaps. The dashboard uses it for the [Components](./../configure/components.mdx) tab, and is the only pod with its service account token mounted. ::: See [Hardening](./hardening.mdx) for the container security context. ## Can I install without cluster-admin? Partly, if you split the install between two roles. This is a supported pattern but it is not the documented golden path, so plan for one extra iteration. | Step | Needs | Who | |---|---|---| | Apply CRDs | Cluster-scoped write on `customresourcedefinitions` | Your platform team, once per release | | Create cluster roles and bindings | Cluster-scoped RBAC write | Your platform team, once | | Install the three charts | Namespace-admin on both namespaces | The installer | Render what a chart would create, and hand that to whoever holds the privilege: ```bash helm template g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system -f $B/values/gen0sec-infra-values-onprem.yaml \ | yq eval-all 'select(.kind == "ClusterRole" or .kind == "ClusterRoleBinding")' ``` ## Verifying after install Confirm nothing holds more than it should: ```bash kubectl get clusterrolebinding -o json \ | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | .metadata.name + " -> " + ([.subjects[]?.name] | join(","))' ``` No entry in that output should reference a `gen0sec` service account. If one does, tell us: that is a defect, not a configuration choice. ## What an uninstall leaves behind | | Removed | Reason | |---|---|---| | Namespaced resources | Yes | Helm owns them | | Objects the operators created outside the releases | Yes | The script removes them; `helm uninstall` alone does not | | CRDs | Yes, unless in use elsewhere or `--keep-crds` | Removal is cluster-wide destructive, so the script checks before it acts | | Cluster roles and bindings | Depends on the chart that created them | Check what remains and remove by hand if nothing else uses the operators | [Uninstall](./../operate/uninstall.mdx) reports what it removed and what it left. --- # Provenance, signatures and SBOM Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/security/supply-chain # Provenance, signatures and SBOM Everything we publish is signed, and every signature is verifiable by you without asking us for a key. This page also includes the one case where verification cannot follow the artifacts. ## What is signed, and how | Artifact | Signed | Attested | |---|---|---| | Install kit tarball | Detached cosign bundle, published beside it | | | Container images | `cosign sign --recursive`, so the index **and** each per-architecture manifest | SPDX JSON SBOM, one per architecture | | Helm charts | `cosign sign` on the OCI chart artifact | SPDX JSON SBOM | **Keyless, with GitHub OIDC.** There is no long-lived private key to steal, and no public key for you to fetch and trust. Each signature is bound to a workflow identity recorded in a public transparency log, so verification asserts *who built it*, not *who holds a key*. SBOMs are generated with syft and attached as SPDX JSON attestations. :::note[SLSA provenance is deliberately off] BuildKit's provenance attestations add manifests to the multi-architecture index carrying a placeholder platform. That breaks tools which walk an index by platform, including our own mirroring. We ship the SPDX SBOM attestation instead, which is the thing customers actually verify. If you require SLSA provenance, tell us: it is a build-flag change plus mirroring work, not a redesign. ::: ## Verify the install kit Do this before you unpack it. It is step 3 of [Prepare the artifacts](./../04-prepare-artifacts.mdx). ```bash cosign verify-blob \ --bundle cerebellum-install-kit.tar.gz.cosign.bundle \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/release\.yaml@refs/tags/' \ cerebellum-install-kit.tar.gz ``` **Both certificate flags are required.** `cosign` refuses to verify without an expected issuer and an expected identity, so a shortened command fails rather than passing weakly. The identity is pinned to **one workflow, on a tag reference**. A looser pattern would accept a signature produced by any workflow, on any branch, in our organization. That is the difference between "built by our release pipeline from a tag" and "built by somebody at Gen0Sec". ## Verify an image or a chart Against our registry, or against yours after mirroring: ```bash cosign verify \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/' \ registry.gen0sec.com/gen0sec/auth-api:0.2.2 ``` Two images are built and signed in their own repositories, so their identity names that repository: | Image | Identity regexp | |---|---| | `spilo` (Postgres) | `^https://github\.com/gen0sec/spilo/\.github/workflows/build-images\.yaml@refs/tags/` | | `synapse-operator` | `^https://github\.com/gen0sec/synapse-operator/\.github/workflows/build-images\.yaml@refs/tags/` | Both are signed by cosign v3, which stores signatures where cosign before 2.6.3 cannot find them. An older cosign reports `no signatures found`. ## Read the SBOM ```bash cosign download attestation \ --predicate-type https://spdx.dev/Document \ registry.gen0sec.com/gen0sec/auth-api:0.2.2 \ | jq -r '.payload | @base64d | fromjson | .predicate' ``` The predicate is an SPDX JSON document. Feed it to whatever consumes SBOMs in your pipeline. ## Verification across the air gap This is the part nobody else documents, so it gets its own section. ### What crosses, and what does not Signature verification needs the public transparency log, and an isolated network does not have one. So the bundler verifies **once, on the connected staging host, before packaging**, and protects everything downstream with checksums. ``` connected staging host isolated network ────────────────────── ──────────────── pull every artifact by digest cosign verify each one (cannot re-verify: no transparency log) write SHA256SUMS -> verify-package re-checks SHA256SUMS print the bundle fingerprint -> mirror re-checks it before pushing package ``` | Stage | Guarantee | |---|---| | Before packaging | Full cryptographic signature verification, every image and every chart | | The tarball | `SHA256SUMS` over every file, plus a **fingerprint**, which is a sha256 of that list | | On the isolated side | `verify-package` and `mirror` both re-check `SHA256SUMS` and refuse to proceed on a mismatch | **Record the fingerprint** that verification prints on the connected side, out of band. Comparing it on the isolated side is what closes the loop between the two networks. :::note[Signature re-verification inside the isolated network is not supported] It would need the transparency log, or a pinned trust bundle carried across and kept current. We do not pretend otherwise. If your policy requires cryptographic verification at the destination rather than checksum verification, tell us. That is a different design, and we would rather discuss it now than have you find the gap during an audit. ::: ### The one case where attestations are lost `--arch amd64` or `--arch arm64` halves the transfer. It also extracts **one manifest out of each multi-architecture index**, which changes every image digest by definition. | | Multi-architecture bundle (default) | Single-architecture bundle | |---|---|---| | Image signatures | Verify | **Verify.** Each per-architecture manifest is signed individually | | SBOM attestations | Verify | **Lost.** They hang off the index, and the index is what is left behind | | Digests match the release manifest | Yes | No, by definition | The bundler warns at the time and annotates the bundle rather than letting this be a silent difference. **If your acceptance process checks attestations, stay on the default.** ## Digest pinning `release-manifest.yaml` in the install kit pins **every image and chart to a digest**, not to a tag. Tags move. Digests do not. That is what makes a release reproducible: the bundler resolves nothing at build time, it fetches exactly the digests the manifest names, and it fails if one is missing. ```bash yq '.spec.images[] | [.image, .imageTag, .digest] | @tsv' $KIT/release-manifest.yaml ``` ## If verification fails | Symptom | Cause | |---|---| | `no matching signatures` | Wrong identity pattern, or the artifact is not ours | | Certificate fetch errors | The Sigstore hosts are blocked. See [Network and connectivity](./../03-network.mdx#egress-list) | | The image verifies, the SBOM does not | Single-architecture bundle. Expected, see above | | `SHA256SUMS` mismatch on the isolated side | The transfer altered the bundle. Re-transfer, do not proceed | **A failed verification is not a step to skip.** The bundler has a flag that drops the check, and a bundle built with it is annotated as unverified. Use it only if cosign genuinely cannot be installed on your staging host, and record that you did. --- # Collect diagnostics Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/troubleshoot/diagnostics # Collect diagnostics Collect this before you contact us. Everything here is something we would otherwise ask for, so sending it means the first reply is an answer rather than a request for more output. There is no single collection command at this release. This page is the list. ## Set up Run the commands from a directory you can archive afterwards. ```bash NS_SYS=gen0sec-system NS_APP=gen0sec OUT=gen0sec-diagnostics mkdir -p $OUT && cd $OUT ``` Cluster and storage: ```bash kubectl version -o yaml > cluster-version.yaml kubectl get nodes -o wide > nodes.txt kubectl describe nodes > nodes-describe.txt kubectl get storageclass -o yaml > storageclasses.yaml kubectl get pvc -A -o wide > pvc.txt ``` Releases and workloads: ```bash helm list -A > helm-releases.txt for ns in $NS_SYS $NS_APP; do kubectl -n $ns get all -o wide > workloads-$ns.txt kubectl -n $ns get events --sort-by=.lastTimestamp > events-$ns.txt kubectl -n $ns get pods -o yaml > pods-$ns.yaml done ``` Logs from anything not ready: ```bash for ns in $NS_SYS $NS_APP; do for p in $(kubectl -n $ns get pods -o json \ | jq -r '.items[] | select(any(.status.containerStatuses[]?; .ready == false)) | .metadata.name'); do kubectl -n $ns logs $p --all-containers --tail=2000 > logs-$ns-$p.txt 2>&1 kubectl -n $ns logs $p --all-containers --previous --tail=2000 > logs-$ns-$p-previous.txt 2>&1 done done ``` Operators and custom resources: ```bash kubectl -n $NS_SYS logs deploy/g0s-infra-postgres-operator --tail=2000 > logs-postgres-operator.txt kubectl -n $NS_SYS logs deploy/strimzi-cluster-operator --tail=2000 > logs-kafka-operator.txt kubectl -n $NS_SYS logs deploy/g0s-infra-synapse-operator --tail=2000 > logs-ingress-operator.txt kubectl -n $NS_SYS get postgresql,kafka,kafkanodepool,kafkatopic -o yaml > custom-resources.yaml ``` The migration job, which is the most common single cause: ```bash kubectl -n $NS_APP logs job/g0s-db-migrate --tail=2000 > logs-db-migrate.txt 2>&1 ``` Images actually in use, including init containers: ```bash for ns in $NS_SYS $NS_APP; do kubectl -n $ns get pods -o yaml | grep -E '^\s+image:' | sort -u > images-$ns.txt done ``` Routing and the relay: ```bash kubectl -n $NS_APP get ingress -o yaml > ingress.yaml kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' > relay-healthz.json 2>&1 ``` Then pack it: ```bash cd .. && tar -czf gen0sec-diagnostics-$(date +%Y%m%d-%H%M%S).tar.gz gen0sec-diagnostics ``` :::danger[Read the archive before you send it] It is plain text and it is yours. Nothing here redacts anything, so check it, and remove any sensitive data it might have, particularly if you added `helm get values`, which prints whatever you set. None of the commands above read secret contents, database rows or object store objects. Logs can contain request paths, key prefixes, up to 400 bytes of a telemetry body that failed to decode, and the full content of any block event the pipeline parked. ::: :::note[There should be one command for this] A single `collect-diagnostics.sh` in the install kit would replace this page with two lines, and the list above is the specification for it. A single diagnostics tool can accidentally collect data that's sensitive and may fail to redact that information, so that's why we ask you to assemble your own report. Once we're 100% sure we can provide a diagnostics tool that doesn't leak any information of your sensitive data, we'll do so. ::: ## What to tell us alongside it Four lines, and they save a round trip: 1. Which install method you used. 2. The release version. 3. The last command you ran, verbatim. 4. What you expected to happen. --- # Known limitations in this release Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/troubleshoot/known-limitations # Known limitations If you hit something that is not on this page, we'd like to hear about it! ## No zero-egress deployment **What it means.** Your cluster must reach `api.gen0sec.com` on port 443. That connection carries threat intelligence, GeoIP, ML models and IDS rules, which are licensed upstream data that is in no artifact we ship. **What you can remove.** All dependency on Gen0Sec-hosted registries. The [offline bundle method](./../04-prepare-artifacts.mdx) mirrors every image and chart into your own registry, so your cluster pulls nothing from us. **What you cannot remove.** The runtime connection. There is no supported configuration with zero cluster egress. **If your policy forbids all egress**, please contact us. See [Network and connectivity](./../03-network.mdx). ## Our registry challenges pulls that send no user agent **By design on our side, and it can decide which install method you use.** **Symptom.** Every image pull from `registry.gen0sec.com` returns **403 with a CAPTCHA challenge**, and pods sit in `ImagePullBackOff`. A manual `docker pull` or `skopeo copy` from the same node **succeeds**. That discrepancy is the whole diagnosis. If the CLI works and the kubelet does not, this is it. **Cause.** On some Kubernetes distributions, the component that services the kubelet's pull sets no `User-Agent` header. Go's `net/http` then sends its own default, `Go-http-client/1.1`, and our edge treats that as a scanner signature and issues a challenge. A kubelet cannot solve a CAPTCHA, so the pull never succeeds and never will. It is not your credentials, not the pull secret, not TLS, and not an allowlist. ### What passes and what does not Measured against our registry: | User-Agent | Result | |---|---| | `containerd/1.7.0` | 200 | | `containerd/v2.1.4` | 200 | | `docker/29.4.0` | 200 | | `curl/8.7.1` | 200 | | `skopeo/1.24.0` | 200 | | `Go-http-client/1.1` | **403, challenged** | | `Go-http-client/2.0` | **403, challenged** | **containerd's own user agent is fine.** Any real one is. The failure is specifically the absence of a user agent, which Go turns into `Go-http-client`. ### Confirming it The blocked response identifies what it matched: ``` "user_agent": "Go-http-client/1.1", "ja4_fingerprint": null, "captcha_validated": false ``` Compare a manual pull against what the kubelet gets: ```bash # on the node: succeeds, because the CLI sends its own user agent skopeo inspect docker://registry.gen0sec.com/gen0sec/auth-api:0.2.2 ``` ```bash # what the kubelet sees kubectl -n gen0sec describe pod | grep -A3 Failed ``` Success from the CLI plus 403 from the kubelet confirms it. ### Two ways forward **Option 1: use the offline bundle method.** Nothing to configure on your nodes, and your own registry does not inspect user agents. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). **Option 2: make the pull path send a user agent**, on every node. :::note[Settle this before you choose an install method] If you cannot change container runtime configuration on your nodes, whether by policy or because your distribution manages it, use the offline bundle method. This constraint forces that choice, and it is better made before you start than at the first failed pull. ::: ## The relay cannot use an explicit egress proxy **Symptom.** You set `HTTPS_PROXY` on the relay. Nothing changes, and nothing errors. **Cause.** Every HTTP client in the platform builds its own transport without proxy support, so `HTTP_PROXY` and `HTTPS_PROXY` are ignored inside the cluster. This is by design. **What works instead.** | Option | Notes | |---|---| | Allow `api.gen0sec.com:443` directly | One firewall exception that bypasses the proxy | | Redirect transparently at the network layer | The client does not need to be proxy-aware | A TLS-intercepting proxy also cannot work: the charts expose no way to mount a CA bundle into the platform containers, and the relay rejects the intercepted certificate. ## The object store bucket is not created for you **Symptom.** Services start and then fail to read or write the object store. `download-api` answers `500 NoSuchBucket`. **Cause.** Nothing in the charts creates the bucket. **What to do.** [Install](./../05-install.mdx#5-create-the-object-store-bucket) step 5 creates it. It is a first-class step, not an optional one. ## Three services report readiness from a weaker probe `download-api`, `ids-rules-api` and `service-graph-api` serve a richer `/status/healthz` and, in doing so, never register `/status/readyz`. Requesting it returns 404, so a readiness probe pointed at it fails forever while liveness stays healthy, and the pod sits at `0/1` with zero restarts. **What that means for you.** Do not repoint those services at `/status/readyz`. It will fail. ## Helm never upgrades CRDs Not our defect, but it will bite you on the first upgrade that contains CRDs, so it is here. Helm installs a chart's CRDs on first install and ignores them from then on. An upgrade that ships a new CRD schema fails with a server-side-apply error, or silently rejects new fields. [Upgrade](./../operate/upgrade.mdx) applies them by hand as step 1. Do not skip it. ## Kafka volumes survive an uninstall `deleteClaim: false` is deliberate, so `helm uninstall` leaves the Kafka volumes behind. Reinstalling against them produces `Invalid cluster.id ... Expected X, but read Y`, because the new cluster generates a new identity. Delete the volume claims first, or restore the original cluster. See [Kafka will not start](./symptoms.mdx#kafka-will-not-start). ## The audit log needs new partitions after two years The upgrade to v0.2.0 creates monthly audit log partitions for the previous month and the next 24. Nothing creates later ones yet. Past that, audited actions fail with `no partition of relation "audit_log" found for row`, and rule changes made through the API are lost. Create partitions ahead of time, as the database superuser: ```sql SELECT create_audit_log_partition(date '2028-11-01'); ``` ## The audit log is never pruned It is append-only and kept indefinitely, and the application cannot delete from it. To remove old history, back up and then drop a monthly partition (`audit_log_YYYY_MM`) as the superuser. ## Emails are sent as "My App" Invitation and password-reset emails carry the sender name `My App`, whatever you configure. Only the address, `HOST_ALIAS_EMAIL`, is yours. ## New WAF rule functions need current agents `regex_replace()` and `http.request.raw_path` validate from v0.2.0. The rule language is generated from Synapse 0.8.5, and agents on an older version may not evaluate a rule that uses them. Update agents before you rely on either. ## Connection-state rules need current agents A firewall rule with a connection-state condition, such as `allow established,related`, is only safe on an agent that understands the condition. One that does not ignores it, and the rule then matches every packet. So when a workspace has such a rule, its config is published only if every agent that reported in the last hour advertises support: Synapse 0.8.0 or later, with firewall rules enabled. If one does not, the workspace **keeps its previous config** and no rule change reaches its agents. The config generator names the agents holding it back: ```bash kubectl -n gen0sec logs deploy/g0s-config-generator | grep conn_state ``` Upgrade the listed agents, or remove the rule. Removing only its condition widens it to every packet. An agent silent for an hour stops counting; set `CAPABILITY_STALE_AFTER_SECONDS` to change that. See [Values](./../configure/values.mdx#services). ## Migration `force` means the opposite of what people expect `migrate force ` marks that version **applied** and continues with the next one. To re-run a migration that failed, force the version **before** it. Migrations are not idempotent. Forcing to the wrong version re-runs DDL that already exists and fails with `already exists`. Example: if `20261004150000` fails, run `force 20261004120000`. --- # Symptom index Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/troubleshoot/symptoms # Symptom index Everything here was reproduced on a real cluster. ## Images ### ImagePullBackOff Three different causes, and the first thing to establish is which. ```bash kubectl -n get pod -o jsonpath='{.spec.containers[*].image}' kubectl -n describe pod | tail -20 ``` | The image reference is | Cause | Fix | |---|---|---| | Correct, and the error mentions authorization | The `gen0sec-registry` secret is missing from that namespace | It is needed in **both** namespaces. See [Install step 2](./../05-install.mdx#the-image-pull-secret) | | Pointing at the wrong registry | A values file was not passed with `-f` | Re-run with `-f $B/values/-values-onprem.yaml` | | A path that does not exist in your registry | Mirror path mismatch, on the offline bundle method | See the next entry | ### 403 with a CAPTCHA challenge from our registry All images, not one, and only on the registry install method. **The tell:** a manual `docker pull` or `skopeo copy` from the same node succeeds while the kubelet's pull returns 403. ```bash # succeeds: the CLI sends its own user agent skopeo inspect docker://registry.gen0sec.com/gen0sec/auth-api:0.2.2 ``` ```bash # 403: what the kubelet gets kubectl -n gen0sec describe pod | grep -A3 Failed ``` On some distributions the component servicing the kubelet's pull sets no `User-Agent`, so Go sends `Go-http-client/1.1`, which our edge challenges as a scanner signature. A kubelet cannot solve a CAPTCHA, so it never recovers. Not credentials, not TLS, not the pull secret. Fix: set a user agent in the container runtime on every node, or use the offline bundle method. Both in [Known limitations](./known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ### ImagePullBackOff on a pod you did not configure Operator-launched pods (Postgres, the connection pooler, Kafka brokers, the entity operator) get their image reference from the **operator**, not from the chart. If one of those is stuck, the mirror path is wrong, not the pull secret. Confirm the path exists in your registry: ```bash curl -su : https://$REGISTRY/v2/_catalog?n=200 | tr ',' '\n' ``` Two paths look surprising and are not negotiable: | Image | Must be mirrored to | |---|---| | Kafka broker | `/gen0sec/strimzi/kafka:1.0.0-kafka-4.1.1` | | Entity operator | `/gen0sec/strimzi/operator:1.0.0` | Everything is under `gen0sec/`. Kafka additionally keeps the upstream `strimzi/` path inside it, because the operator's `defaultImageRegistry` replaces only the registry host and keeps the repository path. A Kafka pod asking for `/strimzi/kafka` with no `gen0sec/` means that setting lost the namespace. ### 401 Unauthorized pulling into a Postgres or pooler pod The Postgres operator does not copy `imagePullSecrets` onto the database pods it creates. They inherit them from their ServiceAccount, which `values-onprem.yaml` configures through `configKubernetes.pod_service_account_definition`. If you overrode `configKubernetes` wholesale, you dropped it. Merge instead of replacing. ### manifest unknown A chart is asking for a tag that was never published. Check what it rendered: ```bash helm get manifest -n | grep -E '^\s+image:' | sort -u ``` Then compare against [Component versions](./../configure/components.mdx). ## Postgres ### Postgres will not start `postgresql/core` sits in `CreateFailed`. ```bash kubectl -n gen0sec-system logs deploy/g0s-infra-postgres-operator | grep -i 'could not create' ``` | Log line contains | Cause | Fix | |---|---|---| | `endpoints is forbidden` | The operator's own ClusterRole omits `endpoints`, but it creates a master Endpoints object per cluster | The infra chart adds the missing rule. Confirm: `kubectl get clusterrole g0s-infra-postgres-operator-endpoints` | | `Unsupported value: "18"` | The CRDs in the cluster are older than the operator chart | Helm never upgrades CRDs. Apply them first. See [Upgrade](./../operate/upgrade.mdx) | | Nothing, and no volume is bound | No default StorageClass, or not enough capacity | See [Sizing](./../02-requirements.mdx#sizing-a-production-install) | After fixing the cause the operator does not retry promptly. Force it: ```bash kubectl -n gen0sec-system rollout restart deploy/g0s-infra-postgres-operator ``` ### Migration failures The migration job runs as a Helm hook. On failure Helm aborts and deploys nothing, so the cluster is in a safe state. ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` | Log line contains | Cause | Fix | |---|---|---| | `relation "..." does not exist` | The chain is running against a database that is not at the version it expects. Usually a forced version, or a `db-migrate` image older than the chart | Read the version table in the next entry before you force anything, then [collect diagnostics](./diagnostics.mdx) | | `role "..." does not exist` | A database role is missing. Older migrations `GRANT` to roles by name and roll back if one is absent. From v0.2.0 a missing role skips the grant with a `NOTICE`, and the service fails later with `permission denied` | The data chart creates them. Confirm: `kubectl -n gen0sec-system get secret \| grep credentials` | | `already exists` | A migration was forced to the wrong version and re-ran DDL | See the next entry | | `canceling statement due to lock timeout` | `20261004150000` could not lock the block events table within 10 seconds. Nothing changed | See [Upgrade](./../operate/upgrade.mdx#5-upgrade-the-platform) | | `no migration found for version` | The `db-migrate` image is older than the schema, typically after a rollback | Use the image of the newest release that ran | ### The schema is marked dirty The migration tool marks the schema dirty when a migration fails halfway, and refuses to continue until you say what happened. ```bash kubectl -n gen0sec-system exec -it core-0 -c postgres -- \ psql -U postgres -d ax_prod -c 'select * from schema_migrations' ``` `force ` means "**this version is applied**, continue with the next one". So to re-run a failed migration, force the version **before** it. Migrations are not idempotent. ## Kafka ### Kafka will not start Broker crash loops with `Invalid cluster.id ... Expected X, but read Y`. The volume survived a previous `helm uninstall`, which is deliberate, and the recreated cluster generated a new identity. Either restore the original cluster or wipe the volumes: ```bash kubectl -n gen0sec-system delete kafkanodepool kafka kubectl -n gen0sec-system delete pvc -l strimzi.io/cluster=core helm upgrade g0s-data ... ``` **This deletes the message data on those volumes.** ### No KafkaTopic resources exist although the chart rendered them Helm can skip re-applying objects that were deleted out of band. Re-apply the stored manifest: ```bash helm get manifest g0s-data -n gen0sec-system | kubectl apply -f - ``` ### Rule changes made through the signal API never appear The request is accepted, but the rule is not created. The signal worker writes an audit record in the same transaction as the rule, and drops the change if that fails. ```bash kubectl -n gen0sec logs deploy/g0s-signal | grep 'Failed to process v2 signal' ``` `permission denied for table audit_log` means the grant was skipped during the migration. `no partition of relation "audit_log"` means the audit log ran out of partitions. See [Known limitations](./known-limitations.mdx#the-audit-log-needs-new-partitions-after-two-years). ### Block events are missing from the dashboard Rows the database refused are parked on the `block_events_dlq` Kafka topic, with the reason in a `dlq_reason` header. Only the refused row is parked, not the batch it arrived in. telemetry-api also logs what it dropped or stored without a field, once a minute per organization: ```bash kubectl -n gen0sec logs deploy/g0s-telemetry-api | grep 'OTLP block events rejected' ``` ## Object store and cache ### Services cannot reach the object store The object store Service is `rustfs-svc`, with the suffix. `S3_ENDPOINT` and `ARXIGNIS_DATA_URL` must match it. ```bash kubectl -n gen0sec-system get svc | grep rustfs kubectl -n gen0sec get deploy g0s-download-api \ -o jsonpath='{.spec.template.spec.containers[0].env}' | tr ',' '\n' | grep -E 'S3_ENDPOINT|DATA_URL' ``` See [Install step 7](./../05-install.mdx#7-install-the-platform). ### 500 NoSuchBucket The bucket does not exist. [Install step 5](./../05-install.mdx#5-create-the-object-store-bucket) creates it. Nothing in the charts does. ### Everything works but reads are slow The shared cache is never fatal. Missing or unreachable, each service logs a warning once and runs on its in-process cache only. Confirm from the service side, not the pod list: ```bash kubectl -n gen0sec logs deploy/g0s-auth-api | grep -i "L2 cache" ``` `L2 cache (Redis) connected to server` means the cache is in use. `falling back to L1-only cache` means `REDIS_URL` is wrong or unreachable. ```bash kubectl -n gen0sec-system get svc dragonfly ``` The Service is deliberately not prefixed with the release name, because `REDIS_URL` names it directly. ### The cache pod crash loops ```bash kubectl -n gen0sec-system logs deploy/dragonfly --previous ``` | Log shows | Cause | Fix | |---|---|---| | `io_uring` errors | Blocked by the host kernel or a seccomp profile | Add `--force_epoll` to `dragonfly.extraArgs` in the infra values | | Read-only filesystem errors | The pod runs with a read-only root, which is why the chart disables snapshotting | Remove any flag you added that re-enables it | ### The cache is being evicted or OOM-killed `maxMemory` is the eviction threshold and the memory limit is the kill threshold. Keep the first comfortably below the second, or the kubelet kills the pod before it gets a chance to evict. ## Install and workstation ### The bundler fails with a yq syntax error You have the wrong `yq`. Two unrelated tools share the name. ```bash yq --version ``` The output must contain `mikefarah`. If not, install the Go version and point the bundler at it with `MIKEFARAH_YQ_PATH`. See [Requirements](./../02-requirements.mdx#check-your-yq). ### zsh: no matches found, before helm runs `zsh` expands `[0]` in a `--set` argument as a glob. Single-quote the whole argument: ```bash --set 'services.ui.ingress.hosts[0].host=example.internal' ``` ### The namespace hangs in Terminating Strimzi puts a finalizer on all 23 KafkaTopics, and once the operator is gone nothing clears it. ```bash kubectl get kafkatopics.kafka.strimzi.io -n gen0sec-system -o name \ | xargs -r -n1 kubectl patch -n gen0sec-system --type=merge \ -p '{"metadata":{"finalizers":null}}' ``` `uninstall.sh` does this for you, in the right order. See [Uninstall](./../operate/uninstall.mdx). ### A pod reports Running but nothing works A pod in `CrashLoopBackOff` still reports `status.phase: Running`. Read the `READY` column, never a field selector on phase. ```bash kubectl -n gen0sec get pods ``` ### Agents get 401 on every relayed artifact Your Gen0Sec API key is wrong, or the secret holds a trailing newline. ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` ``` artifact relayed from upstream path=/v1/ids-rules/version status=401 ``` Recreate the secret with `printf`, not `echo`. `--from-file` stores the bytes exactly, so a newline becomes part of the key. See [Data relay](./../configure/relay.mdx#getting-the-key-right). A missing secret does not produce this: the relay pod fails to start instead. See the next entry. ### CreateContainerConfigError on the relay pod The pod references a secret that does not exist. Almost always `gen0sec-download-proxy`: `downloadProxy.apiKeySecret.name` is set but the secret was never created. ```bash kubectl -n gen0sec describe pod -l app.kubernetes.io/name=download-api | tail -20 kubectl -n gen0sec get secret gen0sec-download-proxy ``` Create it. See [Install step 2](./../05-install.mdx#your-gen0sec-api-key). This failure is the visible one. The dangerous case is the opposite: leaving `apiKeySecret.name` unset renders no secret reference at all, the pod starts, and the failure is the silent 401 above. ### The relay reports 502 on every relayed route The upstream is unreachable. A 502 is deliberate rather than a 404 or 500: a polling agent reads it as "ask again later" and keeps the artifact it already holds. ```bash kubectl -n gen0sec exec deploy/g0s-download-api -- curl -fsS localhost:9999/status/healthz ``` Check that `api.gen0sec.com:443` is allowed. See [Ports and egress](./../security/egress.mdx). If you route egress through an explicit proxy, that is the cause: proxies are not honoured in-cluster. See [Known limitations](./known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy). ## Routing ### An agent gets HTML, or a redirect, from an API The request reached the dashboard's catch-all instead of the intended service. Routing is by longest matching prefix and the dashboard owns `/`, so any path missing from the route table lands there. ```bash kubectl -n gen0sec get ingress -o wide ``` Compare against [Publish the endpoints](./../06-publish.mdx#the-route-table). The failures this causes are quiet by design of HTTP, not by ours: | Missing route | What the agent sees | |---|---| | `/v1/authcheck` | A redirect and HTML. The agent cannot distinguish a bad key from a routing mistake and continues unverified | | `/v1/agents/events` | `SSE connect returned HTTP 404`, then silent fallback to polling, which looks like a slow platform | | `/v1/ban-ledger` | A redirect the agent reads as success. The contribution is lost silently | ### kubectl get ingress shows an empty HOSTS column The ingress controller ignores host-less rules: the object is created, nothing is logged, and no traffic is served. Set `entrypointHost`. See [Publish the endpoints](./../06-publish.mdx). ## Dashboard | Symptom | Cause | |---|---| | Components says it could not read infrastructure versions | The infra chart's `components.ui.*` values do not match the platform release, or a NetworkPolicy blocks the dashboard from the API server. See [Component versions](./../configure/components.mdx#verifying-what-you-are-actually-running) | | Components says data versions are unavailable | The relay or `ids-rules-api` is unreachable from the dashboard | | The online update check cannot reach `releases.gen0sec.com` | Expected when the cluster has no route out. Turn the check off | | No **Forgot your password?** link | SMTP is not configured. See [Invitations and email](./../configure/identity.mdx#invitations-and-email) | | Invitations show a link instead of being emailed | SMTP is not configured, or sending failed. Check `kubectl -n gen0sec logs deploy/g0s-ui` | ## Air gap and mirroring ### Confirming the cluster really is isolated From a pod, not from a node. ```bash kubectl -n gen0sec run netcheck --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/busybox:1.36.1-glibc -- wget -T5 -qO- https://ghcr.io/ ; echo "exit=$?" ``` A non-zero exit is the expected result. If it succeeds, the cluster still has egress and an install that appears to work may be pulling from the internet. ### An image is missing for one architecture You built the bundle with `--arch`. A single-architecture bundle fails when the kubelet tries to pull on a node of the other architecture, which on an isolated cluster is after the transfer. Rebuild without `--arch`, or with the architecture your nodes actually run. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). ## Still stuck [Collect diagnostics](./diagnostics.mdx), then send us the archive. --- # Verify the install Source: https://docs.gen0sec.com/docs/cerebellum/0.2.2/verify # Verify the install Running pods does not mean a working platform. A worker with no Service, a misconfigured credential, or a missing route all leave every pod `Ready` while no data moves. This page defines what "installed successfully" means. ## Before you start - [ ] [First sign-in](./07-first-signin.mdx) complete - [ ] An agent API key was created ## 1. Every pod is ready ```bash kubectl -n gen0sec-system get pods kubectl -n gen0sec get pods ``` Every pod reports `Running` and `n/n` in the `READY` column. On a correct install there are no exceptions: nothing is expected to sit unready. :::note[Check readiness, not phase] A pod in `CrashLoopBackOff` still reports `status.phase: Running`. A `--field-selector` on phase reports a broken install as clean, which is why the command above reads the `READY` column instead. ::: ## 2. The migration job is gone ```bash kubectl -n gen0sec get job g0s-db-migrate ``` The output is similar to this: ``` Error from server (NotFound): jobs.batch "g0s-db-migrate" not found ``` `NotFound` is the pass. Helm deletes the job the moment it succeeds. If it is still there, it failed: `kubectl -n gen0sec logs job/g0s-db-migrate`. ## 3. The relay is connected ```bash kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' ``` The output is similar to this: ```json { "status": "ready", "mode": "proxy", "upstream": "https://api.gen0sec.com", "external_relays": ["geoip", "ids-rules"] } ``` `"mode": "proxy"` confirms the relay came up. If the upstream is unreachable, relayed routes answer 502, which a polling agent reads as "ask again later" rather than "the artifact is gone". Then confirm it is actually authorized upstream, which the mode alone does not tell you: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means your Gen0Sec API key is wrong. ## 4. The platform serves an agent This is the check that matters. It makes the same requests an agent makes, from outside the cluster, over your published address. ```bash export GEN0SEC_BASE_URL=https://$ENTRYPOINT_HOST export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) $KIT/scripts/verify-deployment.sh ``` It checks four things: 1. The API refuses unauthenticated callers. 2. Configuration is served. 3. Threat intelligence and IDS rule bundles are published. 4. A downloaded bundle matches the SHA-256 its manifest advertises. It is read-only and safe to run against production. It exits non-zero and names the failing check. **Run it after every install and every upgrade.** It is the cheapest way to catch a deployment that looks healthy and serves nothing. ## Offline bundle only: confirm nothing pulls from outside ```bash kubectl -n gen0sec-system get pods -o yaml | grep -E '^\s+image:' | sort -u kubectl -n gen0sec get pods -o yaml | grep -E '^\s+image:' | sort -u ``` Every image listed is under your registry. Anything else means a values file was not passed with `-f`. :::note[Why both namespaces, and why read the YAML] The application runs in `gen0sec`; only the operators, object store and cache run in `gen0sec-system`. Checking one namespace covers well under half the images. Reading the YAML rather than a container template also catches init containers, which is where the object store's init image is. ::: ## Definition of installed successfully All five are true: 1. Every pod in both namespaces reports `Running` and `n/n` ready. 2. `kubectl -n gen0sec get job g0s-db-migrate` returns `NotFound` or appears as `Completed`. 3. `download-api` reports `"mode": "proxy"` with your upstream, and its log shows no `status=401` or `status=402` on relayed paths. 4. `verify-deployment.sh` exits zero. 5. On the offline bundle method, every image reference is under your registry. If all five hold, the install is done. If any fail, start at the [symptom index](./troubleshoot/symptoms.mdx). ## You may be interested in... - [Upgrade](./operate/upgrade.mdx), so you know how to move before you need to - [Roll back](./operate/rollback.mdx), and where the data boundary is - [Known limitations in this release](./troubleshoot/known-limitations.mdx) --- # Component versions Source: https://docs.gen0sec.com/docs/cerebellum/configure/components # Component versions :::note[Read this from your release, not from this page] Every release pins its own versions, and the authoritative list is in the install kit you downloaded. ```bash yq '.spec.images[] | [.image, .imageTag, .digest] | @tsv' $KIT/release-manifest.yaml yq '.spec.charts[] | [.chart, .version, .digest] | @tsv' $KIT/release-manifest.yaml ``` This page describes what those versions mean and which of them have surprising requirements. ::: {/* This page should be generated from release/components.yaml and release-manifest.yaml in CI, per the structure proposal section 7. Hand-maintained, it drifts within one release and is then worse than nothing, because operators trust it. */} ## Third-party components | Component | Version | Notes | |---|---|---| | Postgres operator | 2.0.1 | Zalando. **1.15.x cannot be used**: its schema rejects PostgreSQL 18 | | PostgreSQL | 18, with extensions | Includes AGE, TimescaleDB, `pg_cron`, `pg_net`, `pglogical`, `pgsodium` | | Connection pooler | 2.0.1 | Operator-launched. Services connect through it, never directly | | Logical backup | 2.0.1 | Only used when logical backups are enabled | | Kafka operator | 1.0.0 | Strimzi. v1 CRD API only | | Apache Kafka | 4.1.1 | **KRaft, no ZooKeeper.** Operator-launched | | Object store | Chart 0.10.0 | S3-compatible. Its Service has a `-svc` suffix | | Shared cache | 1.38.1 | Redis protocol. Application services degrade to in-process caching without it | | Ingress cache | Valkey | Deployed by the ingress chart and used only by it. Kept separate on purpose, so a shared-cache outage cannot take the ingress data plane with it. One replica: a restart costs a cold start, not data | | Ingress operator | 0.1.13 | `ingressClassName: synapse`. Built and versioned in its own repository | | Platform services | The release version | Resolved from the registry host, namespace and image tag | ## Mirror paths, for the offline bundle method Every mirrored image lands under `gen0sec/`. Two paths look wrong and are not negotiable. | Image | Must be at | |---|---| | Kafka broker | `/gen0sec/strimzi/kafka:1.0.0-kafka-4.1.1` | | Kafka entity operator | `/gen0sec/strimzi/operator:1.0.0` | **Why Kafka keeps an extra path segment.** The Kafka operator's image-registry setting replaces only the registry *host* and preserves the repository path. So the chart points it at `/gen0sec` rather than ``, and the upstream `strimzi/` segment survives inside our namespace. :::note[This exact thing shipped wrong once] With the namespace omitted, the chart asked for `/strimzi/operator` while the release mirrored `/gen0sec/strimzi/operator`. One path segment apart, and the entire Kafka stack failed to pull on a cluster with no fallback. Every chart is now rendered in CI and the build fails on any image reference not backed by the release manifest, so the two cannot drift again. That gate is also why the table above is trustworthy: it catches a chart asking for a tag nobody publishes, which is how the ingress operator once requested a version that was never released. ::: `bundler.sh mirror` reproduces these paths exactly. You do not compute them. ## Not everything a chart renders is mirrored The vendored Kafka chart hardcodes an image map covering four Kafka versions plus Connect and Bridge build tooling. This edition ships one Kafka version and no Connect clusters, so the rest are deliberately absent from the bundle. That is roughly 1.5 GB of images nothing ever pulls. **If you enable Kafka Exporter, Cruise Control, Kafka Bridge or Connect builds, mirror their images first.** The release does not carry them. ## Kubernetes | | | |---|---| | Supported | 1.23 or later | See [Requirements](./../02-requirements.mdx#kubernetes-versions). ## Support policy | | | |---|---| | Supported releases | The latest two minor trains | | Upgrade path | Sequential minors. **Skipping a minor is not supported.** Patches in place | | Upgrade order | Always infra, then data, then platform | | CRDs | Helm never upgrades them. Apply them before the chart. See [Upgrade](./../operate/upgrade.mdx) | | Downgrade | Not supported across a minor that ran a schema migration. Within a train, use rollback. See [Roll back](./../operate/rollback.mdx) | | Kubernetes upgrades | Independent of ours. Stay within the supported range | ## Verifying what you are actually running The dashboard shows it under **Organization settings**, **Components**: what each component is running, and the latest version known to this release. Each chart records its versions in a ConfigMap, which you can also read directly: ```bash kubectl -n gen0sec get cm g0s-installed-versions -o jsonpath='{.data.versions\.json}' kubectl -n gen0sec-system get cm g0s-infra-installed-versions -o jsonpath='{.data.versions\.json}' kubectl -n gen0sec-system get cm g0s-data-installed-versions -o jsonpath='{.data.versions\.json}' ``` Two things about that page: - **Check for updates online** is off by default and needs `org:write`. When on, the dashboard fetches the latest release's version list from `releases.gen0sec.com` at most once an hour. Nothing about your install is sent. Air-gapped installs leave it off. - Opening it creates an API key named **Components tab (internal)**, which it uses to read data versions from the relay. Deleting it is safe; it is recreated on the next visit. If the page says it could not read infrastructure versions, the infra chart's `components.ui.namespace` and `components.ui.serviceAccountName` do not match the platform release. See [Values reference](./values.mdx#gen0sec-infra). The manifest says what a release pins. This says what your cluster pulled: ```bash kubectl -n gen0sec-system get pods -o yaml | grep -E '^\s+image:' | sort -u kubectl -n gen0sec get pods -o yaml | grep -E '^\s+image:' | sort -u ``` Reading the YAML rather than a container template also catches init containers, which is where the object store's init image is. On the offline bundle method, every reference should be under your own registry. Anything else means a values file was not passed. --- # Sign-in and identity Source: https://docs.gen0sec.com/docs/cerebellum/configure/identity # Sign-in and identity ## What is available | Method | Status | Where credentials live | |---|---|---| | Email and password | **Available** | Your Postgres | | Passkeys | **Available** | Your Postgres holds the public key. The private key never leaves the user's device | | Multi-factor authentication | **Available, and required during onboarding** | Your Postgres | | Password reset by email | **Available when [SMTP is configured](#invitations-and-email)** | Your mail server sends the link | | Microsoft Entra ID | **In development, not supported at the moment ** | | **Nothing outside your cluster is involved in sign-in.** Accounts, password hashes, passkeys and sessions are all in your own database. If you configure SMTP, invitation and password-reset emails go out through your mail server. :::note[Single sign-on is not available yet] Microsoft Entra ID is in development. No generic OIDC provider, no SAML and no LDAP integration exists either. If single sign-on is a requirement for you, tell us which provider, because that shapes what ships first. ::: ## The first account Registration is invite-only and invites need an existing member, so a fresh install seeds one account with a **published password**. Signing in, completing onboarding and changing that password is a required install step, not an optional one. See [First sign-in](./../07-first-signin.mdx). ## The published address decides whether sign-in works The dashboard derives its sign-in base URL from the host you publish it on. If that does not match the address people actually type in the browser, every sign-in is rejected as an invalid origin. Two rules: 1. **Use the address people type.** Not an internal service name, not a load balancer's own hostname. 2. **If TLS terminates upstream of the ingress**, set the scheme explicitly, because the chart cannot infer it and guesses `http` while the browser is on `https`. ```bash --set externalScheme=https ``` Leave that unset when TLS terminates at the ingress. See [Publish the endpoints](./../06-publish.mdx#if-tls-terminates-upstream). ## Passkeys Users can register with a passkey instead of a password, and add, rename or remove passkeys under **Profile**. Signing in with a passkey does not ask for the authenticator-app code: the passkey's own PIN or biometric check stands in for it. Enrolling multi-factor is still required during onboarding. A passkey is bound to the host name the dashboard is published on, taken from its sign-in URL. So: - **They need HTTPS and a DNS name.** A plain-HTTP address or a bare IP cannot register one. - **Changing `entrypointHost` makes every registered passkey stop working.** Users then sign in with a password, or register a new passkey on the new address. - **A user who registered with a passkey only has no password to fall back on.** Ask them to set one under **Profile**, especially when SMTP is not configured and nobody can reset it by email. ## Invitations and email Invitations grant one or more workspaces, each with a role no higher than the inviter's own. Pending invitations are listed on the **Team** tab, where they can be revoked. A link expires after seven days, and inviting the same address again replaces the previous link. With SMTP configured, the invitation is emailed and users can reset a forgotten password. Without it, the dashboard shows the invitation link for you to copy and send yourself, and the sign-in page offers no password reset. SMTP is configured through the dashboard's environment. The chart has no dedicated keys for it, so set them under `services.ui` in your overrides file, with the credentials in a Secret you create: ```bash kubectl -n gen0sec create secret generic g0s-ui-smtp \ --from-literal=user='' --from-literal=password='' ``` ```yaml title="my-overrides.yaml" services: ui: env: HOST_EMAIL: smtp.example.com HOST_PORT: "587" HOST_ALIAS_EMAIL: no-reply@example.com HOST_APP_URL: https://cerebellum.example.com envSecrets: HOST_AUTH_USER: { name: g0s-ui-smtp, key: user } HOST_AUTH_PASS: { name: g0s-ui-smtp, key: password } ``` | Variable | Notes | |---|---| | `HOST_EMAIL` | The SMTP server's host name | | `HOST_PORT` | `587` by default, with STARTTLS. `465` switches to implicit TLS | | `HOST_AUTH_USER`, `HOST_AUTH_PASS` | **Required.** SMTP is only used when the host and both credentials are set. A relay without authentication is not supported | | `HOST_ALIAS_EMAIL` | **Required.** The sender address | | `HOST_APP_URL` | **Required.** The address people type, with its scheme. Links in emails are built from it, and it is not derived from `entrypointHost` | Pass the file to every later `helm upgrade` as well. The dashboard's outbound connection to your mail server is listed on [Ports and egress](./../security/egress.mdx). ## Sessions The signing secret lives in `g0s-db-ui`, created with all the other database secrets in [Install step 6](./../05-install.mdx#6-create-the-application-secrets). | Action | Effect on sessions | |---|---| | Re-running `make-db-secrets.sh` | None. The existing signing secret is preserved deliberately | | Rotating the signing secret | Every session ends and every outstanding invite link stops working. Passwords and passkeys are unaffected | | Changing `entrypointHost` | Registered passkeys stop working | | Upgrading | None. Upgrading from v0.1.x to v0.2.0 invalidates invite links issued before the upgrade | ## Turning the dashboard off entirely If your operators use the API only: ```bash --set services.ui.enabled=false ``` The agent-facing APIs are unaffected. Note that agent API keys are issued through the dashboard, so turn it off only once you have the keys you need. ## If it fails | Symptom | Cause | |---|---| | Sign-in rejected as an invalid origin | The published host does not match the address typed, or the scheme is wrong. See above | | Sessions drop after a re-run of the secret script | The signing secret changed. It should not; check whether the secret was deleted first | | Every sign-in fails after an upgrade | Check that `externalScheme` and the host survived the upgrade. `helm get values g0s -n gen0sec` | | The passkey prompt fails or offers no passkey | The address typed differs from `entrypointHost`, the page is plain HTTP, or the host changed after the passkey was registered | | Emailed links start with `undefined/` | `HOST_APP_URL` is not set | | Invitations show a link instead of being emailed | SMTP is not configured, or sending failed. Check `kubectl -n gen0sec logs deploy/g0s-ui` | --- # Reference configurations Source: https://docs.gen0sec.com/docs/cerebellum/configure/reference-configs # Reference configurations Two named configurations rather than a blank values file. Each ships in the install kit, and each is reproduced in full on this page, so you can read it before you download anything. | Configuration | For | Redundancy | Storage | |---|---|---|---| | [Single node](#single-node) | Evaluation, training, upgrade rehearsal | **None** | Smallest | | [Production](#production) | Anything you depend on | Full | Smallest production grade | ## Single node **Not for production.** Everything at one replica on one node. Losing the node loses the data. Ships as three overlays in the kit: ``` $KIT/examples/single-node/infra-values.yaml $KIT/examples/single-node/data-values.yaml $KIT/examples/single-node/platform-values.yaml ``` Layer each one **after** the production values file, so it wins. Full procedure: [Quickstart](./../quickstart.mdx). | | Production | Single node | |---|---|---| | Object store | 3 replicas, erasure coded, StatefulSet | 1 replica, standalone, **Deployment** | | Postgres | 3 instances, both poolers on | 1 instance, replica pooler off | | Kafka | 3 brokers, replication factor 3 | 1 broker, every factor 1 | | Services | 2 or 3 replicas | 1 each | | StorageClass | Yours, set explicitly | The cluster's default (empty in the overlay) | ### What the overlays set Verbatim from the v0.2.3 kit, so you can read them without unpacking it first.
infra-values.yaml — object store and cache ```yaml title="$KIT/examples/single-node/infra-values.yaml" # ============================================================================ # gen0sec-infra — SINGLE-NODE EVALUATION overlay # ============================================================================ # Layered ON TOP of values-onprem.yaml (pass both, in this order): # helm install g0s-infra charts/gen0sec-infra -n gen0sec-system \ # -f values/gen0sec-infra-values-onprem.yaml \ # -f examples/single-node/infra-values.yaml # # Shrinks the footprint to fit one worker node. NOT for production: the object # store loses erasure coding and every component becomes a single point of # failure. Use it for evaluation, training, and dry-running an upgrade. # ============================================================================ rustfs: # Distributed mode needs >= 2 nodes for erasure coding; on one node the pods # would schedule but the cluster is not fault-tolerant anyway, so run the # single-process server and be explicit about it. mode: standalone: enabled: true distributed: enabled: false replicaCount: 1 storageclass: # Empty = the cluster's default StorageClass, matching values-onprem.yaml. On # k3s that resolves to local-path, which this overlay used to hardcode — but a # literal `local-path` exists on no other distribution, so the PVCs bound # nowhere and only surfaced as pods Pending past the rollout timeout. name: "" dataStorageSize: 20Gi logStorageSize: 5Gi dragonfly: # Cache only — shrink it so the node's RAM goes to Postgres and Kafka. Keep # maxMemory under resources.limits.memory so it evicts instead of being killed. maxMemory: 256mb proactorThreads: 1 resources: requests: { cpu: 100m, memory: 128Mi } limits: { cpu: 500m, memory: 512Mi } ```
data-values.yaml — Postgres and Kafka ```yaml title="$KIT/examples/single-node/data-values.yaml" # ============================================================================ # gen0sec-data — SINGLE-NODE EVALUATION overlay # ============================================================================ # helm install g0s-data charts/gen0sec-data -n gen0sec-system \ # -f values/gen0sec-data-values-onprem.yaml \ # -f examples/single-node/data-values.yaml # # One Postgres instance, one Kafka node, replication factor 1. Any node loss is # data loss — evaluation only. # ============================================================================ postgres: numberOfInstances: 1 # With a single instance there is no replica to pool. enableReplicaConnectionPooler: false volume: size: 20Gi # Empty = the cluster's default StorageClass (see infra-values.yaml). storageClass: "" resources: requests: { cpu: 500m, memory: 1Gi } limits: { cpu: "4", memory: 6Gi } parameters: shared_buffers: 512MB effective_cache_size: 1536MB max_connections: "400" max_worker_processes: "16" max_parallel_workers: "8" timescaledb.max_background_workers: "8" kafka: replicas: 1 storage: size: 20Gi # Empty = the cluster's default StorageClass (see infra-values.yaml). class: "" # min.insync.replicas must stay < replicas, so every factor drops to 1 # together with the node count. config: offsets.topic.replication.factor: 1 transaction.state.log.replication.factor: 1 transaction.state.log.min.isr: 1 default.replication.factor: 1 min.insync.replicas: 1 log.compression.type: zstd topicDefaults: partitions: 1 replicas: 1 ```
platform-values.yaml — one replica per service ```yaml title="$KIT/examples/single-node/platform-values.yaml" # ============================================================================ # gen0sec-platform — SINGLE-NODE EVALUATION overlay # ============================================================================ # helm install g0s charts/gen0sec-platform -n gen0sec \ # -f values/gen0sec-platform-values-onprem.yaml \ # -f examples/single-node/platform-values.yaml \ # --set global.imageTag=$VERSION --timeout 20m # # One replica of everything: no HA, and a rolling update is a short outage per # service. Resource requests are unchanged — they are already small. # ============================================================================ services: auth-api: { replicaCount: 1 } ban-ledger-api: { replicaCount: 1 } config-api: { replicaCount: 1 } data-api: { replicaCount: 1 } download-api: { replicaCount: 1 } feeds-api: { replicaCount: 1 } ids-rules-api: { replicaCount: 1 } rules-validator: { replicaCount: 1 } service-graph-api: { replicaCount: 1 } signal-api: { replicaCount: 1 } telemetry-api: { replicaCount: 1 } threat-api: { replicaCount: 1 } agent-status: { replicaCount: 1 } ui: { replicaCount: 1 } ```
## Production The default. `values-onprem.yaml` in the bundle **is** the production configuration, so there is no overlay to apply: what you install in [Install](./../05-install.mdx) is this. What you set yourself: ```yaml # gen0sec-infra rustfs: storageclass: name: your-block-storage-class # gen0sec-data postgres: volume: storageClass: your-block-storage-class kafka: storage: class: your-block-storage-class ``` For requirements see [Sizing](./../02-requirements.mdx#sizing-a-production-install). ## Extending retention The shipped sizes are deliberately modest, so growing is the normal direction. Which key you grow depends on what you are retaining. See [Sizing storage for your retention](./../02-requirements.mdx#sizing-storage-for-your-retention). eg. ```yaml # gen0sec-data postgres: volume: size: 200Gi storageClass: your-block-storage-class kafka: storage: size: 200Gi class: your-block-storage-class ``` ```yaml # gen0sec-infra rustfs: storageclass: name: your-block-storage-class dataStorageSize: 100Gi ``` :::note[You can also grow later] Volume expansion grows a volume if your StorageClass has `allowVolumeExpansion: true`. Starting at the defaults and expanding when you need to is a reasonable plan, and it is the direction that works: nothing shrinks a volume. ::: ## Publishing endpoints Independent of the three above, and needed by all of them: ``` $KIT/examples/single-entrypoint/ingress-values.yaml ``` One address for the dashboard and every agent-facing API, routed by path prefix. See [Publish the endpoints](./../06-publish.mdx) for the route table and the two values you set.
The header of that file, with both customer inputs and why they exist The rest of the file is the per-service route list. It is reproduced as a table on [Publish the endpoints](./../06-publish.mdx#the-route-table) rather than twice here. ```yaml title="$KIT/examples/single-entrypoint/ingress-values.yaml (head)" # ============================================================================ # gen0sec-platform — SINGLE-ENTRYPOINT INGRESS overlay # ============================================================================ # Publishes the dashboard and the agent-facing APIs through ONE address, served # by the Synapse proxy that gen0sec-infra deploys for the `synapse` IngressClass. # # helm upgrade --install g0s charts/gen0sec-platform -n gen0sec \ # -f values/gen0sec-platform-values-onprem.yaml \ # -f examples/single-entrypoint/ingress-values.yaml # # Layered ON TOP of values-onprem.yaml, which leaves every ingress disabled. # That default is deliberate — turning this on makes the APIs reachable from # wherever the proxy Service is reachable — so this is opt-in, not the default. # # A HOST IS REQUIRED — set `entrypointHost`, either below or with # `--set entrypointHost=...`. The chart reads it and publishes every rule under # it, so it is named once and nowhere else. # # It used to be a YAML anchor aliased into each rule. That could only ever be # changed by editing this file: anchors are expanded by the YAML parser before # Helm merges `--set`, so `--set entrypointHost=...` rewrote the top-level key # while every aliased copy kept `gen0sec.internal`, and the install published # on a host nobody could reach. The chart consumes the value now, so the # documented `--set` works. # # Leaving it empty FAILS THE RENDER rather than publishing anything. Host-less # rules do not work with this controller: synapse's upstreams.yaml is keyed by # host, and the operator skips any rule with an empty host # (controllers/ingress_controller.go, "if host == \"\" || rule.HTTP == nil # { continue }"). Such a rule produces an Ingress object that its own controller # silently ignores, with no event and no log line, so the chart refuses to # render one at all. # # The dashboard needs this host to be the address people actually type: # BETTER_AUTH_URL is derived from it (services.ui.urlFromIngress), and sign-in # breaks otherwise. # # ROUTING IS BY PATH PREFIX, because one address serves every backend. The # prefixes below are the real route roots, read off each service's router -- # they are not a naming convention and do not survive being guessed at. # # One overlap to keep in mind, resolved by longest-prefix matching: # /v1/agents/events config-api serves this; data-api owns /v1/agents # If a request lands on the wrong backend, that is the first thing to check. # # /v1/ids-rules goes to download-api, NOT the in-cluster ids-rules-api: in proxy # mode IDS rules are relayed from the platform, and ids-rules-api serves a local # ET ingest that this edition does not run. # ============================================================================ # CUSTOMER-INPUT: the single address every rule below is published under. entrypointHost: "" # CUSTOMER-INPUT: the scheme users actually reach that address on. # # Only needed when TLS terminates UPSTREAM of the Ingress -- a cloud load # balancer, a CDN, a tunnel. The chart otherwise infers the scheme from the # Ingress's own `tls:` block, which is absent in that setup, so it guesses http # while the browser is on https. The dashboard hands this to Better Auth as its # base URL and every sign-in is then rejected with "Invalid origin". # # Leave it unset when TLS terminates at the Ingress (i.e. you set ingress.tls), # and the chart reads the scheme off that TLS block instead. # # A single service can still override this with # `services..ingress.externalScheme`, which wins over this value. externalScheme: https ```
## Keeping your overrides Put them in a file under version control, not in `--set` arguments. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ -f my-site/platform.yaml \ --set global.imageTag=$VERSION \ --timeout 20m ``` `helm upgrade` uses only what you pass it. Anything set with `--set` on a previous install and omitted now reverts to the chart default, silently. A file makes that impossible to forget. Keep `entrypointHost` in that file. The ingress overlay ships it empty, and the render fails without it. Pass your file after the overlay so your value wins. Two things that are awkward to keep in a file, because they are per-release or per-secret: | Keep as `--set` | Why | |---|---| | `global.imageTag` | Changes every release | | `downloadProxy.apiKeySecret.name` | Or put it in your file. Just never omit it. See [Data relay](./relay.mdx) | ## What the kit contains ```bash ls $KIT/examples/*/ ``` Everything under `examples/` is a starting point you are meant to read and edit, not a black box. Each file carries comments explaining which values are yours to set. --- # Threat intelligence and the data relay Source: https://docs.gen0sec.com/docs/cerebellum/configure/relay # Threat intelligence and the data relay The relay is one in-cluster service, `download-api`. It is the only component with a route out, and it is how your fleet gets threat intelligence. This is not a choice you make. It is a requirement. See [Known limitations](./../troubleshoot/known-limitations.mdx#no-zero-egress-deployment). ## What it relays Read-only, upstream-produced content only. | Class | Routes | Why it is not shipped | |---|---|---| | CTI indicators | `/v1/indicators/version`, `/v1/indicators/download` | Built by the upstream intelligence pipeline | | ML models | `/v1/models/*` and the per-model aliases | Trained and published by Gen0Sec | | GeoIP | `/v1/geoip/manifest`, `/v1/geoip/{file}/download` | Licensed by MaxMind and IPInfo. The IPInfo Lite bundle is the default source | | IDS rules | `/v1/ids-rules/*` | Licensed as part of the Emerging Threats ruleset | **It is a relay, not a cache.** Nothing is stored on the way through. There is no local artifact store to size, no synchronization job to supervise, and no staleness window. An agent that polls gets whatever upstream has at that moment. **The relayed URLs are identical to the direct ones**, so your agents need no configuration change. ## What it never relays | Class | Routes | Why | |---|---|---| | Workload identity | `/v1/identity/*` | Produced in your cluster, describes your workloads | | Policy edges | `/v1/policy-edges/*` | Pod IPs, namespaces, NetworkPolicy edges | | Fleet bans | `/v1/fleet-bans/*` | Your own agents' ban decisions | These three are tenant-scoped: they describe your infrastructure or your decisions. Relaying them would push that to a service you do not run. They keep the local object store path behind the local authorization check, in relay mode exactly as without it. Uploads are never relayed. The relay is read-only, so `PUT` and `POST` on the download surface are served locally or not at all. ## How authorization works Two hops, two credentials, checked in two places. | Hop | Credential | Checked by | |---|---|---| | Agent to relay | The agent's own key, issued by your dashboard | Your cluster. Relayed routes sit behind the local auth middleware and each carries the same permission it has in direct mode | | Relay to Gen0Sec | Your Gen0Sec API key | Gen0Sec | **A caller's key is never forwarded upstream on a self-hosted deployment.** A locally-issued key cannot authenticate to gen0sec. Forwarding it would leak a local credential to a third party without making the request succeed. With no Gen0Sec API key configured, the relay sends no credential at all. ## The setting that must be right ```yaml downloadProxy: enabled: true upstream: "https://api.gen0sec.com" apiKeySecret: name: gen0sec-download-proxy # named by default key: DOWNLOAD_PROXY_API_KEY ``` ### Why the key is required Your agents' keys are issued by **your** dashboard and exist only in **your** database, so they cannot authenticate to gen0sec. The relay presents your own Gen0Sec API key instead, and that is the only credential the upstream will accept from your installation. Without it, the relay sends no credential, Gen0Sec answers 401, and no relayed artifact reaches your fleet. :::note[The chart names the secret by default, so this fails loudly now] `apiKeySecret.name` defaults to `gen0sec-download-proxy`. If that secret does not exist, the relay pod never starts: `CreateContainerConfigError`, visible immediately. That is deliberate. An empty name renders no environment variable at all, the relay starts, and the only symptom is every relayed class answering 401 while agents report empty rulesets rather than errors. A failed rollout at install time is the cheaper failure. Blank the name only if you relay through a gateway that supplies its own credential. ::: ### Getting the key right ```bash printf '%s' '' > ~/.gen0sec/relay-key kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-file=DOWNLOAD_PROXY_API_KEY=$HOME/.gen0sec/relay-key ``` The value renders as a `secretKeyRef`, so it never appears in a manifest or in a values file. ## All settings | Key | Default | Meaning | |---|---|---| | `downloadProxy.enabled` | `true` | Turning it off is not supported | | `downloadProxy.upstream` | `https://api.gen0sec.com` | The upstream base URL. A malformed value **fails startup** rather than degrading | | `downloadProxy.external` | `geoip,ids-rules` | Which external classes to relay. An unknown name fails startup | | `downloadProxy.timeout` | `60s` | Upstream request timeout, matching the router's own | | `downloadProxy.apiKeySecret.name` | `""` | **Set this.** See above | | `downloadProxy.apiKeySecret.key` | `DOWNLOAD_PROXY_API_KEY` | The key within the secret | :::note[Failing startup is deliberate] A malformed upstream is not treated as "fall back to reading from the local object store". That would leave every agent poll erroring against a store this deployment does not populate. Failing fast is the safer wrong answer. ::: ## Verifying ```bash kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' ``` The output is similar to this: ```json { "status": "ready", "mode": "proxy", "upstream": "https://api.gen0sec.com", "external_relays": ["geoip", "ids-rules"] } ``` **That tells you the relay started, not that it is authorized.** For that, read the log: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means your Gen0Sec API key is missing or wrong. ## Response codes worth knowing | Code | Means | What an agent does | |---|---|---| | 200 | Relayed successfully | Uses the artifact | | **401** | The upstream rejected the credential | **Reports an empty ruleset, not an error.** This is the silent failure | | **502** | The upstream is unreachable | Reads it as "ask again later" and keeps what it holds. Correct degradation | | 404 | The artifact does not exist upstream | Treats it as withdrawn | The 401 and 502 rows are the reason this page exists. ## Preview a change before applying it ```bash helm template g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set downloadProxy.apiKeySecret.name=gen0sec-download-proxy \ | grep -A3 DOWNLOAD_PROXY ``` --- # Secrets Source: https://docs.gen0sec.com/docs/cerebellum/configure/secrets # Secrets **No chart contains a credential.** Every one is referenced by name. This page is explains what must exist, with which keys, before the platform starts. ## The secrets you create | Secret | Namespace | Keys | Created in | |---|---|---|---| | `gen0sec-registry` | **Both** | `.dockerconfigjson` | [Install step 2](./../05-install.mdx#the-image-pull-secret) | | `gen0sec-object-store` | `gen0sec-system` | `RUSTFS_ACCESS_KEY`, `RUSTFS_SECRET_KEY` | [Install step 2](./../05-install.mdx#the-object-store-credentials) | | `gen0sec-download-proxy` | `gen0sec` | `DOWNLOAD_PROXY_API_KEY` | [Install step 2](./../05-install.mdx#your-gen0sec-api-key) | The pull secret is needed in **both** namespaces. The application runs in `gen0sec`; the operators, object store and cache run in `gen0sec-system`. ## The twenty the script creates `make-db-secrets.sh` composes one secret per service from the passwords the Postgres operator generated. | Secret | Contains | Role it uses | Endpoint | |---|---|---|---| | `g0s-db-` | `DATABASE_URL`, plus object store keys for services that use it | Its own per-service role, read-only where it only reads | The connection pooler | | `gen0sec-db-migrate` | `DATABASE_URL` | `postgres`, superuser | The primary, unpooled | | `g0s-db-ui` | `DATABASE_URL`, `BETTER_AUTH_SECRET`, object store keys | `frontend` | The connection pooler | One more is optional and not created by the script: SMTP credentials for invitation and password-reset email, referenced from `services.ui.envSecrets`. See [Invitations and email](./identity.mdx#invitations-and-email). A referenced Secret that does not exist leaves the dashboard in `CreateContainerConfigError`. Every service connects as its own role. The migration job needs a superuser on the primary because it runs `CREATE EXTENSION`, which an owner role cannot do. :::note[The session signing key is preserved] `g0s-db-ui` carries the dashboard's session signing secret. Re-running the script keeps the existing value, so sessions and invite links stay valid across a re-run. Nothing else regenerates it. ::: Four services need no database connection and get no `DATABASE_URL`: the GeoIP updater, the GeoIP database updater, the model publisher and the rules validator. ## Where the database passwords come from The Postgres operator creates a credentials secret per role it provisions, named after the role: ``` .core.credentials.postgresql.acid.zalan.do ``` Underscores in the role become dashes in the secret name. Compose each connection string from that role's password: ``` postgres://:@core-pooler.gen0sec-system.svc.cluster.local:5432/ax_prod?sslmode=require ``` | Consumer | Role | Endpoint | |---|---|---| | Each service | Its own per-service role | `core-pooler` | | Migration job | `postgres` | `core`, the primary, with the on-prem marker appended | | Dashboard | `frontend` | `core-pooler` | The migration job's connection string goes to the primary and carries the on-prem marker, `options=-c%20gen0sec.deployment_mode%3Donprem`: ``` postgres://postgres:@core.gen0sec-system.svc.cluster.local:5432/ax_prod?sslmode=require&options=-c%20gen0sec.deployment_mode%3Donprem ``` Migrations read the marker to tell your database from our hosted one. Without it the seeded admin account is not created, and from v0.2.1 rule changes stop reaching agents. :::note[One deliberate exception] The dashboard's connection string uses `sslmode=no-verify`. Its client library verifies the certificate chain for `sslmode=require` and rejects the pooler's self-signed certificate; the Go library every other service uses reads the same value as encrypt-without-verify. **Do not copy `no-verify` onto the other services.** Their driver rejects it outright. ::: ## For production: use your secret manager `make-db-secrets.sh` is fine for evaluation. In production, treat its output as a **specification** rather than a tool, and have your secret manager produce secrets of the same shape. Two reasons: 1. The script writes plain Kubernetes secrets from operator-generated passwords. Your manager should be the source of truth, with the cluster holding copies. 2. Rotation is then a manager operation rather than a re-run plus a rollout. An external secret can template the connection string from the operator-generated password, so nothing is copied by hand: ```yaml # shape, not a working manifest: adapt to your operator data: - secretKey: DATABASE_URL remoteRef: key: g0s-auth-api-ro.core.credentials.postgresql.acid.zalan.do property: password # then template: postgres://:{{ .password }}@core-pooler...:5432/ax_prod?sslmode=require ``` To see the exact list your install needs, read it out of the values file rather than from this page: ```bash grep -oE 'secretRef:\s*\{?\s*name:\s*[a-z0-9-]+' \ $B/values/gen0sec-platform-values-onprem.yaml | sort -u ``` Or inspect your live installation inside your cluster. ## Rotation | Credential | How | |---|---| | Database role passwords | Rotate through the Postgres operator, then restart the consumers | | Object store keys | Update **both** `gen0sec-object-store` and every `g0s-db-` that carries them, then restart the consumers | | Registry pull credentials | Recreate `gen0sec-registry` in both namespaces. Existing pods keep running; new pulls use the new value | | Your Gen0Sec API key | Recreate `gen0sec-download-proxy`, then restart the relay | | Dashboard session key | Rotating invalidates every session and every outstanding invite link. Passwords and passkeys are unaffected | | SMTP credentials | Update the Secret, then `kubectl -n gen0sec rollout restart deploy/g0s-ui` | :::note[One credential pair, surfaced two ways] The object store keys appear as `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` to the object store itself, and as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to the services that read it. Same values, two naming conventions. Rotating means updating both places. ::: ## Verifying Nothing should be missing before you install the platform: ```bash $KIT/scripts/preflight.sh ``` After install, confirm the count: ```bash kubectl -n gen0sec get secret | wc -l # ~20 service secrets plus the ones you created kubectl -n gen0sec-system get secret | grep credentials | wc -l # expect 6 ``` ## What is never in a secret | | Where it is instead | |---|---| | The registry host and namespace | The values file | | The relay upstream URL | The values file | | The cache connection string | The values file, deliberately with no credential in it | If you enable authentication on the cache, remove that line from the values file and put the connection string in each service's secret instead. --- # Values reference Source: https://docs.gen0sec.com/docs/cerebellum/configure/values # Values reference :::note[This page is the reference] It covers the keys you actually set, which is a much shorter list than every key a chart accepts, and states which of them are required. For anything not listed here, read the chart's own `values.yaml` in the bundle. There is no generated key-by-key reference. The charts carry no JSON schema yet, so nothing can be generated from one; when that changes this page will link to it. ::: ## How values are layered Three files, applied in order, last one wins. ``` chart values.yaml the chart's own defaults, registry deliberately blank values-onprem.yaml shipped in the bundle, points everything at a registry your overrides what you set ``` **Always pass `-f values-onprem.yaml`.** The default `values.yaml` leaves the registry blank on purpose, so installing without it cannot resolve images. Put your overrides in a file rather than in `--set` arguments, because `helm upgrade` uses only what you pass it. Anything set with `--set` last time and omitted this time reverts to the chart default. ```bash helm get values g0s -n gen0sec # what the running release actually has ``` ## gen0sec-infra Top-level keys: `postgres-operator`, `strimzi-kafka-operator`, `synapse-operator`, `synapse`, `rustfs`, `dragonfly`, `components`. | Key | Default | Set it when | |---|---|---| | `rustfs.storageclass.name` | `""`, meaning the cluster default | You want a specific StorageClass | | `rustfs.storageclass.dataStorageSize` | `50Gi` | Sizing the object store. **Four drives per replica** | | `rustfs.storageclass.logStorageSize` | `10Gi` | Rarely | | `rustfs.replicaCount` | `3` | Fewer nodes. Erasure coding assumes they spread | | `rustfs.clusterDomain` | `cluster.local` | Your cluster uses a custom DNS domain | | `dragonfly.extraArgs` | `[]` | Adding `--force_epoll` when `io_uring` is blocked | | `dragonfly.maxMemory` | Chart default | Keep it comfortably under the memory limit, or the kubelet kills the pod before it can evict | | `components.ui.namespace` | `gen0sec` | You install the platform chart in another namespace. Names the dashboard's service account for the Components tab | | `components.ui.serviceAccountName` | `g0s-ui` | You install the platform chart under a release name other than `g0s`. Must be `-ui` | See [Sizing storage for your retention](./../02-requirements.mdx#sizing-storage-for-your-retention). ## gen0sec-data Top-level keys: `postgres`, `kafka`, `kafka2pg`. | Key | Default | Set it when | |---|---|---| | `postgres.numberOfInstances` | `3` | Never below 3 in production | | `postgres.volume.size` | `50Gi` | Sizing the database. This is the system of record, so it is the one to grow first | | `postgres.volume.storageClass` | `""` | You want a specific StorageClass. **Fast block storage** | | `postgres.resources` | 1 vCPU / 2 GiB requested, 8 vCPU / 12 GiB limit | Tuning. The limit sets the per-node minimum | | `kafka.replicas` | `3` | Fewer nodes. **Drop the replication factors together**, or brokers refuse writes | | `kafka.storage.size` | `50Gi` | Sizing the message log and the ingest buffer | | `kafka.storage.class` | `""` | You want a specific StorageClass | :::danger[Kafka replicas and replication factor move together] `min.insync.replicas` must stay below `replicas`. Reducing brokers without reducing the factors gives you a cluster that schedules and then refuses every write. ::: ## gen0sec-platform Top-level keys: `global`, `migration`, `downloadProxy`, `services`. ### global | Key | Default | Notes | |---|---|---| | `global.registry.host` | `registry.gen0sec.com` | Rewritten by the bundler on the offline path | | `global.registry.namespace` | `gen0sec` | Every image lives under this | | `global.imageTag` | Empty | **Required.** Set it to the release version | | `global.imagePullSecrets` | `gen0sec-registry` | Must exist in both namespaces | | `global.commonEnv.S3_ENDPOINT` | The object store service URL | Must match the object store Service, which has a `-svc` suffix | | `global.commonEnv.ARXIGNIS_DATA_URL` | The same URL plus `/platform-data` | **The bucket name must match the bucket you created** | | `global.commonEnv.REDIS_URL` | The cache service URL | No credential in it, deliberately | | `global.podSecurityContext` | Non-root, uid 65532 | Do not weaken. See [Hardening](./../security/hardening.mdx) | | `global.securityContext` | Read-only root, no capabilities | Do not weaken | | `global.ingress.className` | `synapse` | The ingress controller the infra chart installs | | `global.commonEnv.CACHE_L1_TTL` | Unset, so `30s` | Default lifetime of an in-process cache entry, as a duration such as `45s`. An invalid value is ignored | ### Publishing `entrypointHost` is passed from the first install, and `externalScheme` by the [single-entrypoint overlay](./../06-publish.mdx). `values-onprem.yaml` sets neither and leaves every ingress disabled. Both are top level, and the chart applies both to every rule the overlay writes, which is why they are listed here rather than under `services`. | Key | Default | Notes | |---|---|---| | `entrypointHost` | `""` | **Required from the first install**, published or not. The one address that serves the dashboard and every agent-facing API, and the source of the dashboard's sign-in URL even before its Ingress exists. Empty fails the render and names the value to set, unless the dashboard is disabled or reached without an Ingress (see below). It must also be the address people actually type: the dashboard derives its sign-in URL from it | | `externalScheme` | Unset | Set to `https` **only when TLS terminates upstream** of the Ingress — a cloud load balancer, a CDN, a tunnel. With no `tls:` block to read, the chart otherwise infers `http` while the browser is on `https`, and every sign-in is rejected with `Invalid origin`. Leave it unset when TLS terminates at the Ingress | ### downloadProxy | Key | Default | Notes | |---|---|---| | `downloadProxy.enabled` | `true` | Turning it off is not supported | | `downloadProxy.apiKeySecret.name` | `""` | **Required. Empty is a silently broken deployment** | Full detail on [Data relay](./relay.mdx). This is the single most consequential key in the chart. ### services Twenty-one entries. Each one accepts the same shape, and you rarely need to touch any of it. | Key pattern | Notes | |---|---| | `services..enabled` | `threat-api` is not shipped at this release | | `services..replicaCount` | 2 or 3 in production, 1 in the quickstart overlay | | `services..resources` | Requests and limits. See [Sizing](./../02-requirements.mdx#sizing-a-production-install) | | `services..env` | Per-service environment, including `PORT` | | `services..ingress` | Off by default. Set by the entrypoint overlay | | `services..volumes`, `.volumeMounts` | **Do not remove these.** They exist because the read-only root filesystem needs a writable path where each service expects one | | `services.config-generator.env.DB_READ_TIMEOUT` | Unset, so `30s`. Time limit for one config build's database reads, as a duration. A build that runs out is dropped and the workspace keeps its previous config | | `services.config-generator.env.CAPABILITY_STALE_AFTER_SECONDS` | Unset, so `3600`. How long after its last report an agent stops counting toward the [connection-state check](./../troubleshoot/known-limitations.mdx#connection-state-rules-need-current-agents) | | `services.ui.enabled` | `true`. The dashboard | | `services.ui.env.BETTER_AUTH_URL` | Derived. Set it only when the dashboard is reached without an Ingress, such as through `kubectl port-forward` or your own proxy. It then wins over `entrypointHost` | | `services.ui.serviceAccount.automount` | `true`. The dashboard reads the installed-versions ConfigMaps for the Components tab. Every other service keeps `false` | | `services.ui.entraId.enabled` | `false`. Entra ID is in development and not supported at this release. See [Sign-in and identity](./identity.mdx) | :::danger[Removing a service's volume mount breaks that service] Several services default to writing into a path relative to their working directory, which fails on a read-only root filesystem. Each one has an explicit writable volume mounted where it writes. Those mounts are load-bearing, not leftovers. ::: ## Checking what a change will do Always render before you apply. ```bash helm template g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f my-overrides.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ | less ``` To diff against what is running: ```bash helm get manifest g0s -n gen0sec > running.yaml helm template g0s ... > proposed.yaml diff running.yaml proposed.yaml ``` ## Named starting points Two tested configurations rather than a blank page: [Reference configurations](./reference-configs.mdx). --- # First sign-in Source: https://docs.gen0sec.com/docs/cerebellum/first-signin ## Why this page exists Registration is invite-only, and invites can only be created by an existing member. A fresh install can be logged in by a pre-seeded admin account. ## The seeded account | | | |---|---| | Email | `admin@gen0sec.local` | | Password | `Gen0Sec!ChangeMe` | :::danger[This password is published, and it is the same on every install] It is a constant in our migration file, so anyone who has read our documentation knows it. The dashboard will not let you past it: the account is created with a must-change flag, so setting a new password is a step you cannot skip or postpone. What that does not do is protect the account between the moment you publish the address and the moment you first sign in. **Whoever signs in first is who sets the new password.** Sign in before you give the address to anyone else. If you have already shared it, sign in now and treat the interval as exposed. ::: ## 1. Sign in Open your published address and sign in with the details above. ```bash echo https://$ENTRYPOINT_HOST ``` If sign-in is rejected as an invalid origin, the published host or scheme does not match what you typed. See [Sign-in and identity](./configure/identity.mdx#the-published-address-decides-whether-sign-in-works). ## 2. Complete first-run setup The dashboard holds you at each of these in turn, and it will not let you skip ahead. A deep link to any other page returns you to the step you are on. 1. **Enroll multi-factor authentication.** Nothing else opens until you confirm a six-digit code. 2. **Set a new password.** Step 3 below. 3. **Create your organization.** 4. **Finish the remaining onboarding steps.** The wizard creates your first API key, then walks the WAF, access rule and monitoring surfaces. Multi-factor comes first, so the published password stops being enough to sign in before you have even reached the password screen. ## 3. Mandatory password change Because the password is preseeded, after setting up MFA, you'll be asked to change your password. :::note[The change signs out this account everywhere else] Every other session on the seeded account ends when the password changes. On a first install there are none, which is the point: if the address was reachable before you signed in, this is what closes that off. ::: The seed is also idempotent on the email address. Re-running the migration or the secret script neither duplicates the account nor re-flags it, and it does not reset a password you have already changed. ## 4. Invite the people who need access Invite your operators rather than sharing the seeded account. Under **Organization settings**, **Team**, pick the workspaces each person gets and a role for each. If [SMTP is configured](./configure/identity.mdx#invitations-and-email) the invitation is emailed. Otherwise the dashboard shows the link: copy it and send it yourself. Links expire after seven days. Invitees register with a passkey or a password, and enroll multi-factor like you did. ## 5. Mint an agent API key Create an agent API key in the dashboard, then save it where the verification step reads it from: ```bash mkdir -p ~/.gen0sec printf '%s' 'yor-agent-key' > ~/.gen0sec/agent-key ``` ## If it fails | Symptom | Cause | |---|---| | The seeded account does not exist | The migration ran without the on-prem marker. Secrets were provisioned without `make-db-secrets.sh` | | Sign-in rejected as an invalid origin | The published host or scheme does not match the address you typed | | The dashboard loads but onboarding will not advance | Check `kubectl -n gen0sec logs deploy/g0s-ui` | | The password is rejected | It is case-sensitive and contains a `!`. Watch your shell if you pasted it | | The new password is rejected as too weak | 12 characters, one symbol, and three of the four character classes. The server applies the same rules as the form | | The password screen appears again on a later sign-in | The change did not complete. The flag clears only on success | | An invitation link says it is no longer valid | It was revoked, replaced by a newer invitation, already used, expired, or issued before an upgrade from v0.1.x | --- # Install Source: https://docs.gen0sec.com/docs/cerebellum/install # Install ## Before you start - [ ] [Prepare the artifacts](./04-prepare-artifacts.mdx) complete - [ ] `$KIT`, `$B`, `$REGISTRY`, `$REGISTRY_USER`, `$REGISTRY_TOKEN` are set in your shell - [ ] `$G0S_DOWNLOAD_API_KEY` is set. Step 2 writes it into a secret, and an unset variable produces an empty one that fails silently - [ ] You are logged in to the registry - [ ] The address the dashboard will be reached on, in `$ENTRYPOINT_HOST`. Step 7 needs it, and [Publish the endpoints](./06-publish.mdx) publishes under it ```bash export ENTRYPOINT_HOST=cerebellum.example.internal ``` ## 1. Check the cluster ```bash $KIT/scripts/preflight.sh ``` This checks the prerequisites the charts assume but do not create: a reachable cluster, a StorageClass, and the namespaces and secrets created in step 2. On a fresh cluster it reports the namespaces and secrets as missing, which is expected at this point. Pass a StorageClass name to pin it instead of accepting the default: ```bash $KIT/scripts/preflight.sh gen0sec-system gen0sec my-block-storage ``` ## 2. Create the namespaces and the two bootstrap secrets ```bash for ns in gen0sec-system gen0sec; do kubectl create namespace $ns --dry-run=client -o yaml | kubectl apply -f - done ``` Written to converge, so re-running step 2 after a failure further down is safe rather than a wall of `AlreadyExists`. ### The image pull secret Both namespaces need it. The application runs in `gen0sec`; the operators, object store and cache run in `gen0sec-system`. ```bash for ns in gen0sec-system gen0sec; do kubectl -n $ns create secret docker-registry gen0sec-registry \ --docker-server=$REGISTRY \ --docker-username=$REGISTRY_USER \ --docker-password=$REGISTRY_TOKEN \ --dry-run=client -o yaml | kubectl apply -f - done ``` ### The object store credentials :::danger[Run this once, and only once] Unlike the rest of step 2 this does **not** converge. It mints new random keys every time it runs, and the object store's existing data was written with the old ones — recreating the secret on a second pass strands that data behind credentials nothing holds any more. The guard below stops before that happens. ```bash kubectl -n gen0sec-system get secret gen0sec-object-store >/dev/null 2>&1 \ && echo "already exists — skip to step 3" \ || echo "not present — run the block below" ``` ::: ```bash umask 077 S=$(mktemp -d) openssl rand -hex 16 | tr -d '\n' > "$S/RUSTFS_ACCESS_KEY" openssl rand -hex 32 | tr -d '\n' > "$S/RUSTFS_SECRET_KEY" kubectl -n gen0sec-system create secret generic gen0sec-object-store --from-file="$S" ``` **Save both files somewhere safe before you delete them.** Nothing regenerates these keys, and the data in the object store is written with them. ```bash mkdir -p ~/gen0sec-object-store-keys cp "$S"/RUSTFS_* ~/gen0sec-object-store-keys/ # or load them into your secret manager rm -rf "$S" ``` Step 5 reads them back from that directory. ### Your Gen0Sec API key The relay presents this key to Gen0Sec when it fetches threat intelligence, GeoIP, models and IDS rules. The chart expects this exact secret name, so the relay pod will not start without it. See [Credentials from gen0sec](./02-requirements.mdx#your-gen0sec-api-key). ```bash # Refuses to run rather than creating an empty secret. An empty key is accepted # by Kubernetes and by the chart, and only surfaces later as every relay fetch # returning 401 across the whole fleet — with nothing in the install log. : "${G0S_DOWNLOAD_API_KEY:?not set — see Prepare the artifacts, step 0}" kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-literal=DOWNLOAD_PROXY_API_KEY=$G0S_DOWNLOAD_API_KEY ``` ## 3. Install infra ```bash helm install g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml ``` This installs three operators (Postgres, Kafka, ingress), the object store and the shared cache. ### Wait for all five The next step creates resources these operators have to reconcile. Do not continue until every one is available. ```bash kubectl -n gen0sec-system rollout status deploy/g0s-infra-postgres-operator kubectl -n gen0sec-system rollout status deploy/strimzi-cluster-operator kubectl -n gen0sec-system rollout status deploy/g0s-infra-synapse-operator kubectl -n gen0sec-system rollout status deploy/dragonfly kubectl -n gen0sec-system rollout status \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" ``` The output for each is similar to this: ``` deployment "g0s-infra-postgres-operator" successfully rolled out ``` :::note[Why the last one resolves its own kind] The object store is a StatefulSet in the production configuration and a Deployment in the single-node configuration. Resolving the kind means the same command works in both. ::: ## 4. Install data ```bash helm install g0s-data $B/charts/gen0sec-data-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml ``` This creates the Postgres cluster, the Kafka cluster, its topics, and the database roles. ### Wait for both clusters Both take several minutes. Nothing after this works until Postgres reports `Running`. ```bash kubectl -n gen0sec-system wait --for=condition=Ready kafka/core --timeout=15m kubectl -n gen0sec-system wait \ --for=jsonpath='{.status.PostgresClusterStatus}'=Running postgresql/core --timeout=15m ``` ## 5. Create the object store bucket The object store starts empty and nothing creates the bucket for you. Create it now, before the platform install. Services that read or write the object store fail until it exists. This step is required on every install. The two datasets your cluster produces about itself, `identity` and `policy-edges`, live here and are never sent anywhere. See [Network and connectivity](./03-network.mdx#what-never-leaves-your-cluster). Run it as a throwaway pod in the cluster. The pod reads the keys straight from the `gen0sec-object-store` secret, so they never reach your shell and never touch your local AWS configuration. ```bash kubectl -n gen0sec-system run s3-init --rm -i --restart=Never \ --image=amazon/aws-cli:2.15.0 \ --overrides='{"spec":{"containers":[{"name":"s3-init","image":"amazon/aws-cli:2.15.0", "env":[{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"name":"gen0sec-object-store","key":"RUSTFS_ACCESS_KEY"}}}, {"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"name":"gen0sec-object-store","key":"RUSTFS_SECRET_KEY"}}}, {"name":"AWS_EC2_METADATA_DISABLED","value":"true"}, {"name":"AWS_REGION","value":"us-east-1"}], "args":["--endpoint-url","http://rustfs-svc.gen0sec-system.svc.cluster.local:9000","s3","mb","s3://platform-data"]}]}}' ``` Expect `make_bucket: platform-data`. `BucketAlreadyOwnedByYou` is also a pass, so this step converges: running it twice is harmless. :::note[Offline installs: mirror this one image first] `amazon/aws-cli` is **not** in the offline bundle. The bundle carries only what `release-manifest.yaml` pins, and its `SHA256SUMS` covers the whole tree — the bundler refuses a package containing anything the checksums do not list — so it cannot be added after the fact. Mirror it alongside the bundle, then point the pod at your copy: ```bash skopeo copy docker://amazon/aws-cli:2.15.0 docker://$REGISTRY/amazon/aws-cli:2.15.0 ``` Substitute `$REGISTRY/amazon/aws-cli:2.15.0` for `amazon/aws-cli:2.15.0` in both the `--image` flag and the `image` field of the overrides above. They must match: the `--image` flag sets the pod's image, and the override object sets the container's, and Kubernetes uses the latter. ::: Verify by running the same command with `"s3","ls"` in place of `"s3","mb","s3://platform-data"`. The bucket should be listed: ``` 2026-09-01 14:02:11 platform-data ``` An empty result means the bucket was not created, and the services in step 7 will fail with `500 NoSuchBucket`. Do not continue. :::note[If you changed the bucket name] `platform-data` is the value of `global.commonEnv.S3_BUCKET`. If you overrode it, use your name instead. ::: :::caution[Do not export the object-store keys into your shell] The in-cluster method above is recommended because the credentials never enter your shell. That matters most on managed Kubernetes. **On EKS, exporting them breaks `kubectl`.** The kubeconfig authenticates through an `aws eks get-token` exec plugin that reads `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` — the same two variables. Setting them to the object-store keys makes every later command in that shell fail with: ``` error: You must be logged in to the server (Unauthorized) ``` The error names kubectl, not the variables, which makes it hard to attribute. The in-cluster method avoids it entirely. If you use the workstation method below, scope the credentials to the one command with `env ...` rather than exporting them. **If the object store returns `401`,** the keys are not the ones this install generated. Keys left over from a previous install of the same cluster do this, and so does a stale copy in `~/gen0sec-object-store-keys/`. Credentials are genuinely validated: wrong keys return `InvalidAccessKeyId`, and none at all returns `403`. :::
Workstation method, for reference This needs no S3 client image in the cluster, which is why it was preferred: on the offline bundle method an in-cluster image has to be mirrored first, so the bucket comes to depend on a step that exists only to create the bucket. Use it if that trade-off suits you — but read the caution above first, because exporting the object-store keys will break `kubectl` on EKS. ### 5a. Open a port forward In a second terminal: ```bash kubectl -n gen0sec-system port-forward svc/rustfs-svc 9000:9000 ``` Leave it running. The output is similar to this: ``` Forwarding from 127.0.0.1:9000 -> 9000 ``` ### 5b. Create the bucket Back in your first terminal, use the two keys you generated in step 2. Scope them to the one command with `env` rather than exporting them, so they cannot clobber the variables your kubeconfig's exec plugin reads: ```bash env AWS_ACCESS_KEY_ID="$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ AWS_SECRET_ACCESS_KEY="$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" \ AWS_REGION=us-east-1 AWS_EC2_METADATA_DISABLED=true \ aws --endpoint-url http://localhost:9000 s3 mb s3://platform-data ``` The output is similar to this: ``` make_bucket: platform-data ``` `BucketAlreadyOwnedByYou` is also a pass. With the MinIO client instead: ```bash mc alias set g0s http://localhost:9000 \ "$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ "$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" mc mb g0s/platform-data mc ls g0s ``` ### 5c. Verify it exists ```bash env AWS_ACCESS_KEY_ID="$(cat ~/gen0sec-object-store-keys/RUSTFS_ACCESS_KEY)" \ AWS_SECRET_ACCESS_KEY="$(cat ~/gen0sec-object-store-keys/RUSTFS_SECRET_KEY)" \ AWS_REGION=us-east-1 AWS_EC2_METADATA_DISABLED=true \ aws --endpoint-url http://localhost:9000 s3 ls ``` The output is similar to this: ``` 2026-09-01 14:02:11 platform-data ``` An empty result means the bucket was not created, and the services in step 7 will fail with `500 NoSuchBucket`. Do not continue. Then stop the port forward with `Ctrl-C` in the second terminal.
## 6. Create the application secrets ```bash $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml ``` This composes around twenty secrets from the passwords the Postgres operator generated in step 4. That is why it cannot run earlier. Each service gets its own database role through the connection pooler. The migration job gets a superuser connection direct to the primary, because it runs `CREATE EXTENSION`. :::note[In production, treat this script's output as a specification] `make-db-secrets.sh` is fine for evaluation. For production, have your secret manager produce secrets of the same shape. Every name and key is listed in [Secrets](./configure/secrets.mdx). ::: ## 7. Install the platform ```bash helm install g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` **`entrypointHost` is required here**, before anything is published. The dashboard builds its sign-in URL from it, and without it the render fails with `entrypointHost is required`. Use the address people will type, the one you publish under in the next step. :::note[Nothing to set for the relay] The chart already points at `gen0sec-download-proxy` by default. You created that secret in step 2, and if it is missing the relay pod fails to start with `CreateContainerConfigError` rather than coming up half-configured. ::: Schema migrations run first, as a pre-install hook. If they fail, Helm aborts and nothing is deployed. That is deliberate: a partially migrated database is worse than no install. ### Check the migration ```bash kubectl -n gen0sec get job g0s-db-migrate ``` The output is similar to this: ``` Error from server (NotFound): jobs.batch "g0s-db-migrate" not found ``` **`NotFound` is the success case.** The job is a Helm hook with `hook-delete-policy: hook-succeeded`, so Helm deletes it the moment it passes. If the job is still there, it may have failed: ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` ### Wait for the pods ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m ``` :::note[Check readiness, not phase] A pod in `CrashLoopBackOff` still reports `status.phase: Running`. A `--field-selector` on phase reports a broken install as clean. ::: :::note[Two values that must match your cluster] Both are set in `values-onprem.yaml` and both point at the in-cluster object store. Check them if anything cannot read its data. | Key | Must be | |---|---| | `global.commonEnv.S3_ENDPOINT` | `http://rustfs-svc.gen0sec-system.svc.cluster.local:9000` | | `global.commonEnv.ARXIGNIS_DATA_URL` | The same endpoint, plus the bucket: `.../platform-data` | `ARXIGNIS_DATA_URL` is where in-cluster services read their datasets from. It points at your object store, not at Gen0Sec. **The bucket name at the end of it must match the bucket you created in step 5.** If you used a different name there, change it here too. The Service is `rustfs-svc`, with the `-svc` suffix. That suffix is the single most common cause of services that come up and then cannot read anything. ::: ## If it fails | Symptom | Start here | |---|---| | `helm install` fails with `entrypointHost is required` | `entrypointHost` was not passed in step 7. Set it and re-run | | `ImagePullBackOff` | [ImagePullBackOff](./troubleshoot/symptoms.mdx#imagepullbackoff) | | Install aborts at `g0s-db-migrate` | [Migration failures](./troubleshoot/symptoms.mdx#migration-failures) | | `postgresql/core` never reaches `Running` | [Postgres will not start](./troubleshoot/symptoms.mdx#postgres-will-not-start) | | Kafka broker crash loops | [Kafka will not start](./troubleshoot/symptoms.mdx#kafka-will-not-start) | --- # Network and connectivity Source: https://docs.gen0sec.com/docs/cerebellum/network # Network and connectivity ## Why the platform must reach Gen0Sec No platform is useful without data, neither is Gen0Sec. Because the datasets are licensed per customer, they are not in the install kit, not in the offline bundle, and not in any container image. The platform fetches them at runtime from `api.gen0sec.com`. | Dataset | Licensed by | |---|---| | CTI indicators | Upstream threat intelligence providers | | GeoIP databases | MaxMind, IPInfo | | IDS rules | Emerging Threats ruleset | | ML models | Trained and published by Gen0Sec | One in-cluster service, `download-api`, does the fetching. It relays each request upstream and streams the response straight back. It is a relay, not a cache, so there is no artifact bucket to size, no sync job to supervise and no staleness window. An agent that polls gets whatever upstream has at that moment. This also means you get the most recent data we can provide. ## One hop, not many Your agents never talk to the internet. They talk to `download-api`, and `download-api` is the only service in either namespace that needs a route out. The dashboard makes optional calls of its own, listed in the [egress list](#egress-list); none of them carries agent data. ``` agents -> download-api -> api.gen0sec.com:443 ^ | the only egress point in the deployment ``` That means one destination to allow, one to audit. It also means the relayed URLs are identical to the direct ones, so agents need no configuration change. ## What never leaves your cluster Two datasets describe your own infrastructure, and they are never relayed: | Dataset | What it contains | Where it stays | |---|---|---| | `identity` | Workload identity produced in your cluster | Your object store | | `policy-edges` | Pod IPs, namespaces, NetworkPolicy edges | Your object store | No hidden switches, no configuration options, it is not implemented. Your data is yours and yours only. Uploads are not relayed either. The relay is read-only, so `PUT` and `POST` on the download surface are served locally or not at all. ## How authorization works **Your agents' API keys never leave your cluster.** Agents authenticate to your installation with keys your own dashboard issued, held in your own database. On a correct configuration the relay does not forward those keys upstream. It presents **your own Gen0Sec API key**, the one issued with your subscription, on the second hop only. ``` agent --[your agent's key]--> relay --[your Gen0Sec API key]--> api.gen0sec.com agent key stops here ``` | | | |---|---| | What Gen0Sec sees | One credential, belonging to your installation | | What Gen0Sec never sees | Any individual agent key, or how many you have issued | | Revoking an agent | Entirely local, in your dashboard | | Revoking your installation's access | Revoke or rotate your Gen0Sec API key | :::danger[This key is required, and omitting it fails silently] With no Gen0Sec API key configured, the relay falls back to forwarding the caller's own key. That is correct for the hosted product, where the same platform issued it. Here it is always wrong: your keys exist only in your cluster, so Gen0Sec rejects every one with a 401, and **the agent reports an empty ruleset rather than an error**. The install may look healthy, but the fleet quietly stops receiving threat intelligence, GeoIP, models and IDS rules. Set up in [Install](./05-install.mdx) step 2, enabled in step 7, and checked by [Verify the install](./08-verify.mdx). ::: Routes that are not relayed, including `identity` and `policy-edges`, keep the local authorization check against your own database. ## Egress list All connections are outbound, TCP, TLS. Nothing needs an inbound rule from the internet. | Destination | Port | Needed by | Purpose | Required | |---|---|---|---|---| | `api.gen0sec.com` | 443 | Cluster pods (`download-api` only) | Threat intelligence, GeoIP, ML models, IDS rules | **Always** | | `releases.gen0sec.com` | 443 | Your workstation | Download the install kit and its signature | **Always** | | `registry.gen0sec.com` | 443 | Your workstation | `helm pull` for charts, `skopeo copy` for images | **Always** | | `registry.gen0sec.com` | 443 | Cluster nodes (kubelet) | Image pulls at install time | Registry install only | | `fulcio.sigstore.dev` | 443 | Your workstation | `cosign` certificate verification | **Always** | | `rekor.sigstore.dev` | 443 | Your workstation | `cosign` transparency log verification | **Always** | | `tuf-repo-cdn.sigstore.dev` | 443 | Your workstation | `cosign` trust root refresh | **Always** | | Your own registry | 443 or 80 | Cluster nodes, and the host that runs `bundler.sh apply` | Image pulls at install time | Offline bundle only | | `helm.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest agent version, shown on the Clients and Components pages | No. Always attempted. Blocking it only hides the latest version | | `releases.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest component versions | No. Only if an administrator turns on **Check for updates online** | | Your SMTP server | 587 or 465 | Cluster pods, the dashboard only | Invitation and password-reset email | No. Only if you [configure SMTP](./configure/identity.mdx#invitations-and-email) | {/* ASSUMPTION: this list is derived from the release workflow, the bundler and the chart values. Confirm it is complete, and confirm the Sigstore host set, which changes with cosign versions. */} :::note[The Sigstore hosts are the ones people forget] `cosign verify-blob` checks a signature against a public transparency log. Without those three destinations it fails, and it fails on the first command of the install. Allow them on the workstation, not on the cluster. ::: ### Inside the cluster No NetworkPolicy is required for the install. If you enforce a default-deny policy, the two namespaces need to reach each other: the application in `gen0sec` talks to Postgres, Kafka, the object store and the cache in `gen0sec-system`. ## What this rules out `api.gen0sec.com` on port 443 is required for the platform to serve data. An agent fleet with the platform hosted on a cluster that is permitted no outbound connections at all cannot function correctly. The offline bundle method removes every dependency on gen0sec-hosted registries: your cluster pulls images only from your own. It does not remove the runtime connection, because the mentioned datasets above are not ours to ship. If your policy allows no cluster egress, talk to us. We would rather hear it now than during your evaluation. > **Air-gapped.** Often used to mean "no connection of any kind". Under that definition Cerebellum > is not deployable. If it means "no dependency on vendor-hosted registries and one audited > outbound destination", the offline bundle method fits, and this page is the list your firewall team > needs. --- # Back up and restore Source: https://docs.gen0sec.com/docs/cerebellum/operate/backup # Back up and restore ## What to back up Not everything in the deployment is equally valuable. Backing up the wrong things is how backup jobs become expensive and untested. | Component | Back up | If you lose it | |---|---|---| | **Postgres** | **Yes, always** | Everything. Configuration, agents, policy, history, the audit log. This is the system of record | | **Object store** | **Yes** | The datasets and the artifacts your cluster produced about itself. Datasets are re-fetchable, `identity` and `policy-edges` are not | | Kafka | No | In-flight messages only. It is a transport, not a store. Producers re-send | | Shared cache | No | Nothing. It repopulates | | Secrets | **Yes** | Access to your own volumes. See the warning below | :::danger[Back up the volumes along with the secrets, or the volumes are useless] The database passwords are generated by the Postgres operator and stored as Kubernetes secrets. Data is written with those credentials. A volume backup without the matching secret backup restores data you cannot open. ::: ## Postgres The Postgres operator ships a logical backup image, already mirrored as part of the release. Enable it in the infra values rather than scripting `pg_dump` yourself. ```yaml # gen0sec-infra values postgres-operator: configLogicalBackup: logical_backup_schedule: "0 2 * * *" logical_backup_provider: s3 ``` Verify a backup ran: ```bash kubectl -n gen0sec-system get cronjob | grep logical-backup kubectl -n gen0sec-system get job | grep logical-backup ``` :::note[A backup you have not restored is not a backup] Restore into a scratch namespace once, before you need to. The [Quickstart](./../quickstart.mdx) cluster is a reasonable place to rehearse it, and rehearsing a restore is also a rehearsal of an upgrade rollback. ::: ## Object store Two prefixes matter, and they matter differently. | Prefix | Recoverable without a backup? | |---|---| | `identity/`, `policy-edges/` | **No.** Produced by your cluster about your cluster | | Dataset prefixes | Yes. They are fetched from Gen0Sec | Back up the whole bucket with any S3-compatible tool, with an in-cluster job or manually, over a port forward as in [Install step 5](./../05-install.mdx#5-create-the-object-store-bucket): ```bash kubectl -n gen0sec-system port-forward svc/rustfs-svc 9000:9000 ``` ```bash aws --endpoint-url http://localhost:9000 s3 sync s3://platform-data ./platform-data-backup ``` Restore is the same command with the arguments reversed. ## Secrets ```bash kubectl -n gen0sec-system get secret -o yaml > secrets-gen0sec-system.yaml kubectl -n gen0sec get secret -o yaml > secrets-gen0sec.yaml ``` **These files contain live credentials in base64.** Treat them exactly as you treat the database itself: encrypted at rest, access controlled, and out of any shared drive. If you run an external secret manager, you do not need this: your manager is already the source of truth and the cluster holds copies. See [Secrets](./../configure/secrets.mdx). ## Recovery point and recovery time | | Value | Determined by | |---|---|---| | **Recovery point** | Your backup interval | The logical backup schedule. Nightly gives up to 24 hours of loss | | **Recovery time** | Measure it | Database size and your storage throughput. Rehearse it and record your own figure | We do not publish an RTO as it depends entirely on your configuration specifics and the data size. Time your own rehearsal and use that number in your own runbook. ## Restoring Order matters, and it is the install order. 1. Install infra and data as on a [first install](./../05-install.mdx), so the operators and empty clusters exist. 2. Restore the secrets, before anything tries to use the volumes. 3. Restore Postgres from the logical backup. 4. Restore the object store bucket. 5. Install the platform. **The migration hook runs against restored data**, so the restored schema must match the release you are installing. Restore into the version it came from, then upgrade. 6. Run the smoke test: `$KIT/scripts/verify-deployment.sh`. :::note[Do not restore an old database into a new release] The migration chain expects to run forward from the schema version in the dump. Restoring a v0.1 backup and installing v0.3 skips the intermediate migrations. Restore v0.1, then upgrade one minor at a time. See [Upgrade](./upgrade.mdx). ::: --- # Monitor Source: https://docs.gen0sec.com/docs/cerebellum/operate/monitor # Monitor ## What the platform exposes today | Signal | Where | Notes | |---|---|---| | Health and readiness | `/status/healthz` and `/status/readyz` on each service | Three services serve only `healthz`. See [Known limitations](./../troubleshoot/known-limitations.mdx#three-services-report-readiness-from-a-weaker-probe) | | Relay mode and upstream | `/status/healthz` on the relay | Returns the mode, the upstream and the active relay classes | | Prometheus metrics | Worker port 8080 | Exposed, but no Service is created for workers, so scraping needs a direct pod target | | Kubernetes state | Pods, Deployments, custom resources | The primary signal | | Block events refused by the database | Kafka topic `block_events_dlq`, 14 days retention | Each message carries the reason in its `dlq_reason` header. Steady growth means events are being lost | | Block events dropped at ingest | telemetry-api log line `OTLP block events rejected, or stored without a field they carried` | At most once a minute per organization, with counts per reason | | Installed and latest versions | Dashboard, **Organization settings**, **Components** | See [Component versions](./../configure/components.mdx#verifying-what-you-are-actually-running) | | Who changed what | Dashboard, **Organization settings**, **Audit Log** | Owner and Admin roles only | **No ServiceMonitor, alert rules or dashboards ship with the charts at the moment.** If you run Prometheus, you build the scrape configuration. ## The four checks worth alerting on If you set up nothing else, set up these. ### 1. Pods not ready ```bash kubectl -n gen0sec get pods --no-headers \ | awk '{split($2,r,"/"); if (r[1] != r[2]) print}' ``` Any output is a problem. **Read the ready column, never a field selector on phase:** a pod in `CrashLoopBackOff` still reports `status.phase: Running`. ### 2. The database cluster is not Running ```bash kubectl -n gen0sec-system get postgresql core \ -o jsonpath='{.status.PostgresClusterStatus}' ``` Expect `Running`. Anything else and nothing downstream works. ### 3. The relay lost its upstream ```bash kubectl -n gen0sec exec deploy/g0s-download-api -- \ curl -fsS localhost:9999/status/healthz ``` Expect `"status": "ready"` and `"mode": "proxy"`. An unreachable upstream answers **502** on relayed routes rather than 404 or 500. That is deliberate: a polling agent reads 502 as "ask again later" and keeps the artifact it already holds, instead of treating it as withdrawn. So a broken upstream degrades your fleet slowly rather than immediately, which is safer and also easier to miss. Alert on it. ### 4. The platform stopped serving agents ```bash $KIT/scripts/verify-deployment.sh ``` Read-only, safe against production, exits non-zero and names the failing check. Running it on a schedule is the single most useful synthetic check available, because it is the only one that proves data actually moves. ## Signals that are noisy on purpose Do not page on these. | Signal | Why it is not an incident | |---|---| | `falling back to L1-only cache` | The shared cache is never fatal. Services run on their in-process cache. Reads get slower, nothing breaks | | A 502 from a relayed route, once | Upstream hiccup. Agents retry and keep what they have | | The migration job absent | That is success. Helm deletes it on completion | | `unknown_source` in a telemetry-api warning | An agent newer than the platform. The event is stored, without its source | ## Logs ```bash kubectl -n gen0sec logs deploy/ --tail=200 kubectl -n gen0sec-system logs deploy/g0s-infra-postgres-operator --tail=200 ``` All services log to standard output as structured JSON, so any collector that reads container stdout works without configuration. Log level is set per service through the values file. See [Values reference](./../configure/values.mdx). ## What a healthy install looks like Five statements, the same five as [Verify the install](./../08-verify.mdx#definition-of-installed-successfully). If you build a dashboard, build it to answer these: 1. Every pod in both namespaces is `Running` and fully ready. 2. `postgresql/core` reports `Running`. 3. `kafka/core` reports `Ready`. 4. The relay reports `"mode": "proxy"` with your upstream. 5. The smoke test exits zero. --- # Roll back Source: https://docs.gen0sec.com/docs/cerebellum/operate/rollback # Roll back Read the boundary section first. It decides whether a rollback is safe or whether you need a restore. ## The boundary, stated plainly **`helm rollback` reverts manifests. It does not revert your database.** That single sentence is the whole risk. Helm restores the previous Deployments, images and configuration. Any schema change the upgrade's migration hook applied is still applied. If the old application cannot read the new schema, rolling back the manifests gives you old code against a new database. | Situation | Rollback is | |---|---| | Patch upgrade within a train, no migration ran | Safe. Just do it | | Minor upgrade whose migrations only added things | Safe. Old code ignores what it does not know | | Minor upgrade whose migrations removed or renamed something the old code reads | **Not safe.** Restore from backup instead | | You are not sure which | Check the release notes. If they do not say, ask us before rolling back | :::note[v0.2.3 back to v0.2.2] Safe, and there is nothing to run. While rolled back, do not `helm upgrade` the v0.2.2 chart: its migration job does not know the v0.2.3 schema version and fails with `no migration found for version`. ::: :::note[v0.2.2 back to v0.2.1] Rolling back the platform alone is safe: v0.2.1 works against the v0.2.2 schema. **Do not run the down migrations.** They delete the CVE-2026-109618 and CVE-2026-109619 managed WAF rules and every workspace's switch for them. Rolling back brings the firewall defect back: rules with a connection-state condition again reach agents without it, and match every packet. Remove those rules first. Keep infra on v0.2.2: rolling it back restarts Postgres again and leaves arm64 nodes unable to run it. While rolled back, do not `helm upgrade` the v0.2.1 chart: its migration job does not know the v0.2.2 schema version and fails with `no migration found for version`. ::: :::note[v0.2.1 back to v0.2.0] Safe, and there is nothing to run: all 17 v0.2.1 down migrations are no-ops, and v0.2.0 works against the v0.2.1 schema. Both v0.2.1 fixes live in the database, so rolling back does not bring either defect back. While rolled back, do not `helm upgrade` the v0.2.0 chart: its migration job does not know the v0.2.1 schema version and fails with `no migration found for version`. ::: :::note[v0.2.0 back to v0.1.x] Rolling back the platform alone is safe: every v0.2.0 migration only adds, so v0.1.x runs against the v0.2.0 schema. **Do not run the down migrations.** There are 18, and they delete the audit log, registered passkeys and pending invitations. While rolled back, do not `helm upgrade` the v0.1.x chart: its migration job does not know the v0.2.0 schema version and fails with `no migration found for version`. ::: :::note[Why we prefer expand and contract] Migrations that add in one train and remove in a later one mean a one-release rollback never needs a schema change. That is the design intent, and it is what makes most rollbacks safe. It is not a guarantee for every release, which is why this page exists rather than a one-line reassurance. ::: ## Before you start - [ ] You know which revision you are going back to - [ ] You have read the release notes for the version you are leaving - [ ] The previous release's images are still in your registry **Offline bundle method:** mirroring is additive, so the old tags are still there unless you deleted them. Do not prune your registry between an upgrade and the point at which you are confident. ## 1. Find the revision ```bash helm history g0s -n gen0sec ``` The output is similar to this: ``` REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Mon Sep 1 10:14:02 2026 superseded gen0sec-platform-0.1.0 0.1.0 Install complete 2 Mon Sep 1 15:41:18 2026 deployed gen0sec-platform-0.2.0 0.2.0 Upgrade complete ``` ## 2. Roll back the platform Start here. In most cases this is the only step you need, because the application is what broke. ```bash helm rollback g0s 1 -n gen0sec ``` ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` If the smoke test passes, stop. Do not roll back infra or data unless they are the problem. ## 3. Roll back the schema, only if you must Needed when the upgrade applied a change the old code cannot work with. Check the release notes first. ```bash kubectl -n gen0sec run db-migrate-down --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/db-migrate: \ --env DATABASE_URL="$OWNER_DSN" -- down ``` Use the `db-migrate` image of the release you are **leaving**, not the one you are returning to. Only it contains the down migrations for its own schema; an older image fails with `no migration found for version `. `down` reverts one migration per run, so run it once per migration the release added. **Never run `down all`**: it reverts the entire schema. `$OWNER_DSN` is the superuser connection string from the `gen0sec-db-migrate` secret. See [Secrets](./../configure/secrets.mdx). :::danger[A down migration can lose data] Reversing a migration that dropped a column recreates the column, not its contents. If the upgrade removed data, only a restore brings it back. See [Back up and restore](./backup.mdx). The v0.2.0 down migrations drop the audit log, passkeys and pending invitations, and clear the `fleet_ban` source from stored block events. One of them also decompresses recent block events and locks that table while it runs. ::: ## 4. Roll back infra and data, only if they are the problem Same order as the upgrade, in reverse: platform, then data, then infra. ```bash helm rollback g0s-data 1 -n gen0sec-system helm rollback g0s-infra 1 -n gen0sec-system ``` :::note[CRDs do not roll back] Helm never manages CRDs after the first install, in either direction. A rolled-back operator runs against the newer CRD schema. That works when the change was additive, which is the usual case, and it is another reason to roll back the platform alone where possible. ::: ## When a restore is the right answer instead Rollback is for "the new version misbehaves". Restore is for "the data is wrong". | You want to | Use | |---|---| | Undo a bad release | Rollback | | Undo a destructive migration | Restore | | Recover from data corruption | Restore | | Return to a point in time | Restore | [Back up and restore](./backup.mdx). ## If it fails | Symptom | Cause | |---|---| | `ImagePullBackOff` after rollback | The previous version's images were pruned from your registry | | Old pods start, then error on database queries | Schema mismatch. Go to step 3, or restore | | `helm rollback` reports no such revision | `helm history` was truncated. `helm get manifest g0s --revision N` to confirm what exists | ## Next [Uninstall](./uninstall.mdx), or [Back up and restore](./backup.mdx). --- # Rotate credentials Source: https://docs.gen0sec.com/docs/cerebellum/operate/rotate # Rotate credentials Credentials rotates differently, and some of them have a consequence worth knowing before you start. | Credential | Rotating it interrupts | Reversible | |---|---|---| | [Database role passwords](#database-role-passwords) | The services that use them, briefly | Yes | | [Object store keys](#object-store-keys) | Everything that reads the object store | Yes, **if you kept the old values** | | [Registry pull credentials](#registry-pull-credentials) | Nothing running. New pulls only | Yes | | [Your Gen0Sec API key](#your-gen0sec-api-key) | Artifact delivery, briefly | Yes | | [Dashboard session signing key](#dashboard-session-signing-key) | **Every session and every invite link** | No | If you run an external secret manager, rotation happens there and the cluster picks up the new value. This page is for installs holding plain Kubernetes secrets. See [Secrets](./../configure/secrets.mdx). ## Database role passwords Rotate through the Postgres operator, not by editing the secret. The operator owns these values, and an edited secret is overwritten on its next reconciliation. After the operator has regenerated the credentials, recompose the application secrets and restart the consumers: ```bash $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy ``` The script preserves the dashboard's session signing key, so re-running it does not sign people out. ## Object store keys These are the ones you generated during install, and **nothing regenerates them**. Rotating means updating two places. ```bash umask 077 S=$(mktemp -d) openssl rand -hex 16 | tr -d '\n' > "$S/RUSTFS_ACCESS_KEY" openssl rand -hex 32 | tr -d '\n' > "$S/RUSTFS_SECRET_KEY" kubectl -n gen0sec-system delete secret gen0sec-object-store kubectl -n gen0sec-system create secret generic gen0sec-object-store --from-file="$S" ``` Then restart the object store, and recompose the application secrets, because the services receive the same values under different names: ```bash kubectl -n gen0sec-system rollout restart \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy ``` Save the new files where you kept the old ones, then remove the temporary directory. :::danger[Do not lose these] Data in the object store was written with them. A retained volume without the matching keys is data you cannot open. See [Back up and restore](./backup.mdx). ::: ## Registry pull credentials Recreate the secret in **both** namespaces. Running pods keep running; the new value applies to the next pull. ```bash umask 077 AUTH=$(printf '%s:%s' "$REG_USER" "$(cat $REG_PASS_FILE)" | base64 | tr -d '\n') printf '{"auths":{"%s":{"auth":"%s"}}}' "$REGISTRY" "$AUTH" > dockerconfig.json unset AUTH for ns in gen0sec-system gen0sec; do kubectl -n $ns delete secret gen0sec-registry kubectl -n $ns create secret generic gen0sec-registry \ --type=kubernetes.io/dockerconfigjson \ --from-file=.dockerconfigjson=dockerconfig.json done rm -f dockerconfig.json ``` Rotate before the old credential expires, not after. An expired pull secret is invisible until something reschedules. ## Your Gen0Sec API key ```bash printf '%s' '' > $RELAY_KEY_FILE kubectl -n gen0sec delete secret gen0sec-download-proxy kubectl -n gen0sec create secret generic gen0sec-download-proxy \ --from-file=DOWNLOAD_PROXY_API_KEY=$RELAY_KEY_FILE kubectl -n gen0sec rollout restart deploy/g0s-download-api ``` Confirm it worked, because a wrong key fails silently: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means the new key is wrong, or has a trailing newline. Use `printf`, not `echo`. See [Data relay](./../configure/relay.mdx). ## Dashboard session signing key :::danger[This signs everyone out] Rotating it invalidates every active session and every outstanding invite link. There is no partial rotation and no grace period. ::: Only do this if you believe the key is compromised. Delete it and let the secret script regenerate it: ```bash kubectl -n gen0sec delete secret g0s-db-ui $KIT/scripts/make-db-secrets.sh $B/values/gen0sec-platform-values-onprem.yaml kubectl -n gen0sec rollout restart deploy/g0s-ui ``` The script preserves an existing key, which is why deleting the secret first is the step that actually rotates it. ## Certificates | Certificate | Owned by | |---|---| | Ingress TLS | You. Rotate in your ingress or through cert-manager | | Postgres pooler | The Postgres operator, self-signed, rotated by it | | Relay to Gen0Sec | Public certificate on our side. Nothing to do | The platform presents no certificate of its own outside the ingress you configure. ## After any rotation ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` The smoke test is the only check that proves the platform still serves an agent rather than merely starting. --- # Uninstall Source: https://docs.gen0sec.com/docs/cerebellum/operate/uninstall # Uninstall ## What you are about to delete | | Default | `--keep-data` | `--keep-crds` | |---|---|---|---| | The three Helm releases | Removed | Removed | Removed | | Objects the operators created outside the releases | Removed | Removed | Removed | | Kafka topic finalizers | Cleared | Cleared | Cleared | | Namespaces `gen0sec` and `gen0sec-system` | **Deleted** | Kept | Deleted | | Volumes: Postgres, Kafka, object store | **Deleted** | Kept | Deleted | | Secrets, including the database passwords | **Deleted** | Kept | Deleted | | Operator CRDs | **Removed** | Removed | Kept | :::danger[The default deletes your data, and it asks you to prove you mean this cluster] Without `--keep-data` the databases and the object store go, and nothing can undo that. So that run also **asks you to type the current context name back**, and refuses to run at all with no terminal attached. `--keep-crds` does not trigger that prompt, because a schema the next install reinstalls anyway is no reason to skip asking about the databases. ::: ## Remove everything ```bash $KIT/scripts/uninstall.sh --yes ``` It prints the cluster context, the release list and the volume count, then waits for you to type the context name. Read what it prints before you type. ## Keep the data ```bash $KIT/scripts/uninstall.sh --yes --keep-data ``` Stops after clearing the topic finalizers. The namespaces stay, and so do the volumes and the secrets in them. No context confirmation, because nothing irreversible happens. :::note[Why the secrets have to stay with the volumes] A reinstall over retained volumes needs the original database passwords: the data was written with them and a fresh Postgres operator generates different ones. Keeping the volumes without the secrets leaves you with data you cannot open. ::: ## Keep the CRDs ```bash $KIT/scripts/uninstall.sh --yes --keep-crds ``` Use this when something else in the cluster runs the same operators, or when you want the next install to skip reinstalling schemas. ## How CRD removal is made safe CRDs are cluster-scoped, and **deleting one deletes every resource of that kind in every namespace**. Another Postgres or Kafka cluster run by the same operators would go with them. So the script checks first: if any of those kinds is in use outside the two namespaces it is removing, **it skips the CRDs** and tells you. `--keep-crds` skips them unconditionally. That check is why removing CRDs by default is safe. It is not a blanket delete. If you want to see what would be affected before you run anything: ```bash kubectl get postgresqls,kafkas,kafkatopics,kafkanodepools -A 2>/dev/null | grep -v 'gen0sec' ``` Any output there means the script will leave the CRDs alone. ## The Kafka finalizer, and why order matters Strimzi puts a `strimzi.io/topic-operator` finalizer on all 23 KafkaTopics. Once the operator is gone, nothing clears it, and deleting the namespace leaves it in `Terminating` indefinitely. The script uninstalls the releases first, then clears the finalizers, then deletes the namespaces. That order is the whole reason the script exists. If you already deleted the namespace and it is stuck, this is the fix: ```bash NS=gen0sec-system kubectl get kafkatopics.kafka.strimzi.io -n $NS -o name \ | xargs -r -n1 kubectl patch -n $NS --type=merge \ -p '{"metadata":{"finalizers":null}}' ``` ## Reinstalling over retained volumes Two things bite here. **Kafka.** The volumes survive on purpose, and a recreated cluster generates a new identity. Brokers then crash loop with `Invalid cluster.id`. See [Kafka will not start](./../troubleshoot/symptoms.mdx#kafka-will-not-start). **Postgres.** You need the original role passwords. If you kept the secrets, the operator adopts the existing data. If you did not, the volumes are unreadable. ## Uninstalling by hand The script does more than these commands: it also removes objects the operators created outside the releases, and it handles the CRD safety check. Use it if you can. ```bash helm uninstall g0s -n gen0sec helm uninstall g0s-data -n gen0sec-system helm uninstall g0s-infra -n gen0sec-system # clear the finalizers before touching the namespaces kubectl get kafkatopics.kafka.strimzi.io -n gen0sec-system -o name \ | xargs -r -n1 kubectl patch -n gen0sec-system --type=merge \ -p '{"metadata":{"finalizers":null}}' kubectl delete namespace gen0sec kubectl delete namespace gen0sec-system ``` Do not pass `--wait` to `helm uninstall`. The Kafka topics cannot finish deleting until the finalizers above are cleared, so waiting only burns the timeout. --- # Upgrade Source: https://docs.gen0sec.com/docs/cerebellum/operate/upgrade # Upgrade When this page is done you are running the new release, with the same data. Read [Roll back](./rollback.mdx) before you start. The rollback boundary changes what a safe upgrade looks like. ## Before you start - [ ] Read the release notes for every version between yours and the target - [ ] A current backup. See [Back up and restore](./backup.mdx) - [ ] The new release's install kit, verified - [ ] A maintenance window. The application is briefly unavailable while pods roll ## Rules that are not negotiable | Rule | Why | |---|---| | One minor at a time | Skipping a minor is not supported. Each train's migrations assume the previous train ran | | Patches in place | Within a train, upgrade directly | | Same order as install: infra, then data, then platform | Each chart depends on the previous one | | Apply CRDs by hand first | Helm never upgrades CRDs. See step 1 | | Back up first | The migration hook changes your schema, and `helm rollback` does not undo that | ## 1. Apply the new CRDs **Do this before anything else.** Helm installs a chart's CRDs on first install and ignores them from then on. An upgrade that ships a new CRD schema fails with a server-side-apply error, or worse, silently rejects fields it does not know. ```bash cd $B/charts tar xzf gen0sec-infra-*.tgz kubectl apply --server-side --force-conflicts \ -f gen0sec-infra/charts/postgres-operator/crds/ kubectl apply --server-side --force-conflicts \ -f gen0sec-infra/charts/strimzi-kafka-operator/crds/ ``` The output is similar to this: ``` customresourcedefinition.apiextensions.k8s.io/postgresqls.acid.zalan.do serverside-applied ``` :::note[This is not hypothetical] The Postgres operator 1.15 to 2.0 change adds PostgreSQL 18 to the `postgresqls` schema. Without the CRD apply, the `postgresql/core` resource is rejected outright and the database never starts. ::: ## 2. Get the new artifacts Follow [Prepare the artifacts](./../04-prepare-artifacts.mdx) for the new release. Both methods work the same as on a first install, with one addition for the offline bundle. **Offline bundle only.** Mirroring is additive: the new tags land beside the old ones. Your running install keeps working until you upgrade, and a rollback still has its images. Do not reuse the previous bundle's directory. The bundler refuses to write into an existing one, and each bundle's values files are targeted at the version they shipped with. ## 3. Upgrade infra Run every command from the **new** bundle directory, so `charts/` and `values/` are the new release's. ```bash helm upgrade g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml ``` If you installed the platform under a release name or namespace other than `g0s` in `gen0sec`, add `--set components.ui.serviceAccountName=-ui,components.ui.namespace=` here. The infra chart binds the dashboard's service account by name, and a wrong name fails silently: the Components tab then cannot show infrastructure versions. Wait for the operators to finish rolling before continuing. ```bash kubectl -n gen0sec-system rollout status deploy/g0s-infra-postgres-operator kubectl -n gen0sec-system rollout status deploy/strimzi-cluster-operator kubectl -n gen0sec-system rollout status deploy/g0s-infra-synapse-operator kubectl -n gen0sec-system rollout status deploy/dragonfly kubectl -n gen0sec-system rollout status \ "$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)" ``` :::note[v0.2.2: Postgres restarts here] v0.2.2 changes the Postgres image, so the Postgres operator rolls every database pod onto it. With one database instance, as on the quickstart, the platform is unavailable until that pod is back. Step 4 waits for it. ::: ## 4. Upgrade data ```bash helm upgrade g0s-data $B/charts/gen0sec-data-*.tgz \ -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml ``` ```bash kubectl -n gen0sec-system wait --for=condition=Ready kafka/core --timeout=15m kubectl -n gen0sec-system wait \ --for=jsonpath='{.status.PostgresClusterStatus}'=Running postgresql/core --timeout=15m ``` ## 5. Upgrade the platform The schema migration runs first, as a **pre-upgrade** hook, before any new pod rolls. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` Upgrading from v0.1.x, `entrypointHost` is required even if you never published the endpoints: the dashboard now builds its sign-in URL from it, and the render fails without it. `$KIT` is the **new** release's kit. Pass every file you layered at install time, in the same order: `helm upgrade` applies only what you give it. Leaving out the ingress overlay removes every Ingress, and leaving out `single-node/platform-values.yaml` on a quickstart install scales every service back to its production replica count. See [Publish the endpoints](./../06-publish.mdx). **If the migration fails, Helm aborts and the old application keeps running.** That is the designed behavior and it is why the hook runs before the rollout. ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` Fix the cause, then re-run the same command. See [Migration failures](./../troubleshoot/symptoms.mdx#migration-failures). :::note[v0.2.0: one migration needs a quiet moment] `20261004150000` changes a constraint on the block events table and waits at most 10 seconds for its lock. On a busy install the job fails with `canceling statement due to lock timeout` and leaves the schema dirty. Nothing was changed. Pause the block events pipeline, mark the previous version and re-run this step: ```bash kubectl -n gen0sec-system scale deploy/g0s-data-kafka2pg-block-events --replicas=0 kubectl -n gen0sec run db-migrate-force --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/db-migrate:$VERSION \ --env DATABASE_URL="$OWNER_DSN" -- force 20261004120000 ``` Scale the pipeline back to 1 once the upgrade completes. Events wait in Kafka meanwhile. ::: :::note[v0.2.1: check the on-prem marker first] The migration job's connection string must carry the on-prem marker, or v0.2.1 stops rule changes reaching agents. `make-db-secrets.sh` always adds it. If your secret manager produces `gen0sec-db-migrate`, confirm before you run this step: ```bash kubectl -n gen0sec get secret gen0sec-db-migrate -o jsonpath='{.data.DATABASE_URL}' \ | base64 -d | grep -c 'gen0sec.deployment_mode%3Donprem' ``` `1` means it is there. See [Secrets](./../configure/secrets.mdx#where-the-database-passwords-come-from). A `WARNING: ... left NOT VALID` in the migration log is not a failure. It means existing rows break a check that new writes now have to pass, and the migration continues. ::: :::note[v0.2.2: check connection-state rules first] If a workspace has a firewall rule with a connection-state condition, its config stops updating after this step until every agent reporting to it runs Synapse 0.8.0 or later with firewall rules enabled. Upgrade those agents first, or remove the rule. See [Known limitations](./../troubleshoot/known-limitations.mdx#connection-state-rules-need-current-agents). The migration rebuilds the signal event chart views from every stored signal event, so it takes longer the more of them you keep. ::: ## 6. Verify ```bash kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m $KIT/scripts/verify-deployment.sh ``` Run the smoke test after every upgrade, not just after installs. It is the only check that proves the platform still serves an agent rather than merely starting. Full checklist: [Verify the install](./../08-verify.mdx). ## If you changed values along the way `helm upgrade` uses the values files you pass, not the ones from last time. Anything you set with `--set` on the previous install must be set again, or it reverts to the chart default. Check what the running release actually has: ```bash helm get values g0s -n gen0sec ``` Keep your overrides in a file rather than in shell history. See [Reference configurations](./../configure/reference-configs.mdx). ## If it fails | Symptom | Start here | |---|---| | CRD schema error on `helm upgrade` | You skipped step 1 | | `Unsupported value` from the Postgres operator | You skipped step 1 | | Migration job fails | [Migration failures](./../troubleshoot/symptoms.mdx#migration-failures) | | Migration job fails with `lock timeout` | A busy block events table. See the note in step 5 | | Pods roll but the smoke test fails | [Roll back](./rollback.mdx) | | `ImagePullBackOff` on new tags | The new images were not mirrored. Offline bundle method, step 2 | ## Next [Roll back](./rollback.mdx). --- # Prepare the artifacts Source: https://docs.gen0sec.com/docs/cerebellum/prepare-artifacts import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; # Prepare the artifacts ## Before you start - [ ] [Requirements](./02-requirements.mdx) met, tools installed, `yq` version checked - [ ] [Network and connectivity](./03-network.mdx) configured by your network team - [ ] Your [registry token](./02-requirements.mdx#the-registry-token), created on the dashboard under **Repository Access** - [ ] Your [Gen0Sec API key with the required scopes](./02-requirements.mdx#your-gen0sec-api-key) ## 0. Set your keys Create your token files. ```bash mkdir -p ~/.gen0sec cd ~/.gen0sec/ && touch .registry-token .download-api-key ``` Then open up an editor of your choice to fill each file with their respective content. Then run ```bash export REGISTRY_TOKEN=$(cat ~/.gen0sec/.registry-token) export G0S_DOWNLOAD_API_KEY=$(cat ~/.gen0sec/.download-api-key) ``` :::tip[Manage your keys securely] By storing keys in files, you make sure they don't get etched into your shell history. If you have history turned off or you know what you're doing, you can also go with ```bash export REGISTRY_TOKEN="your-registry-token" export G0S_DOWNLOAD_API_KEY="your-gen0sec-api-key" ``` ::: ## 1. Set the release Pin the release explicitly. A pinned tag installs the same bytes in six months; a resolved one does not. You can find available release versions on the [Releases page](https://releases.gen0sec.com/cerebellum/). ```bash export TAG=v0.2.3 export VERSION=${TAG#v} ``` To always use the latest version instead: ```bash export TAG=$(curl -fsSL https://releases.gen0sec.com/api/repos/cerebellum/latest | jq -r .tag_name) export VERSION=${TAG#v} ``` :::note[Release candidates are not published here] Only stable releases are mirrored, so an `-rc.N` tag returns 404 no matter how it is spelled. The [release index](https://releases.gen0sec.com/cerebellum/) is the list of what exists. ::: ## 2. Download the install kit ```bash curl -fsSLO https://releases.gen0sec.com/cerebellum/$TAG/cerebellum-install-kit.tar.gz curl -fsSLO https://releases.gen0sec.com/cerebellum/$TAG/cerebellum-install-kit.tar.gz.cosign.bundle ``` ## 3. Verify the signature Run this before you unpack anything. ```bash cosign verify-blob \ --bundle cerebellum-install-kit.tar.gz.cosign.bundle \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/release\.yaml@refs/tags/' \ cerebellum-install-kit.tar.gz ``` The output is similar to this: ``` Verified OK ``` Anything else means stop. Do not unpack the archive. :::note[Why both certificate flags are required] `cosign` refuses to verify without an expected issuer and an expected identity, so a shortened command fails rather than passing weakly. The identity is pinned to one workflow on a tag reference. A broader pattern would accept a signature produced by any workflow in our organization. The signature covers the file's contents, so it proves the bytes are ours regardless of how they reached you. ::: ## 4. Unpack it ```bash tar -xzf cerebellum-install-kit.tar.gz export KIT=$PWD/cerebellum-install-kit-$VERSION cd $KIT ``` The kit contains the bundler, the release manifest, the helper scripts used later in this install, and the sizing overlays. It does not contain images or charts. ## 5. Log in to our registry ```bash export REGISTRY=registry.gen0sec.com export REGISTRY_USER=any printf '%s' "$REGISTRY_TOKEN" | helm registry login $REGISTRY -u "$REGISTRY_USER" --password-stdin printf '%s' "$REGISTRY_TOKEN" | skopeo login $REGISTRY -u "$REGISTRY_USER" --password-stdin ``` Both are needed. `helm` and `skopeo` keep separate credential stores, and the next step uses each of them. `cosign` is a third store again, and it needs no login of its own. The bundler lends it the credential `skopeo` holds. ## 6. Get the charts Pull the charts and their values files. Images are not pulled: your cluster pulls each one from `registry.gen0sec.com` when it needs it. :::note[Confirm your nodes can pull from us before you go further] Our registry challenges pulls that send no `User-Agent`, which on some distributions is what the kubelet's pull path does. This step succeeds either way, because it runs from your workstation. The failure appears at step 3 of [Install](./05-install.mdx), as a 403 on every image. Check it now: [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: ```bash ./bundler.sh assemble-charts release-manifest.yaml export B=$PWD/gen0sec-cerebellum-multiarch-$VERSION ``` The output ends with a per-chart digest confirmation. :::note[What "digest-verified" means here] `assemble-charts` checks two digests per chart: the registry's manifest against the digest this release pins, and the chart archive against the layer named in that manifest. What you install provably came out of the chart this release published. ::: Each chart carries its own `values-onprem.yaml`, extracted into `$B/values/`. That file is what points the install at a registry. The default `values.yaml` leaves the registry blank on purpose, so installing without `-f` will not resolve images. ### 6a. Build the bundle on your workstation ```bash ./bundler.sh create-package release-manifest.yaml ``` This pulls every image and chart the release pins by digest, verifies every signature, then packages the result as `gen0sec-cerebellum-multiarch-$VERSION.tar.gz`. It carries both `linux/amd64` and `linux/arm64`, so it installs on any node in your cluster. Signature verification happens here, on the connected side, because it needs the public transparency log. Everything downstream is protected by checksums instead. :::note[Before you use --arch to halve the transfer] `--arch amd64` or `--arch arm64` extracts one manifest out of each multi-architecture index. That changes every image digest by definition. Per-architecture signatures still verify, because each one is signed individually. **SBOM attestations do not**, because they are attached to the index, and the index is what a single-architecture bundle leaves behind. The bundler warns at the time and annotates the bundle. If your acceptance process checks attestations, stay on the default. If your nodes are mixed, or you are unsure what they run, stay on the default: a missing architecture fails when the kubelet tries to pull, on an isolated cluster, after the transfer. A single-architecture bundle also renames the tarball and the directory inside it, so substitute the name in every command that follows. ::: ### 6b. Move it across In this step you take your bundle onto an intermediary machine inside the network where your internal registry and the target cluster is located you'll install the bundle onto. After transferring the bundle, you can clean up all artifacts it has created. eg. ```bash scp gen0sec-cerebellum-multiarch-$VERSION.tar.gz :~/ # `create-package` ran inside the kit directory, so the bundle is sitting in it. Copying the kit first # would transfer tens of gigabytes twice. `cleanup` removes only what the bundler created. ./bundler.sh cleanup cd .. && scp -r cerebellum-install-kit-$VERSION :~/ # You can even remove the installation kit if you don't intend to keep it. # cd $KIT/.. && rm -rf cerebellum-install-kit.tar.gz $KIT ``` ### 6c. Load it into your registry Log in to your intermediary machine and use the bundler to load the images into your internal registry. ```bash export REGISTRY=registry.internal.example export REGISTRY_USER=your-registry-user export REGISTRY_TOKEN=$(cat path/to/your-registry-token) # Carried across because this is a different machine and Install step 2 needs # it. Unset, the secret it creates is empty and every relay fetch 401s. export G0S_DOWNLOAD_API_KEY=$(cat path/to/your-gen0sec-api-key) export KIT=~/cerebellum-install-kit-$VERSION export VERSION=$(cat $KIT/release-manifest.yaml | yq ".spec.version") tar -xzf gen0sec-cerebellum-multiarch-$VERSION.tar.gz export B=~/gen0sec-cerebellum-multiarch-$VERSION cd $B printf '%s' "$REGISTRY_TOKEN" | skopeo login $REGISTRY -u "$REGISTRY_USER" --password-stdin ./bundler.sh apply release-manifest.yaml -r $REGISTRY ``` `apply` pushes every bundled image into your registry, then rewrites the bundled values files to point at it. The charts are not pushed: you install them from `$B/charts/*.tgz`. The bundle is checked against its own `SHA256SUMS` before anything is pushed, so a corrupted transfer fails here rather than at install time. :::note[For a plain-HTTP registry] Set `MIRROR_TLS_VERIFY=false` before `apply`. This affects the push only, and it is the one place in this install where TLS verification can be disabled. Prefer a certificate from your internal CA. ::: ## Checkpoint Whichever method you used, you should now have the following: | Variable | Points at | |---|---| | `$KIT` | The unpacked install kit | | `$B` | A directory containing `charts/` and `values/` | | `$REGISTRY` | The registry your cluster will pull from | | `$REGISTRY_USER` | The username for that registry (`any` for ours) | | `$REGISTRY_TOKEN` | The password or token for that registry | | `$G0S_DOWNLOAD_API_KEY` | Your Gen0Sec API key, set in step 0. [Install](./05-install.mdx#your-gen0sec-api-key) step 2 puts it in a secret | You are logged in to `$REGISTRY`. Every page after this one assumes exactly that and nothing else, which is why nothing after this point branches. --- # Publish the endpoints Source: https://docs.gen0sec.com/docs/cerebellum/publish # Publish the endpoints The ingress controller is installed and running. However it's not configured to serve anything by default. That is deliberate, because creating those routes makes the APIs reachable from wherever the controller is reachable, so it is your decision, not ours. The data plane's LoadBalancer Service already has an external address at this point. Until you complete this page it serves nothing, because no rule names a host. :::note[Ingress and TLS are yours to configure] The platform ships the routing rules. Terminating TLS, issuing the certificate and exposing the LoadBalancer are your cluster's responsibility. For the configuration options, see the [Kubernetes ingress reference](https://docs.gen0sec.com/docs/client/kubernetes-ingress). ::: ## Before you start - [ ] [Install](./05-install.mdx) complete, all pods running - [ ] A DNS name you control, resolving to your ingress controller's address - [ ] A LoadBalancer address for the ingress data plane, or a Service type your cluster can expose - [ ] A TLS certificate for that name, or cert-manager configured ## 1. Choose the address One address serves the dashboard and every agent-facing API. Requests are routed by path prefix. Use the address you set as `entrypointHost` in [Install step 7](./05-install.mdx#7-install-the-platform). It is already in `$ENTRYPOINT_HOST` if you followed the install in this shell: ```bash export ENTRYPOINT_HOST=cerebellum.example.internal ``` **The host is required, and the chart enforces it.** A rule with no host is silently ignored by the ingress controller: the Ingress object is created, no event is logged, and nothing serves traffic. Rather than publish one, the chart refuses to render and names the value to set, so this fails at `helm upgrade` instead of in production. **Use the address people will actually type.** The dashboard derives its sign-in URL from this host, and sign-in fails if the two do not match. ## 2. Publish The kit ships the overlay. Layer it on top of the values file you have been using. ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` :::warning[Re-pass every values file you installed with] `helm upgrade` applies only the files you pass it. Anything layered at install time and omitted here reverts to the chart default. On the [Quickstart](./quickstart.mdx) you also passed `-f $KIT/examples/single-node/platform-values.yaml`, so include it in this upgrade too (before the ingress overlay), or every service scales back to its production replica count: ```bash helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \ -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-node/platform-values.yaml \ -f $KIT/examples/single-entrypoint/ingress-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` ::: :::note[Quote any --set argument containing brackets] `zsh` treats `[0]` as a glob and fails before `helm` runs. Single-quote the whole argument: `--set 'services.ui.ingress.hosts[0].host=...'`. ::: ## 3. Verify the routes exist ```bash kubectl -n gen0sec get ingress ``` The output is similar to this: ``` NAME CLASS HOSTS ADDRESS PORTS AGE g0s-platform synapse cerebellum.example.internal 10.0.0.42 80 30s ``` An empty `HOSTS` column means the host was not applied, and nothing will serve traffic. The upgrade fails before it can leave an Ingress in that state, so if you see one it was created by an earlier install. Then check from outside the cluster. The dashboard is published at `/` and returns a redirect to its sign-in page, so follow it with `-L`: ```bash curl -fsSL -o /dev/null -w '%{http_code}\n' https://$ENTRYPOINT_HOST/ ``` The output is similar to this: ``` 200 ``` :::note[There is no health endpoint on the ingress] `/status/healthz` is served per pod inside the cluster and is deliberately **not** published through the ingress. A request to it from outside falls through to the dashboard catch-all and returns `404`, so do not use it as an external check. The dashboard root above confirms reachability; the [smoke test](./08-verify.mdx#4-the-platform-serves-an-agent) confirms the platform serves agents. ::: ## Find the address to point DNS at The ingress data plane is a `LoadBalancer` Service. ```bash kubectl -n gen0sec-system get svc g0s-infra-synapse -o wide ``` Point your DNS name at the external address. If `EXTERNAL-IP` stays ``, your cluster has no LoadBalancer implementation: change the Service type to something it can expose. It cannot stay cluster-internal. ## The route table You do not configure these. The overlay sets them, read off each service's own router. They are here because they are the answer to "a request landed on the wrong backend", and because a missing route fails quietly rather than loudly. | Path prefix | Served by | |---|---| | `/` | Dashboard (catch-all, shortest prefix, so it is last resort) | | `/v1/indicators`, `/v1/models`, `/v1/geoip`, `/v1/ids-rules` | Relay | | `/v1/penalty-model`, `/v1/traffic-model`, `/v1/ja4-classifier`, `/v1/flow-classifier`, `/v1/jepa-embedder` | Relay, per-model aliases | | `/v1/identity`, `/v1/policy-edges`, `/v1/fleet-bans` | Relay, served locally | | `/v1/authcheck`, `/v1/auth` | Authentication | | `/v1/events`, `/v1/agents`, `/v1/log` | Agent ingest | | `/v1/agents/events` | Configuration push (longer prefix than `/v1/agents`, so it wins) | | `/v1/config` | Configuration | | `/v1/telemetry`, `/v1/logs` | Telemetry | | `/v1/signal`, `/v1/rules` | Signal | | `/v1/ban-ledger` | Ban ledger | | `/v1/threat` | Threat feeds | | `/v1/service-graph` | Service graph | Routing is by **longest matching prefix**, because one address serves every backend. The configuration push stream on `/v1/config` is long-lived, so the chart sets read, write and idle timeouts of 3600 seconds on that route. Do not lower them: agents fall back to polling with no push, which looks like a slow platform rather than a timeout. :::note[Why a missing route is worse than a broken one] The dashboard owns `/`, so any path absent from this table falls through to it and gets a redirect or HTML. HTTP has no way for the caller to tell that apart from a real answer, so the failures are silent: | Missing route | What the agent actually does | |---|---| | `/v1/authcheck` | Receives HTML, cannot distinguish a bad key from a routing mistake, and continues unverified | | `/v1/agents/events` | Logs `SSE connect returned HTTP 404`, then falls back to polling. Looks like a slow platform, not a routing bug | | `/v1/ban-ledger` | Reads the redirect as success. The contribution is lost with no error anywhere | This is why the overlay exists rather than a list of prefixes for you to type. ::: :::note[/v1/ids-rules goes to the relay, not to the in-cluster rules service] IDS rules are relayed from gen0sec. The in-cluster rules service builds rules from a local ingest pipeline that this edition does not run. Route this prefix there and you reach a service with no rules, and the relay is never used. ::: ### If TLS terminates upstream If a load balancer, CDN or tunnel terminates TLS in front of the ingress, the chart cannot infer the external scheme and guesses `http` while the browser is on `https`. The dashboard then builds its sign-in URL with the wrong scheme and rejects every sign-in with `Invalid origin`. Set it explicitly: ```bash --set externalScheme=https ``` Leave it unset when TLS terminates at the ingress, because then the chart reads it from that service's ingress TLS block. One service can differ from the rest with `services..ingress.externalScheme`, which wins over the install-wide value above. Only a service that derives its own external URL reads it, and the dashboard is the only one that does. ## The dashboard The dashboard is enabled by default and is published at the root of your address. Sign-in is email and password, held in your own Postgres, so nothing outside the cluster is involved. Its secret, `g0s-db-ui`, was created in [Install](./05-install.mdx) step 6 with all the others. Re-running that script keeps the existing signing secret, so sessions and invite links stay valid. To leave the dashboard out: ```bash --set services.ui.enabled=false ``` ## If it fails | Symptom | Cause | |---|---| | `helm upgrade` fails with `entrypointHost is required` | Expected, and the point: `entrypointHost` was not passed. Set it and re-run | | `kubectl get ingress` shows an empty `HOSTS` column | An Ingress left by an earlier install, created when a host-less rule could still render. Re-run the upgrade with `entrypointHost` set | | Sign-in is rejected with `Invalid origin` | The dashboard's URL does not match the address people type. Check `entrypointHost`, and `externalScheme` if TLS terminates upstream | | `curl` times out | DNS does not resolve to your ingress controller, or a firewall blocks the port | | The dashboard loads but sign-in redirects to the wrong address | The sign-in URL is derived from `entrypointHost`. It must be the address people type | | A request reaches the wrong service | Longest-prefix routing. Check the `/v1/agents` overlap above | --- # Quickstart on one node Source: https://docs.gen0sec.com/docs/cerebellum/quickstart # Quickstart on one node :::danger[Not for production] This runs every component at a single replica on a single node. There is no redundancy anywhere: **losing the node loses the data.** Postgres has no standby, Kafka has one broker at replication factor 1, and the object store has no erasure coding. Use it for evaluation, training and upgrade rehearsals. For anything you depend on, follow [Sizing a production install](./02-requirements.mdx#sizing-a-production-install), which runs on three nodes. ::: ## What you get A working instance you can sign in to, on one machine, without planning a cluster. This is not a separate procedure. It is the same install as the rest of this documentation, with one extra values file per chart. ## What you need | | | |---|---| | One node | 12 vCPU, 32 GB RAM, 200 GB disk | | Kubernetes | k3s | | StorageClass | Any default block StorageClass. k3s installs `local-path` as the default, which is enough | | Tools | The same list as [Requirements](./02-requirements.mdx#tools-on-your-workstation) | | Credentials | The same two as [Requirements](./02-requirements.mdx#credentials-from-gen0sec). The relay is required here too | ## How it differs from production The kit ships an overlay per chart that reduces everything to one replica. | | Production | This quickstart | |---|---|---| | Object store | 3 replicas, erasure coded, distributed | 1 replica, **standalone mode**, and a Deployment rather than a StatefulSet | | Postgres | 3 instances | 1 instance, no replica pooler | | Kafka | 3 brokers, replication factor 3 | 1 broker, every factor 1 | | Application services | 2 or 3 replicas each | 1 each | | Cache | Chart default | 256 MB, one thread | | Storage | Your block StorageClass, 500 GiB per node minimum | The cluster's default StorageClass, about 65 GiB of volumes | | Survives losing a node | Yes | No | :::note[The connection pooler stays on] Only the **replica** pooler is disabled, because there is no replica to pool. Every service still connects through `core-pooler`, exactly as in production, so the secrets that `make-db-secrets.sh` composes are the same shape here. ::: :::note[Two things that surprise people] The object store stops being a StatefulSet and becomes a Deployment of the same name. That is why the readiness checks in [Install](./05-install.mdx) resolve the kind instead of naming it. Volume claims from a previous multi-replica install are not reused. If you are converting an existing install, delete them first. ::: ## Run it Follow the spine exactly as written, starting at [Prepare the artifacts](./04-prepare-artifacts.mdx). Three commands change. Nothing else does. | Spine step | On this install | |---|---| | [Prepare the artifacts](./04-prepare-artifacts.mdx) | Unchanged. Either method works, and the registry method is fewer steps | | [Install](./05-install.mdx) steps 1 and 2 | Unchanged. Same namespaces, same three secrets | | [Install](./05-install.mdx) steps 3, 4 and 7 | Add the overlay. See below | | [Install](./05-install.mdx) steps 5 and 6 | Unchanged. You still create the bucket, and you still run the secret script | | [Publish the endpoints](./06-publish.mdx) | Unchanged. See the note on the address below | | [First sign-in](./07-first-signin.mdx) and [Verify the install](./08-verify.mdx) | Unchanged | Pass the overlay **after** the production values file, so it wins. ```bash helm install g0s-infra $B/charts/gen0sec-infra-*.tgz -n gen0sec-system \ -f $B/values/gen0sec-infra-values-onprem.yaml \ -f $KIT/examples/single-node/infra-values.yaml ``` ```bash helm install g0s-data $B/charts/gen0sec-data-*.tgz -n gen0sec-system \ -f $B/values/gen0sec-data-values-onprem.yaml \ -f $KIT/examples/single-node/data-values.yaml ``` ```bash helm install g0s $B/charts/gen0sec-platform-*.tgz -n gen0sec \ -f $B/values/gen0sec-platform-values-onprem.yaml \ -f $KIT/examples/single-node/platform-values.yaml \ --set entrypointHost=$ENTRYPOINT_HOST \ --set global.imageTag=$VERSION \ --timeout 20m ``` The wait gates after each one are the same as on [Install](./05-install.mdx). Read the files before you use them: each overlay carries comments explaining what it trades away. :::caution[Your cluster needs a default StorageClass] The overlays leave the storage class empty in all three places — `rustfs.storageclass.name` in `infra-values.yaml`, `postgres.volume.storageClass` and `kafka.storage.class` in `data-values.yaml` — which means "use the cluster's default", the same as the production values file. On k3s that resolves to `local-path` and there is nothing to do. If your cluster has **no** default, every PVC stays `Pending`. `helm install` still reports success, and the only symptom is the wait gate above timing out without naming a cause. Check first: ```bash kubectl get storageclass # exactly one row must be marked (default) ``` To pin a class explicitly instead of relying on the default, set those three keys, or layer a fourth `-f` file last. ::: ## The address [Publish the endpoints](./06-publish.mdx) is unchanged, including the requirement for a host, which the platform install above already needs in `$ENTRYPOINT_HOST`. On k3s the bundled LoadBalancer implementation gives the ingress data plane the node's own address, so there is nothing further to provision. For an evaluation, a name in `/etc/hosts` on the machine you browse from is enough. It has to be the name you actually type in the browser, because the dashboard builds its sign-in URL from it. :::note[About TLS terminatotion] In case your TLS termination does not happen on a higher level, configure Synapse Ingress to work with cert-manager (see the [Kubernetes ingress reference](https://docs.gen0sec.com/docs/client/kubernetes-ingress)). The shipped ingress overlay sets `externalScheme: https` and configures no certificate at the ingress. A single-node evaluation usually has none. Either provision a certificate, or serve plain HTTP: set `externalScheme` to `http` in the ingress overlay and use an `http://` base URL in the sign-in and verify steps below. ::: ## First sign-in Nothing is different here, and it is worth knowing before you get there: the dashboard makes you enroll multi-factor authentication and replace the seeded password before anything opens. See [First sign-in](./07-first-signin.mdx). ## Verify Run the same checks as a production install, from [Verify the install](./08-verify.mdx). The smoke test is the one that matters, because it proves the platform serves an agent rather than merely starting: ```bash # Save the agent key you minted at first sign-in, then read it back below. mkdir -p ~/.gen0sec printf '%s' '' > ~/.gen0sec/agent-key export GEN0SEC_BASE_URL=https://$ENTRYPOINT_HOST export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) $KIT/scripts/verify-deployment.sh ``` A single-node install that passes it is a good rehearsal for a real one. That is most of the value of building this. ## Tearing it down ```bash $KIT/scripts/uninstall.sh --yes ``` It asks you to type the cluster context name back before deleting anything, so it will not run unattended. See [Uninstall](./operate/uninstall.mdx). ## When you are ready for production [Sizing a production install](./02-requirements.mdx#sizing-a-production-install) has the node counts, per-node capacity and storage figures. There is no in-place conversion from this install to a production one: build the production cluster separately. --- # Bundler CLI Source: https://docs.gen0sec.com/docs/cerebellum/reference/bundler # Bundler CLI `bundler.sh` ships in the install kit. It builds an offline bundle on a connected host and loads it into your registry on a host that has no internet access. Its only input is the `release-manifest.yaml` inside the kit. No access to our source is needed, so you can build your own bundle from any published release. For the procedure rather than the reference, see [Prepare the artifacts](./../04-prepare-artifacts.mdx). ## Which commands need internet access This is the table an isolated site actually needs. The pipeline is split on internet access, not on environment. | Command | Internet | Talks to | |---|---|---| | `validate` | **Required** | Parses the manifest, resolves tooling, then fetches one image manifest to confirm registry access | | `assemble` | **Required** | Source registry: images and charts | | `assemble-charts` | **Required** | Source registry: charts only | | `assemble-images` | **Required** | Source registry: images only | | `verify-assembly` | **Required** | Source registry for signature artifacts, plus Sigstore trust material | | `create-package` | **Required** | Inherits `assemble` and `verify-assembly` | | `package` | None | Local `tar` | | `verify-package` | None | Local extract and checksum comparison | | `mirror` | None | Destination registry, over your local network | | `retarget` | None | Local files | | `apply` | None | Destination registry. This is `mirror` plus `retarget` | | `cleanup`, `version`, `help` | None | Local | **Split the pipeline at the tarball, not earlier.** `create-package` cannot run on an isolated host: it fails in preflight with no route to the source registry. ## Commands | Command | What it does | |---|---| | `validate` | Preflight. The manifest exists and its schema is sane, every image and chart is digest-pinned, the tooling resolves, and the source registry is readable. Fetches one image manifest, so a bad host, missing login, TLS mismatch or unpublished digest fails here rather than partway through a multi-gigabyte download | | `assemble` | Creates the bundle directory, pulls every image and chart, extracts each chart's production values file, copies in the manifest and itself. Fails if the directory exists | | `assemble-charts` | The charts half. **What the registry install method uses**, since the cluster pulls its own images | | `assemble-images` | The images half. Separable so a failed multi-hour pull can be retried without re-fetching charts | | `verify-assembly` | Keyless signature verification of every pinned image and chart, then writes `SHA256SUMS` and prints the bundle **fingerprint**. Reports every failure and refuses to continue on any | | `package` | Tars the bundle directory into a tarball **beside** it, never inside | | `verify-package` | Unpacks to a temporary directory and re-checks `SHA256SUMS` | | `create-package` | `assemble`, `verify-assembly`, `package`, `verify-package`, in order | | `mirror` | Verifies the bundle against `SHA256SUMS`, then pushes every image to your registry | | `retarget` | Rewrites the registry host in the bundle's own values files, in place, and updates the affected `SHA256SUMS` entries so the bundle stays self-consistent | | `apply` | `mirror` then `retarget`. The usual customer step | | `cleanup` | Removes every directory and archive the bundler created | | `version` | The bundler's version, stamped at release | | `help` | Usage | Every command runs `validate` first, so any of them is safe to invoke on its own. ## Options | Option | Effect | |---|---| | `--arch amd64` or `--arch arm64` | Single-architecture bundle, roughly half the size. **Loses SBOM attestations.** See below | | `-o ` | A different tarball name | | `-r ` | Destination registry for `mirror`, `retarget` and `apply` | | `--dangerously-skip-cosign-verify` | Drops signature verification and its tooling requirement. The bundle is annotated as unverified | ### The --arch trade-off A single-architecture bundle extracts one manifest out of each multi-architecture index, which changes every image digest by definition. | | Default | With `--arch` | |---|---|---| | Image signatures | Verify | **Verify.** Each per-architecture manifest is signed individually | | SBOM attestations | Verify | **Lost.** They hang off the index, which is what is left behind | | Digests match the manifest | Yes | No | | Tarball and directory name | `multiarch` | Named after the architecture, so substitute it everywhere | If your acceptance process checks attestations, or your nodes are mixed, or you are not certain what they run: take the default. A missing architecture fails when the kubelet tries to pull, on an isolated cluster, after the transfer. ## Environment variables | Variable | Purpose | |---|---| | `MIKEFARAH_YQ_PATH` | Path to the Go `yq`, when the one on `PATH` is the Python one | | `BUNDLE_TLS_VERIFY=false` | Disables TLS verification on the **pull** side | | `MIRROR_TLS_VERIFY=false` | Disables TLS verification on the **push** side, for a plain-HTTP registry | The two TLS variables are the only places in the whole install where verification can be disabled. Prefer a certificate from your internal CA. See [Hardening](./../security/hardening.mdx). ## Credentials, and the three stores Three tools, three credential stores. You log in to two of them. | Tool | Store | Login needed | |---|---|---| | `skopeo` | The containers store | Yes, both source and destination | | `helm` | `HELM_REGISTRY_CONFIG` | Yes, source only | | `cosign` | The Docker config | **No.** The bundler lends it the one `skopeo` holds | If you see the bundler say it is reusing skopeo's credential for cosign, that is the expected path and not a warning. ## Behavior worth knowing | | | |---|---| | `assemble` refuses to write into an existing directory | So a stale bundle cannot be silently mixed with a new one | | `retarget` takes no file arguments | Rewriting arbitrary paths handed in on a command line is not something an isolated install should be able to ask for | | `retarget` verifies integrity **before** editing, never after | That order is what stops a checksum update from re-blessing a bundle that arrived altered | | `retarget` is idempotent | Running `apply` twice is safe | | `mirror` runs before `retarget` inside `apply` | So a registry failure leaves your values files untouched | | Mirroring is additive | New tags land beside old ones, so a running install keeps working and a rollback still has its images | | An egress proxy is fine for the bundler | Provided the host can reach the registry and Sigstore. Unlike the relay, these are ordinary CLI tools that honour proxy variables | ## Bundle contents ``` gen0sec-cerebellum--/ ├── oci/ every image, as an OCI layout ├── charts/ installable chart archives ├── values/ each chart's production values, retargeted by `apply` ├── release-manifest.yaml the digests this bundle was built from ├── bundler.sh a copy └── SHA256SUMS over every file above ``` `assemble` extracts each chart's values file as `values/-values-onprem.yaml`. There is no bare `values-onprem.yaml` in the bundle root, which is why the install commands name the file in full. The bundle does **not** carry the install kit's `scripts/` or `examples/`. Those cross the air gap inside the kit directory. See [Install kit contents](./kit.mdx). --- # Release notes Source: https://docs.gen0sec.com/docs/cerebellum/reference/changelog # Release notes ## v0.2.3 A patch release that fixes schema migrations. No action is needed beyond the upgrade. Upgrade in place from v0.2.2 following [Upgrade](./../operate/upgrade.mdx). From v0.2.1 or earlier, upgrade straight to v0.2.3, and read the notes for every version in between first: everything v0.2.2 warns about applies. Rolling back to v0.2.2 is safe; see [Roll back](./../operate/rollback.mdx). ## v0.2.2 A patch release. Upgrade in place from v0.2.1 following [Upgrade](./../operate/upgrade.mdx), including infra: this release changes the Postgres and ingress operator images. From v0.2.0 or v0.1.x, upgrade straight to v0.2.2, and read the notes for every version in between first. It carries 4 schema migrations, which run as the pre-upgrade hook. Two add managed WAF rules, one rebuilds the signal event views, and one grants a read the config generator now needs. Rolling the platform back to v0.2.1 is safe, but it brings back the firewall defect below; see [Roll back](./../operate/rollback.mdx). :::danger[Connection-state firewall rules matched all traffic] On v0.2.1 and earlier, a firewall rule with a connection-state condition, such as `allow established,related`, reached agents **without** that condition. An allow rule then allowed every packet in its direction, ahead of the deny it was written to protect, and a drop rule dropped every packet. Nothing reported it. If you use connection-state conditions, review what your agents let through since you created those rules. ::: :::warning[Connection-state rules now need Synapse 0.8.0 or later] v0.2.2 sends the condition, but an agent that does not understand it would ignore it and fail open as above. So when a workspace has a connection-state rule, the config generator publishes its config only if every agent that reported in the last hour advertises support. Synapse 0.8.0 or later, with firewall rules enabled, does. Otherwise the workspace **keeps its previous config**, and no rule change reaches its agents until you upgrade them or remove the rule. Check before upgrading; see [Known limitations](./../troubleshoot/known-limitations.mdx#connection-state-rules-need-current-agents). ::: :::warning[The infra upgrade restarts Postgres] The new Postgres image makes the Postgres operator roll every database pod. With one database instance, as on the quickstart, the platform is unavailable while that pod restarts. ::: ### Fixed **Connection-state firewall rules.** See the warnings above. **arm64 nodes could not run Postgres or the ingress operator.** The Postgres image was published for `linux/amd64` only, and the ingress operator's `linux/arm64` image carried an x86-64 binary, so on an arm64 node one failed to pull and the other could not start. Both are now built for each architecture. **Agents could stop hearing about new configs.** config-api connected to Kafka once, at startup. If that failed, or the connection later dropped, it stopped telling connected agents a new config was ready until it was restarted. It now retries and reconnects, and after a gap asks connected agents to fetch their config again. **One slow database query could stall every config build.** The config generator builds one workspace at a time, and its database reads had no time limit. Each build now has one, [`DB_READ_TIMEOUT`](./../configure/values.mdx#services), 30 seconds by default, and a build that runs out of time is dropped and the workspace keeps its previous config. Also, when reading the IDS ruleset version failed, configs were published with no ruleset version. The last version read is used instead. **Signal events in the newer format showed no IP, name or action** in the dashboard, and were not counted in its charts. Both formats are now read. **Config builds slowed with every country and ASN access rule.** Each rule scanned the whole GeoIP database again. A workspace with 44 such rules took about 70 seconds per build, and now takes under two. Resolved ranges are kept in memory until the GeoIP database changes. **A changed `DATA_DIR` left the config generator without a GeoIP database.** It downloaded the database to `DATA_DIR` but read it from a fixed path. Nothing to do if you never changed it. ### Added - **Managed WAF rules for CVE-2026-109618 and CVE-2026-109619**, five in all. They block oversized form field names, which exhaust PHP's memory, and a `PHPSESSID` form field or a PHP open tag or serialized object in an upload filename, which reach PHP's session upload progress. They match part names and headers only, never values. **They are switched on for every workspace that has at least one managed WAF rule switched on.** Others are left alone. Switch them off per workspace like any managed rule. - **Repository credentials can carry the SynapseOS updates scope**, so an appliance can fetch its OS updates with them. It is ticked by default with the other scopes. - `DB_READ_TIMEOUT` and `CAPABILITY_STALE_AFTER_SECONDS` on the config generator. See [Values](./../configure/values.mdx#services). ### Changed **The config generator's memory limit is 1 GiB**, up from 512 MiB, for the GeoIP cache. Blocking every country and ASN peaks at about 470 MiB. If you set `services.config-generator.resources` yourself, raise its limit to match. **Component versions.** PostgreSQL 18.4 to 18.6, in Postgres image `spilo:4.1.0-18-age-2`. The ingress operator is `0.1.13`, a version rather than a commit hash. **The bundler needs cosign 2.6.3 or later.** The Postgres and ingress operator images are now signed by cosign v3, which stores signatures where older cosign cannot see them. The bundler stops with a message saying so, rather than reporting every image as failing. **Image signatures come from three repositories.** The Postgres and ingress operator images are built and signed in their own repositories. See [Supply chain](./../security/supply-chain.mdx#verify-an-image-or-a-chart). ### Security - The firewall fix above. - Next.js 16.4.1, for the advisories open against 16.3.7. The dashboard's base image is pinned to Node.js 24.22.1 by digest. - The Postgres image is signed and carries an SBOM. It was unsigned before, which the bundler reported as a warning. ### Documentation - [Requirements](./../02-requirements.mdx#tools-on-your-workstation) asks for cosign 2.6.3 or later. - v0.2.0 and v0.2.1 list the firewall and arm64 defects in their known limitations. ## v0.2.1 A patch release. Upgrade in place from v0.2.0 following [Upgrade](./../operate/upgrade.mdx). From v0.1.x, upgrade straight to v0.2.1, and read the v0.2.0 notes below first. It carries 17 schema migrations, which run as the pre-upgrade hook. They bring our hosted and on-premises databases onto one migration chain, so most of them re-apply schema an on-premises database already has and change nothing there. What they do change on yours is listed below. None removes anything and every down migration is a no-op, so rolling back to v0.2.0 is safe; see [Roll back](./../operate/rollback.mdx). :::warning[Check the on-prem marker before upgrading] Migrations tell an on-premises database from our hosted one by a marker on the migration job's connection string. Without it, v0.2.1 points rule-change notifications at an address that exists only in our hosted deployment, and rule changes stop reaching agents. `make-db-secrets.sh` always adds it. If your secret manager produces `gen0sec-db-migrate`, confirm it is there: ```bash kubectl -n gen0sec get secret gen0sec-db-migrate -o jsonpath='{.data.DATABASE_URL}' \ | base64 -d | grep -c 'gen0sec.deployment_mode%3Donprem' ``` `1` means it is there. On `0`, add it before upgrading; see [Secrets](./../configure/secrets.mdx#where-the-database-passwords-come-from). ::: ### Fixed **Rule changes did not reach agents.** Creating, editing or deleting an access, firewall, WAF, smart firewall or IDS rule, or switching a managed rule on or off, saved the change but did not regenerate the configuration agents fetch. The fleet kept enforcing the previous rules, deleted ones included. A database trigger asks the config generator to regenerate after every rule write, and it needs elevated privileges to make that call. A migration shipped in v0.1.0 replaced the trigger without them, so the call was refused for the dashboard's and the services' database roles. The refusal was logged as a PostgreSQL warning and the rule write still succeeded, so nothing in the dashboard showed it: ``` WARNING: notify_config_generator: POST to http://g0s-config-generator... failed: permission denied for function http_post ``` The trigger runs with those privileges again. Rules saved before the upgrade are picked up by the next rule change in the same workspace, which regenerates that workspace's whole configuration. **Managed WAF rules did not work.** On an on-premises database the column linking a workspace to a managed WAF rule was named `rule_managed_id`, while the dashboard, the config generator and the database trigger all read `waf_rules_managed_id`. So: - The dashboard could not list managed WAF rules or switch them on or off. - Managed WAF rules switched on for a workspace were never sent to its agents. - Changing a managed WAF rule failed with `column s.waf_rules_managed_id does not exist`. v0.2.0 listed the last of these as fixed. That fix was correct for our hosted database, whose column already had the new name, and it is what produced this error on yours. The column is now renamed on upgrade, so every reader finds it. **Creating your organization ended on a page not found.** Right after first-run setup created the organization, the dashboard sent you to `//check-workspace`, which does not exist. You now land on the workspace overview. ### Changed **The overview no longer offers Threats.** The Threats module is not part of the self-hosted release, and its navigation item was already hidden, but the overview still showed its card and counted it. The overview now shows the six modules you have. **Some roles gain permissions.** Where a role, built-in or custom, already holds the first permission, the upgrade grants it the second: | Holds | Gains | |---|---| | `api_keys:read` | `registry_keys:read` | | `api_keys:write` | `registry_keys:write` | | `smart_firewall_rules:read` | `fleet_bans:read` | | `smart_firewall_rules:write` | `fleet_bans:write` | So anyone who could manage API keys can now manage registry tokens under **Repository Access**, and anyone who could manage smart firewall rules can now manage fleet bans. Review custom roles after the upgrade if either should stay separate. Nothing is revoked. ### Security - The dashboard's database role can no longer hard-delete firewall rules. Firewall rules are soft-deleted, so the dashboard never needed it. ### Documentation - [Secrets](./../configure/secrets.mdx#where-the-database-passwords-come-from) now spells out the on-prem marker the migration connection string needs, rather than only naming it. ## v0.2.0 A minor release. Upgrade from v0.1.0 or v0.1.1 following [Upgrade](./../operate/upgrade.mdx). It carries 18 schema migrations, which run as a pre-upgrade hook, so take a backup first. Every one only adds, so rolling the platform back to v0.1.x afterwards is safe; see [Roll back](./../operate/rollback.mdx). :::warning[Fleet bans start being enforced] On v0.1.x the fleet-ban artifact was written in a format agents could not read, so every agent loaded zero fleet bans while the publishing job reported success. From v0.2.0 they load. Agents with `platform.fleet_bans.enforce` on will start refusing traffic they did not refuse before. Review your fleet bans, or turn enforcement off, before you upgrade. ::: :::warning[Invitation links issued on v0.1.x stop working] Invitations are now tracked in the database, and links issued before the upgrade are not in it. Re-invite anyone who has not accepted yet. ::: :::warning[Check `externalScheme` in your values before upgrading] `externalScheme` was ignored on v0.1.x and is read from v0.2.0. If your values carry a top-level `externalScheme` that has never had any effect, **it takes effect on upgrade** and changes the scheme in the dashboard's sign-in URL. A scheme that does not match the address people actually type is rejected as `Invalid origin`. ```bash helm get values g0s -n gen0sec | grep externalScheme ``` Remove it or correct it before upgrading. Nothing to do if you pass `examples/single-entrypoint/ingress-values.yaml` unchanged: it ships `externalScheme: https`, which is the scheme v0.1.x effectively used anyway. ::: ### Added - **Passkeys.** Users can register and sign in with a passkey, and manage them under **Profile**. Passkeys are bound to the published host name and need HTTPS. See [Sign-in and identity](./../configure/identity.mdx#passkeys). - **Invitation and password-reset email**, through your SMTP server. Without SMTP, invitations fall back to a link you copy. See [Invitations and email](./../configure/identity.mdx#invitations-and-email). - **Role-scoped invitations.** An invitation grants specific workspaces, each with a role no higher than the inviter's. Pending invitations are listed and can be revoked, and access to one workspace can be revoked without removing the person. - **Audit log**, under **Organization settings** for the Owner and Admin roles. It records changes made in the dashboard and rule changes made through the signal API, with source IP and user agent. It is kept indefinitely; see [Known limitations](./../troubleshoot/known-limitations.mdx). - **Components**, under **Organization settings**: what each component runs, against the latest version known to the release. An online check for newer versions is off by default. See [Component versions](./../configure/components.mdx#verifying-what-you-are-actually-running). - **WAF rules accept `regex_replace()` and the `http.request.raw_path` field.** See [Known limitations](./../troubleshoot/known-limitations.mdx#new-waf-rule-functions-need-current-agents) for agent versions. - `CACHE_L1_TTL` sets the default lifetime of the in-process cache, as a duration. Default `30s`, up from a fixed `15s`. Set it through `global.commonEnv`. ### Fixed **Fleet bans never reached agents.** See the warning above. **Block events were lost in several ways.** Each of these applied to v0.1.x: - A batch of more than 20,000 records was refused whole, and OTLP senders do not retry that. The first 20,000 are now kept. - One row the database refused sent its whole insert batch, other organizations' rows included, to the dead-letter topic. Only the refused row is parked now, with the reason in a `dlq_reason` header. - A NUL byte in any field, a `source_ip` that was not an IP address, or a non-ASCII country code made the database refuse the row. These are now cleaned or dropped, and the event is kept. - A non-integer hit count stalled the dashboards' aggregates for every organization. It is now dropped, and the event kept. - An agent newer than the platform could report a source the platform did not know, and the event was refused. It is now stored without its source. `fleet_ban` is also accepted as a source. - IDS alerts were rejected as malformed block events and warned about once a minute. Records that are not block events are now skipped quietly. **Block events lacked country and ASN** whenever the GeoIP lookup missed, although the agent had sent them. The country is also stored in upper case, so `us` and `US` no longer count twice. **Drops made by the agent's kernel IP filter were not stored.** The agent reports them as `dropped_ip` records rather than block events, and those were skipped, so for an agent enforcing in the IP filter the access-rules views stayed empty. A drop the access rules caused is now stored as an access-rules block event, with its count as the hits and `bpf:ip_filter` as the rule. **Writing a managed WAF rule failed** with `column s.rule_managed_id does not exist`. **`--set entrypointHost` had no effect.** Following the [Publish the endpoints](./../06-publish.mdx) procedure verbatim published every route on the placeholder `gen0sec.internal` whatever address you passed. The overlay carried the host as a YAML anchor, and anchors are expanded when the file is parsed, before Helm merges `--set`, so the flag rewrote the top-level key while all ten rules kept the value the file shipped with. No template read the value either, so removing the anchor alone would not have helped. The failure was silent. The Ingress objects were created and looked correct, but Synapse keys its upstreams by host and serves nothing for an address no rule names. The chart now resolves the entrypoint itself and publishes every rule under it. **The dashboard's sign-in URL could be absent.** `BETTER_AUTH_URL` was derived from the first entry of a service's `ingress.hosts`. With the address named once in `entrypointHost` that list is empty, so the variable was not wrong but missing, and Better Auth rejected every sign-in with `Invalid origin`. It now follows the same entrypoint the rules do. ### Changed **An empty `entrypointHost` now fails the render** instead of publishing host-less rules. Such a rule matches every `Host` header and Synapse's operator skips it, so it could only ever be a misconfiguration that reported nothing. `helm upgrade` now stops and names the value to set. Rules that carry their own `ingress.hosts[].host` are unaffected and still need no `entrypointHost`. **`entrypointHost` is required from the first install**, before anything is published. The dashboard's sign-in URL now always comes from, in order: `services.ui.env.BETTER_AUTH_URL`, the dashboard's own `ingress.hosts[0].host`, then `entrypointHost`, whether or not its Ingress is enabled. With none of them the render fails and names `entrypointHost`. It was previously derived only once the dashboard's Ingress was enabled. [Install](./../05-install.mdx#7-install-the-platform) and [Upgrade](./../operate/upgrade.mdx#5-upgrade-the-platform) pass it. **`--set externalScheme` had no effect either.** Same defect, same shape: nothing read the top-level value, and the single-entrypoint overlay pinned the dashboard's own scheme on top of it, so the flag was inert twice over. It only ever looked correct because what it pinned was `https`, which is what a right answer produces anyway. The wrong scheme in the dashboard's base URL is rejected as `Invalid origin`, so this mattered to anyone terminating TLS upstream who needed anything else. The chart reads the top-level value now, and the overlay stops pinning the dashboard. `services..ingress.externalScheme` still overrides it for one service. A value that was inert becomes live. See the warning at the top before you upgrade. **The dashboard holds one namespaced permission.** The infra chart adds the Role and RoleBinding `g0s-ui-read-installed-versions` in `gen0sec-system`, allowing `get` on two named ConfigMaps, and the dashboard's service account token is now mounted. Nothing to do on the documented release names. Otherwise set `components.ui.*`; see [Upgrade](./../operate/upgrade.mdx#3-upgrade-infra) and [Privileges and RBAC](./../security/privileges.mdx). **The dashboard makes optional outbound calls**, none carrying agent data. See [Ports and egress](./../security/egress.mdx). **telemetry-api logs what it drops**, once a minute per organization, and its existing decode warnings are rate-limited the same way. See [Monitor](./../operate/monitor.mdx). **Rule changes made through the signal API are audited** in the same transaction. If the audit record cannot be written, the rule change is not applied. See [Symptoms](./../troubleshoot/symptoms.mdx#rule-changes-made-through-the-signal-api-never-appear). **The Agents item is hidden** from the dashboard navigation on self-hosted installs. **One migration waits at most 10 seconds for a lock** on the block events table. On a busy install it can fail and is safe to retry. See the note in [Upgrade](./../operate/upgrade.mdx#5-upgrade-the-platform). ### Security - gRPC 1.83.2, fixing an HTTP/2 memory exhaustion reachable in the APIs (GO-2026-6348), and OpenTelemetry 1.47.0. Other Go modules updated for advisories that were not reachable. - Next.js 16.3.7 and React 19.3.0, fixing CVE-2026-94545 as first shipped in v0.1.1. - better-auth 1.7.7, and the dashboard's dependency overrides enforced so `pnpm audit` is clean. ### Documentation - These docs are now versioned, with a version selector. v0.1.0 and v0.1.1 each keep their own version, with both defects above listed in their known limitations. - [Publish the endpoints](./../06-publish.mdx#if-tls-terminates-upstream) documents `--set externalScheme=https` again, now that it works. - [Upgrade](./../operate/upgrade.mdx) now passes the ingress overlay to the platform upgrade. Without it, the upgrade removes every Ingress. - [Roll back](./../operate/rollback.mdx) uses the `db-migrate` image of the release you are leaving. The previous image cannot run the down migrations. ## v0.1.1 A security patch for the dashboard. It fixes [CVE-2026-94545](https://nvd.nist.gov/vuln/detail/CVE-2026-94545) (medium, CVSS 4.0 score 5.3), in which Satori, the HTML-to-SVG library Next.js bundles for `next/og`, did not escape certain values in the SVG it generates. The dashboard does not render images through `next/og`, but the vulnerable copy shipped in its image and scanners flag it. The fix updates Next.js from 15.5.24 to 16.3.7, and React from 19.2.3 to 19.3.0. It upgrades in place from v0.1.0, with no migration and no values changes. Nothing else changed: the procedure is v0.1.0's, and its known limitations still apply. ## v0.1.0 First on-premises release. ### Installing Two methods, both documented on one procedure that branches once: [install from our registry](./../04-prepare-artifacts.mdx), or [create an offline bundle](./../04-prepare-artifacts.mdx) and mirror it into your own. ### Requires | | | |---|---| | Kubernetes | 1.23 or later | | Nodes | 3 for production, 16 vCPU and 32 GiB each recommended | | Storage | [sized to your retention](./../02-requirements.mdx#sizing-storage-for-your-retention) | | Cluster egress | `api.gen0sec.com:443`, required | ### Credentials you need **Two**, and the second is easy to miss. Both are self-service from the dashboard: 1. A registry token, for pulling charts and images. Created under **Repository Access**. The install kit itself is public and needs no credentials. 2. **Your Gen0Sec API key**, which the relay uses to fetch data from us. Without it the whole fleet receives empty artifacts with no error. See [Credentials from gen0sec](./../02-requirements.mdx#credentials-from-gen0sec). ### Not in this release | | | |---|---| | Zero-egress deployment | [Known limitations](./../troubleshoot/known-limitations.mdx#no-zero-egress-deployment) | | Explicit egress proxy for the relay | [Known limitations](./../troubleshoot/known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy) | | Single sign-on of any kind | Email and password only. Microsoft Entra ID is in development | | Shipped dashboards and alert rules | [Monitor](./../operate/monitor.mdx) lists what to watch meanwhile | | The threat service | Not shipped at this release | --- ## How we version | | | |---|---| | Scheme | Semantic versioning. Each minor is a release train | | Supported | The latest two minor trains | | Upgrades | Sequential minors, patches in place. **Skipping a minor is not supported** | | Release candidates | Not published to `releases.gen0sec.com`. Only stable releases are mirrored | Full policy: [Component versions](./../configure/components.mdx#support-policy). ## Finding a release ```bash curl -fsSL https://releases.gen0sec.com/api/repos/cerebellum/latest | jq -r .tag_name ``` Returns the latest full release. **Pin the tag explicitly for an install you want to reproduce**, so a rebuild in six months fetches the same bytes. ## What every release publishes | Artifact | Where | |---|---| | Install kit and its signature | `https://releases.gen0sec.com/cerebellum//` | | Images and charts | The registry, each pinned to a digest in the kit's release manifest | | Signatures and SBOM attestations | Alongside each image and chart, as OCI referrers | See [Supply chain](./../security/supply-chain.mdx). --- # Install kit contents Source: https://docs.gen0sec.com/docs/cerebellum/reference/kit # Install kit contents ## What you download Two files, published per release at `https://releases.gen0sec.com/cerebellum//`. | File | What | |---|---| | `cerebellum-install-kit.tar.gz` | The kit | | `cerebellum-install-kit.tar.gz.cosign.bundle` | Its detached signature | The release also publishes `latest-known-versions.json` there. You do not download it: the dashboard reads it only when its online update check is on. It is display data, not signed and not part of the kit. Verify before unpacking. See [Prepare the artifacts](./../04-prepare-artifacts.mdx#3-verify-the-signature). ## Layout ``` cerebellum-install-kit-/ ├── bundler.sh builds and loads the offline bundle ├── release-manifest.yaml every image and chart, pinned to a digest ├── scripts/ │ ├── preflight.sh cluster checks the charts assume but do not create │ ├── make-db-secrets.sh composes the ~20 application secrets │ ├── uninstall.sh the three releases, finalizers, namespaces │ └── verify-deployment.sh external smoke test, read-only ├── examples/ │ ├── single-node/ three overlays, one replica each │ └── single-entrypoint/ one address for the dashboard and every API ├── offline-data/ │ └── README.txt └── SHA256SUMS over every file above ``` **No images and no charts.** Those come from the registry, or from the offline bundle. ## The scripts | Script | Run it | Documented in | |---|---|---| | `preflight.sh` | Before installing | [Install step 1](./../05-install.mdx#1-check-the-cluster) | | `make-db-secrets.sh` | After the databases are ready, before the platform | [Install step 6](./../05-install.mdx#6-create-the-application-secrets) | | `verify-deployment.sh` | After every install and every upgrade | [Verify the install](./../08-verify.mdx#4-the-platform-serves-an-agent) | | `uninstall.sh` | When removing the installation | [Uninstall](./../operate/uninstall.mdx) | `verify-deployment.sh` reads two environment variables: ```bash export GEN0SEC_BASE_URL=https://cerebellum.example.internal export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) ``` ## The examples Everything under `examples/` is a starting point you are meant to read and edit. Each file carries comments marking which values are yours. | Directory | Purpose | |---|---| | `single-node/` | Three overlays that reduce every component to one node. [Quickstart](./../quickstart.mdx) | | `single-entrypoint/` | One address for the dashboard and every agent-facing API. [Publish the endpoints](./../06-publish.mdx) | ## Verifying the kit's own integrity ```bash cd $KIT && sha256sum -c SHA256SUMS ``` `SHA256SUMS` covers the kit **as delivered**. Adding files makes it no longer match, which is expected once you start using it. ## What the offline bundle method has to transfer Two things, and the order matters. | Transfer | Contains | |---|---| | The bundle tarball | Images, charts, values, the manifest, a copy of the bundler | | The kit directory | `scripts/`, `examples/`, and the manifest | The bundle carries its own copy of `bundler.sh`, so loading it needs no tooling from the kit. **The kit still has to cross**, because the install uses its scripts and its examples. Copy the bundle first, then clean up, then the kit. `create-package` runs inside the kit directory, so the bundle is sitting in it, and copying the kit first transfers tens of gigabytes twice. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). Both archives unpack side by side: ``` ~/gen0sec-cerebellum-multiarch-0.2.3/ the bundle ~/cerebellum-install-kit-0.2.3/ the kit ``` ## The offline-data directory The kit creates `offline-data/` containing a single `README.txt`, and nothing else. It is a placeholder for a data-delivery path that is not part of this release: the four datasets the platform needs are fetched at runtime through the relay, not delivered as files. See [Data relay](./../configure/relay.mdx). If you were told to populate it, check with us first. At this release you should not need to. --- # Requirements Source: https://docs.gen0sec.com/docs/cerebellum/requirements # Requirements ## Cluster | | Requirement | |---|---| | Kubernetes | 1.23 or later | | Access | `cluster-admin`, for the install only. See [Privileges and RBAC](./security/privileges.mdx) for what it is used for and what it leaves behind | | StorageClass | A default StorageClass on block storage. **Volume expansion recommended** | | Supported architecture | `linux/amd64` or `linux/arm64`. Mixed-architecture clusters are supported | | LoadBalancer | The ingress data plane is exposed as a `LoadBalancer` type Service by default. See the note below | :::note[The ingress needs to be reachable from outside the cluster] The ingress data plane is a `LoadBalancer` type Service, because nothing it programs can serve traffic if it stays cluster-internal. Your cluster needs a LoadBalancer implementation. If your cluster fronts ingress another way, a NodePort, a hardware balancer or an external proxy, change the Service type to match. ::: Cerebellum brings its own data plane - which contains Postgres, Kafka, object store and cache. You do not connect it to existing instances of any of them. :::info[Maintenance] We've deliberately chosen to use well-known solutions from the software industry in our data layer to ease the burden of maintenance. While Cerebellum ships this data layer by default, to make the setup easier, the data in it is the highest importance to you. So make sure you monitor the dataplane and do backups and maintenances regularly. ::: :::note[Container runtime, for the registry install method only] Our registry challenges pulls that carry no `User-Agent`, and on some distributions the kubelet's pull path sends none. containerd's own user agent passes; the problem is its absence. Either configure one on every node, or use the offline bundle method, which pulls into your own registry instead. Settle this before you choose a method. See [Known limitations](./troubleshoot/known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ::: ### Kubernetes versions 1.23 is the floor because `autoscaling/v2` is the newest API the charts use, and it reached GA in that release. While anything from 1.23 and up should work, we recommend you use the latest stable version. ## Sizing a production install Production runs on **3 nodes**. Several data plane members each form a three-member quorum, so three is the floor for redundancy, not a suggestion. ### CPU & memory per node | | Resources | |---|---| | **Minimum** | 8 vCPU, 16 GiB | | **Recommended** | 16 vCPU, 32 GiB | ### Storage per node We recommend to start your nodes with `500 GiB` **minimum**. Several of the data plane members use PVCs. While the defaults are kept low - as your platform matures the data will grow and you'll need to expand the storage volumes. ### Sizing storage for your retention The shipped sizes are a starting point, not a capacity plan. They are deliberately modest, so for most production workloads you will be growing them rather than shrinking them. Four keys control everything: | Key | Chart | Default | |---|---|---| | `postgres.volume.size` | `gen0sec-data` | 50 GiB | | `kafka.storage.size` | `gen0sec-data` | 50 GiB | | `rustfs.storageclass.dataStorageSize` | `gen0sec-infra` | 50 GiB | | `rustfs.storageclass.logStorageSize` | `gen0sec-infra` | 10 GiB | Which one to grow depends on what you are retaining: | If you need | Grow | |---|---| | Longer event and telemetry history | `postgres.volume.size`. This is the system of record | | A longer Kafka retention window, or a bigger ingest buffer | `kafka.storage.size` | | More room for datasets and locally-produced artifacts | `rustfs.storageclass.dataStorageSize` | Example for longer retention: ```yaml # gen0sec-data postgres: volume: size: 200Gi kafka: storage: size: 200Gi ``` ```yaml # gen0sec-infra rustfs: storageclass: dataStorageSize: 100Gi ``` :::note[Plan for growth] Volume expansion grows a volume if your StorageClass supports it, and it is straightforward. Nothing shrinks one: reducing a size means recreating the volumes and losing what is on them. Starting at the shipped defaults and growing is therefore the recommended direction. It is the reason for low initial limits. Check that your StorageClass has `allowVolumeExpansion: true` before you install. ::: ### What is not on this page **Throughput.** How many agents, events per second or lookups a given topology sustains is not directly derivable from the charts' parameters. We suggest you scale the platform components and storage sizes according to your topology's specific needs. {/* Four workloads declare no CPU or memory requests: the object store, the Kafka brokers, valkey, and the Postgres operator. They are therefore BestEffort or Burstable and are evicted first under node pressure, which is why the per-node recommendation is driven by limits rather than by the requests total. Raised separately as a chart issue. */} ### Evaluating on one node Use the [Quickstart](./quickstart.mdx). It runs everything at a single replica on one node, for evaluation, training and upgrade rehearsals. ## Tools on your workstation Install these on the machine you run the commands from, not on the cluster. | Tool | Version | Used for | Install | |---|---|---|---| | `helm` | 3.16 or later | Installing the charts, and pulling them from the registry | [helm.sh](https://helm.sh/docs/intro/install/) | | `kubectl` | matching your cluster | Everything cluster-side | [kubernetes.io](https://kubernetes.io/docs/tasks/tools/#kubectl) | | `cosign` | 2.6.3 or later | Verifying signatures on the install kit, images and charts. Older versions cannot see image signatures made by cosign v3 | [docs.sigstore.dev](https://docs.sigstore.dev/cosign/system_config/installation/) | | `skopeo` | any | Copying images between registries | [github.com/containers/skopeo](https://github.com/containers/skopeo/blob/main/install.md) | | `yq` | 4 or later, **mikefarah/yq only** | Reading the release manifest | [github.com/mikefarah/yq](https://github.com/mikefarah/yq#install) | | `jq` | any | Reading the release API | [jqlang.github.io](https://jqlang.github.io/jq/download/) | | `aws` or `mc` | any | Creating the object store bucket, one step during install | [aws-cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) or [MinIO client](https://min.io/docs/minio/linux/reference/minio-mc.html) | | `curl`, `tar`, `openssl`, `sha256sum` or `shasum` | any | Downloading, unpacking, generating secrets, checking checksums | Your OS packages | ### Check your yq There are two separate tools that are called `yq`. The Python one (`kislyuk/yq`) cannot parse the expressions the bundler uses, and it fails with a syntax error partway through. ```bash yq --version ``` The output must contain `mikefarah`. If it does not, install the Go version and point the bundler at it: ```bash export MIKEFARAH_YQ_PATH=/path/to/mikefarah-yq ``` ## Credentials from Gen0Sec You need **two**, and they are different things. Missing the second one produces a working-looking install whose entire fleet silently receives nothing. | Credential | Used by | Used for | |---|---|---| | **Registry token** | The bundler, on your workstation | Pulling charts and images from our registry | | **Your Gen0Sec API key** | The relay pod, inside your cluster | Fetching threat intelligence, GeoIP, models and IDS rules from Gen0Sec | Both are self-service from the dashboard. Neither requires contacting us. :::note[The install kit itself needs no credentials] The kit is published openly. Downloading and verifying it, in [Prepare the artifacts](./04-prepare-artifacts.mdx), happens before you use either credential — so if that download fails, the token is not what to check. The usual cause is a tag that is not published; the [release index](https://releases.gen0sec.com/cerebellum/) lists the ones that are. ::: ### The registry token Create it on the dashboard, under **Repository Access**: ``` https://app.gen0sec.com//repositoryAccess ``` Your workspace id is in the address bar once you are signed in, and the same page is linked from **Clients**. The token is shown **once**, at creation — store it before leaving the page, because it cannot be retrieved afterwards and a lost one has to be replaced. **The username is literally `any`.** Wherever a command asks for one, type `any`. Only the token is checked, so the username can be anything at all — `any` is simply what these docs use so the commands are copy-pasteable. ### Your Gen0Sec API key You can create your API key on the [SaaS Dashboard](https://app.gen0sec.com) Scopes required: - `ids_rules:read`, - `indicators:read`, - `ml_models:read` :::note[Why this one is separate from the registry token] Your agents get their API keys from your own dashboard, and those keys exist only in your cluster's database, so they cannot authenticate to Gen0Sec. The API key created in this step is used by your in-cluster relay to authenticate against the Gen0Sec SaaS, so it can fetch data related to threat intelligence, GeoIP, ML models and IDS rules. The chart expects it as a secret named `gen0sec-download-proxy`, and the relay pod will not start without it, so a missing key is a failed rollout rather than a silently degraded fleet. You will create this secret in the step 2 of [Install](./05-install.mdx). ::: --- # Architecture and data flows Source: https://docs.gen0sec.com/docs/cerebellum/security/architecture # Architecture and data flows Written for a security reviewer. Everything here is observable in a running install, and every claim has a page you can check it against. ## What runs where Two namespaces, three charts, one ingress point. ``` internet │ │ api.gen0sec.com:443 (the only required outbound path) │ ┌───────────────────────────┼──────────────────────────────────────────────┐ │ your cluster │ │ │ ▲ │ │ ┌─────────────────────── │ ─────────────────────────────────────────┐ │ │ │ namespace: gen0sec │ │ │ │ │ │ │ │ │ │ dashboard relay(download-api) 13 application services │ │ │ │ │ │ │ └───────────────────────────────┬───────────────────────────────────┘ │ │ │ in-cluster only │ │ ┌───────────────────────────────┼───────────────────────────────────┐ │ │ │ namespace: gen0sec-system │ │ │ │ │ ▼ │ │ │ │ Postgres (3) Kafka (3) object store (4) caches (2) │ │ │ │ Postgres operator Kafka operator ingress operator │ │ │ └───────────────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────┘ ▲ │ one address, TLS, path-prefix routed │ your agents and your users ``` | Namespace | Contains | Reachable from outside | |---|---|---| | `gen0sec` | The application services and the dashboard | Yes, through the ingress you publish | | `gen0sec-system` | Postgres, Kafka, object store, two caches, and the three operators | No | Nothing in `gen0sec-system` is published. The application reaches it by cluster DNS. ## Trust boundaries Four, and they are worth naming separately because they fail differently. | Boundary | Crossed by | Controlled by | |---|---|---| | **Internet to cluster** | The relay, outbound only, one destination | Your firewall. See [Ports and egress](./egress.mdx) | | **Your network to cluster** | Agents and users, inbound, one address | Your ingress and its TLS | | **Namespace to namespace** | The application reading its databases | Cluster DNS. No NetworkPolicy ships. See [Hardening](./hardening.mdx) | | **Cluster to registry** | Image pulls | The image pull secret, one per namespace | ## Data flows ### Inbound: agents and users | Flow | Path | Authentication | |---|---|---| | Agent telemetry and events | Your address, then the ingest services, then Kafka, then Postgres | Agent API key | | Agent configuration pull and push | Your address, then the configuration services | Agent API key | | Agent artifact download | Your address, then the relay, then Gen0Sec | Agent API key, validated locally. The relay uses your installation's own key upstream, so agent keys never leave. See [Data handling](./data-handling.mdx) | | Dashboard sign-in | Your address, then the dashboard, then Postgres | Email and password, or a passkey, held in your Postgres. No external identity provider at this release | ### Outbound: the relay, plus optional dashboard calls The dashboard can also check for new versions and send email. All of it is optional and carries no agent data. See [Data handling](./data-handling.mdx#the-dashboards-optional-connections). The relay has one destination and one purpose: fetching four licensed datasets that are in no artifact we ship. | Dataset | Why it cannot be shipped | |---|---| | CTI indicators | Licensed by upstream providers | | GeoIP databases | Licensed by MaxMind and IPInfo | | ML models | Published by gen0sec, refreshed independently of releases | | IDS rules | Licensed as part of the Emerging Threats ruleset | The relay is a **relay, not a cache**. Nothing is stored on the way through, so there is no local artifact store to size and no staleness window. ### Never outbound | Dataset | What it is | Where it stays | |---|---|---| | `identity` | Workload identity produced in your cluster | Your object store | | `policy-edges` | Pod IPs, namespaces, NetworkPolicy edges | Your object store | Relaying either would push a map of your workload topology to a service you do not run. It is not configurable and there is no flag that enables it. Uploads are not relayed either: the relay is read-only. ## Where data is stored | Store | Contains | Backed up by you | |---|---|---| | Postgres | The system of record: configuration, agents, policy, history | Yes, always | | Object store | Datasets, and the two locally-produced sets above | Yes | | Kafka | In-flight events between ingest and persistence | No, it is transport | | Cache | Nothing durable | No | See [Back up and restore](./../operate/backup.mdx). ## What the platform requires of your cluster | Requirement | Detail | |---|---| | `cluster-admin` during install | For CRDs and cluster-scoped RBAC. See [Privileges and RBAC](./privileges.mdx) | | Custom resource definitions | Postgres and Kafka operator CRDs, cluster-scoped by nature | | Its own ingress controller | Installed by the infra chart, `ingressClassName: synapse` | | One required outbound destination | `api.gen0sec.com:443` | | No node access, no host mounts, no privileged containers | See [Hardening](./hardening.mdx) | ## Questions a reviewer usually asks next | Question | Page | |---|---| | What exactly can leave the cluster? | [Data handling](./data-handling.mdx) | | What ports and destinations do I open? | [Ports and egress](./egress.mdx) | | Why does it need `cluster-admin`? | [Privileges and RBAC](./privileges.mdx) | | How do I verify what you shipped me? | [Supply chain](./supply-chain.mdx) | | What is the container security posture? | [Hardening](./hardening.mdx) | | What does not work yet? | [Known limitations](./../troubleshoot/known-limitations.mdx) | --- # Data handling Source: https://docs.gen0sec.com/docs/cerebellum/security/data-handling # Data handling ## What leaves your cluster One required connection, outbound, to one destination. The dashboard can make three optional ones, listed after it. | | | |---|---| | Destination | `api.gen0sec.com:443` | | Made by | The relay pod only | | Direction | Outbound, request and response | | Purpose | Fetching four licensed datasets | **What is in the request.** A path naming the artifact, and your own Gen0Sec API key. Nothing else. The relay does not add a customer identifier, a cluster identifier, an agent identifier, telemetry or usage data. **What is in the response.** The artifact: a CTI indicator database, a GeoIP database, an ML model, or an IDS rule bundle. ### The dashboard's optional connections | Destination | When | What is sent | |---|---|---| | `helm.gen0sec.com:443` | Opening the Clients or Components page | A request for the public chart index. Nothing about your install | | `releases.gen0sec.com:443` | Only if **Check for updates online** is on | A request for the latest release's public version list. Nothing about your install | | Your SMTP server | Only if you configure SMTP | The recipient's address, your organization's name, and an invitation or password-reset link | All three fail without affecting anything else. See [Ports and egress](./egress.mdx). ## What never leaves your cluster | Data | What it is | |---|---| | `identity` | Workload identity produced inside your cluster | | `policy-edges` | Pod IP addresses, namespaces, NetworkPolicy edges | | Agent telemetry and events | Everything your agents report | | Configuration and policy | Everything you configure | | Logs | All of it, to container stdout in your cluster | | Database contents | Postgres never leaves | | Dashboard user accounts, passwords and passkeys | Held in your Postgres | | Audit log | Who changed what, in the dashboard or through the API: the user or API key, the action, the changed fields with secret-looking fields redacted, source IP and user agent | `identity` and `policy-edges` deserve the explicit statement, because they are the two that are technically served by the same service that does the relaying. **They are never relayed.** Doing so would push a map of your workload topology to a service you do not run. It is not configurable, and no flag enables it. The upload surface is not relayed either, because the relay is read-only. ## Who validates agent credentials **Your agents' keys are validated in your cluster, and they never leave it.** Every relayed route sits behind the local authentication middleware, and each one carries the same permission it carries in direct mode. So a key that would be refused when served locally is refused it when relayed. ``` agent --[agent key, authorized locally, permission checked per route]--> relay --[your Gen0Sec API key]--> api.gen0sec.com agent key stops here ``` Two separate concerns, deliberately separated: | Hop | Credential | Checked by | |---|---|---| | Agent to relay | The agent's own key | Your cluster, against your installation, with the artifact's permission | | Relay to Gen0Sec | Your Gen0Sec API key | Gen0Sec | | | | |---|---| | Credentials Gen0Sec receives | One, belonging to your installation | | Credentials Gen0Sec never receives | Your individual agent key | | What Gen0Sec can infer about your fleet | Nothing from the credential. Requests carry no agent or cluster identifier | | Revoking one agent | Local, in your dashboard, immediate | | Revoking your installation's upstream access | Revoke or rotate your Gen0Sec API key | This is the answer to "does our agent inventory leave our perimeter". It does not. :::note[Agent keys are never forwarded upstream, even by accident] On a self-hosted deployment the relay never puts a caller's key in an outbound request, in any configuration. A locally-issued key could not authenticate to Gen0Sec anyway, so forwarding it could only leak a local credential to a third party without making the request succeed. With no Gen0Sec API key configured the relay sends no credential at all rather than sending yours. ::: ## Credentials at rest | Credential | Where it lives | Generated by | |---|---|---| | Database role passwords | Kubernetes secrets | The Postgres operator | | Object store keys | A Kubernetes secret | You, during install | | Registry pull credentials | A Kubernetes secret per namespace | You, during install | | Dashboard session signing key | A Kubernetes secret | Generated once, preserved across regeneration | | SMTP credentials | A Kubernetes secret | You, optional | **No credential appears in a chart, a values file or a rendered manifest.** Every one is referenced by name. See [Secrets](./../configure/secrets.mdx). The install procedure reads every credential from a file or from standard input rather than passing it as a command-line argument, so nothing lands in shell history or in the process list. ## Credentials in transit | Hop | Encrypted | |---|---| | Agents and users to your ingress | Yes, TLS you terminate | | Relay to `api.gen0sec.com` | Yes, TLS | | Services to Postgres | Yes, TLS to the pooler | | Services to the object store | Plain HTTP, in-cluster only | | Services to the cache | Plain, in-cluster only, no credential in the URL | The last two are in-cluster traffic on the pod network. If your threat model includes an attacker on the pod network, add NetworkPolicies and a service mesh. See [Hardening](./hardening.mdx). :::note[One deliberate exception, and why] The dashboard's database connection verifies encryption without verifying the certificate chain, because its client library rejects the pooler's self-signed certificate outright while every other service's library accepts encrypt-without-verify. The traffic is encrypted and in-cluster. Do not copy that setting onto the other services: their driver rejects it. ::: ## Data retention Retention is yours. Nothing expires data on your behalf, and nothing is sent anywhere before it is deleted. | Store | Grows with | Sized by | |---|---|---| | Postgres | Agents, events, history | `postgres.volume.size` | | Postgres audit log | Administrative changes, kept indefinitely | Small. See [Known limitations](./../troubleshoot/known-limitations.mdx#the-audit-log-is-never-pruned) | | Kafka | In-flight events, and its retention window | `kafka.storage.size` | | Object store | Datasets and locally-produced artifacts | `rustfs.storageclass.dataStorageSize` | See [Sizing](./../02-requirements.mdx#sizing-a-production-install). ## Deleting everything [Uninstall](./../operate/uninstall.mdx) without `--keep-data` deletes the namespaces and every persistent volume in them: the database, the message log and the object store. There is no copy anywhere else. --- # Ports and egress Source: https://docs.gen0sec.com/docs/cerebellum/security/egress # Ports and egress Same as [Network and connectivity](./../03-network.mdx), reproduced here for better visibility. **One canonical source.** If this table and the install page ever disagree, that is a bug in our publishing, not a judgement call for you. Report it. ## Outbound All TCP, all TLS, all outbound. Nothing needs an inbound rule from the internet. | Destination | Port | Needed by | Purpose | Required | |---|---|---|---|---| | `api.gen0sec.com` | 443 | Cluster pods, the relay only | Threat intelligence, GeoIP, ML models, IDS rules | **Always** | | `releases.gen0sec.com` | 443 | Workstation | Download the install kit and its signature | **Always** | | `registry.gen0sec.com` | 443 | Workstation | Pull charts and images | **Always** | | `registry.gen0sec.com` | 443 | Cluster nodes | Image pulls at install time | Registry install only | | `fulcio.sigstore.dev` | 443 | Workstation | Signature certificate verification | **Always** | | `rekor.sigstore.dev` | 443 | Workstation | Transparency log verification | **Always** | | `tuf-repo-cdn.sigstore.dev` | 443 | Workstation | Signature trust root refresh | **Always** | | Your own registry | 443 or 80 | Cluster nodes, and the host that loads the bundle | Image pulls at install time | Offline bundle only | | `helm.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest agent version, shown on the Clients and Components pages | No. Always attempted. Blocking it only hides the latest version | | `releases.gen0sec.com` | 443 | Cluster pods, the dashboard only | Latest component versions | No. Only if an administrator turns on **Check for updates online** | | Your SMTP server | 587 or 465 | Cluster pods, the dashboard only | Invitation and password-reset email | No. Only if you [configure SMTP](./../configure/identity.mdx#invitations-and-email) | **Only one row is required by the running cluster:** `api.gen0sec.com:443`, and only the relay pod uses it. The dashboard rows are optional: blocking them hides version information, or leaves invitations and password resets without email. Everything else is install-time, and most of it is your workstation rather than your cluster. ## Inbound | Source | Destination | Port | Purpose | |---|---|---|---| | Your agents | Your published address | 443 | Telemetry, events, configuration, artifact download | | Your users | Your published address | 443 | Dashboard | One address serves both, routed by path prefix. See [Publish the endpoints](./../06-publish.mdx#the-route-table). ## In-cluster No NetworkPolicy is required, and none ships. If you enforce default-deny, these are the flows to allow. | From | To | Port | |---|---|---| | `gen0sec` services | `core-pooler.gen0sec-system` (Postgres pooler) | 5432 | | Migration job | `core.gen0sec-system` (Postgres primary) | 5432 | | `gen0sec` services | `core-kafka-bootstrap.gen0sec-system` | 9092 | | `gen0sec` services | `rustfs-svc.gen0sec-system` (object store) | 9000 | | `gen0sec` services | `dragonfly.gen0sec-system` (cache) | 6379 | | Ingress controller | `gen0sec` services | Service ports | | `gen0sec` services | `gen0sec` services | Service ports | | Dashboard (`ui`) pods | The Kubernetes API server | 443 or 6443 | The dashboard reads two ConfigMaps from the API server for the Components tab. Find its address with `kubectl get endpoints kubernetes -n default`. Without that flow only the Components tab degrades. See [Hardening](./hardening.mdx) for what we recommend beyond that. ## Notes that save time **The Sigstore hosts are the ones people forget.** Signature verification is the first command of the install, and it fails without them. They are needed on the workstation, not on the cluster. **A proxy does not help for the cluster's one destination.** Explicit forward proxies are not honoured in-cluster. See [Known limitations](./../troubleshoot/known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy). **The offline bundle removes registry egress, not all egress.** The runtime connection stays. See [What this rules out](./../03-network.mdx#what-this-rules-out). --- # Hardening Source: https://docs.gen0sec.com/docs/cerebellum/security/hardening # Hardening Two lists: what the charts already do, and what is left to you. The second list is short but it is not empty, and we would rather name it than let you assume otherwise. ## Hardened by default Every application service runs with this security context, and it is not opt-in. ```yaml podSecurityContext: runAsNonRoot: true runAsUser: 65532 runAsGroup: 65532 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: [ALL] ``` | Control | Status | |---|---| | Runs as non-root | Yes, uid 65532 | | Read-only root filesystem | Yes | | Privilege escalation | Disabled | | Linux capabilities | All dropped | | Privileged containers | None | | Host network, PID or IPC | None | | Host path mounts | None | | Node access | None | :::note[Read-only root filesystems were not free] Several services defaulted to writing into a relative path under their working directory, which fails outright on a read-only root. Each one now gets an explicit writable volume mounted where it expects to write. That is why the values file carries per-service volume mounts: they are the cost of this control, not decoration. If you add a service-level override that removes one, that service will fail to start. ::: ## Pod Security Admission The application pods satisfy the **restricted** profile. Label the namespace to enforce it: ```bash kubectl label namespace gen0sec \ pod-security.kubernetes.io/enforce=restricted \ pod-security.kubernetes.io/enforce-version=latest ``` **Do not label `gen0sec-system` restricted yet.** The Postgres and Kafka pods there are created by upstream operators and have not been assessed against that profile. Use `baseline` if you need something, and tell us if you require restricted there. ## What you should add None of these ship, and all of them are worth doing. ### NetworkPolicies **No NetworkPolicy ships with the charts.** On a cluster without default-deny, that means the platform namespaces are as open as everything else. Add default-deny in both namespaces, then allow exactly these: | From | To | Port | |---|---|---| | `gen0sec` services | `core-pooler.gen0sec-system` | 5432 | | Migration job | `core.gen0sec-system` | 5432 | | `gen0sec` services | `core-kafka-bootstrap.gen0sec-system` | 9092 | | `gen0sec` services | `rustfs-svc.gen0sec-system` | 9000 | | `gen0sec` services | `dragonfly.gen0sec-system` | 6379 | | Ingress controller | `gen0sec` services | Service ports | | `gen0sec` services | `gen0sec` services | Service ports | | Dashboard (`ui`) pods | The Kubernetes API server | 443 or 6443 | | Relay pod | `api.gen0sec.com:443` | 443, egress | **Egress default-deny is the high-value one here.** The relay is meant to be the only pod that needs a route out. The dashboard's optional egress is listed on [Ports and egress](./egress.mdx). A NetworkPolicy is what turns that from a design intention into an enforced property, and it is the single strongest control you can add to this deployment. ### TLS everywhere it is not already | Hop | Today | Worth adding | |---|---|---| | Ingress to users and agents | TLS you terminate | Already yours | | Relay to Gen0Sec | TLS | Nothing to do | | Services to Postgres | TLS to the pooler | Nothing to do | | Services to the object store | Plain HTTP, in-cluster | A service mesh, if your threat model includes the pod network | | Services to the cache | Plain, in-cluster, no credential in the URL | Enable cache authentication, then move the connection string into each service's secret | ### Resource limits on the four workloads that have none The object store, the Kafka brokers, the ingress cache and the Postgres operator declare no CPU or memory requests. That makes them BestEffort or Burstable, so the kubelet evicts them first under node pressure. For the object store, holding your data, that is the wrong priority order. Set requests on them, or set a namespace `LimitRange` so nothing lands unbounded. See [Sizing](./../02-requirements.mdx#sizing-a-production-install). ### Secrets from an external manager `make-db-secrets.sh` is fine for evaluation. In production, have your secret manager produce secrets of the same shape so credentials never exist as plain Kubernetes secrets you created by hand. Every name and key is listed in [Secrets](./../configure/secrets.mdx). ## Settings you should never enable Two environment variables exist and both disable a security control. Neither is set by the charts, and neither appears anywhere in the install procedure. | Setting | What it does | When it is acceptable | |---|---|---| | `TLS_INSECURE_SKIP_VERIFY` | Stops the relay verifying the upstream certificate | Never in production. It defeats the protection on your only outbound connection | | `MIRROR_TLS_VERIFY=false` | Stops the bundler verifying your registry's certificate when pushing | Only for a plain-HTTP registry on a trusted network, and preferably not then | If you find either one set in a running deployment, treat it as a finding. ## Verifying the posture Confirm no pod runs as root or privileged: ```bash kubectl -n gen0sec get pods -o json | jq -r ' .items[] | .metadata.name + " runAsNonRoot=" + ((.spec.securityContext.runAsNonRoot // false) | tostring)' ``` Confirm nothing holds cluster-admin: ```bash kubectl get clusterrolebinding -o json \ | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | .metadata.name' ``` Confirm only the relay can reach the internet, once you have added egress policy: ```bash kubectl -n gen0sec run netcheck --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/busybox:1.36.1-glibc -- wget -T5 -qO- https://example.com ; echo "exit=$?" ``` A non-zero exit is the result you want. --- # Privileges and RBAC Source: https://docs.gen0sec.com/docs/cerebellum/security/privileges # Privileges and RBAC ## The short answer `cluster-admin` is required **to install**, not to run. It is needed for two things that are cluster-scoped by nature: 1. Installing custom resource definitions. 2. Creating the cluster roles the three operators need. Once installed, no component in the platform holds cluster-admin. ## What is cluster-scoped ### Custom resource definitions From the bundled operators, not written by us. | CRD group | Installed by | Kinds | |---|---|---| | `acid.zalan.do` | Postgres operator | `postgresqls`, `operatorconfigurations`, `postgresteams` | | `kafka.strimzi.io` | Kafka operator | `kafkas`, `kafkatopics`, `kafkanodepools`, `strimzipodsets`, and the Connect and Bridge kinds we do not use | CRDs are cluster-scoped in Kubernetes. There is no namespaced alternative. :::note[Deleting a CRD deletes every resource of that kind, cluster-wide] Including any outside these two namespaces. If anything else in your cluster runs a Postgres or Kafka operator, removing these CRDs would take its databases and clusters with them. [Uninstall](./../operate/uninstall.mdx) removes them by default, but checks first: if any of those kinds is in use outside the two namespaces it is removing, it skips the CRDs and says so. `--keep-crds` skips them unconditionally. ::: ### Cluster roles | Cluster role | Belongs to | What it is for | |---|---|---| | `g0s-postgres-operator` | Postgres operator | Managing Postgres clusters and their pods | | `g0s-postgres-operator-endpoints` | Postgres operator | The `endpoints` permission its own upstream role omits, while still creating a master Endpoints object per cluster | | `postgres-pod` | Postgres pods | What a database pod itself needs | | `strimzi-cluster-operator-global` | Kafka operator | Cluster-scoped reconciliation | | `strimzi-cluster-operator-namespaced` | Kafka operator | Namespaced resources, bound per namespace | | `strimzi-cluster-operator-leader-election` | Kafka operator | Leader election | | `strimzi-cluster-operator-watched` | Kafka operator | Watching the namespaces it manages | | `strimzi-kafka-broker` | Kafka brokers | What a broker pod needs | | `strimzi-entity-operator` | Topic and user operator | Managing topics and users | | `strimzi-kafka-client` | Kafka clients | Client-side permissions | | `g0s-synapse-operator` | Ingress operator | Watching Ingress objects and services | Eleven, and nine of them come straight from the two upstream operators. :::note[The one we added, and why] `g0s-postgres-operator-endpoints` exists because the Postgres operator ships a cluster role without `endpoints`, yet creates a master Endpoints object for every cluster it manages. Without that rule, `postgresql/core` fails with `endpoints is forbidden` and the database never starts. See [Postgres will not start](./../troubleshoot/symptoms.mdx#postgres-will-not-start). ::: ## What the application itself holds The application services in `gen0sec` are not operators and do not watch the cluster. | | | |---|---| | Cluster-scoped permissions | None | | Namespaced permissions | None, except the dashboard (see below) | | Service accounts | One per service. Tokens are not mounted, except the dashboard's | | Node access | None | | Host mounts | None | | Privileged containers | None | :::note[The one exception: the dashboard reads two ConfigMaps] The `ui` service account (`g0s-ui` in `gen0sec`) is bound to the Role `g0s-ui-read-installed-versions` in `gen0sec-system`, which the infra chart creates. It allows `get` on two ConfigMaps by name, `g0s-infra-installed-versions` and `g0s-data-installed-versions`, and nothing else. There is no `list` or `watch`, so it cannot enumerate other ConfigMaps. The dashboard uses it for the [Components](./../configure/components.mdx) tab, and is the only pod with its service account token mounted. ::: See [Hardening](./hardening.mdx) for the container security context. ## Can I install without cluster-admin? Partly, if you split the install between two roles. This is a supported pattern but it is not the documented golden path, so plan for one extra iteration. | Step | Needs | Who | |---|---|---| | Apply CRDs | Cluster-scoped write on `customresourcedefinitions` | Your platform team, once per release | | Create cluster roles and bindings | Cluster-scoped RBAC write | Your platform team, once | | Install the three charts | Namespace-admin on both namespaces | The installer | Render what a chart would create, and hand that to whoever holds the privilege: ```bash helm template g0s-infra $B/charts/gen0sec-infra-*.tgz \ -n gen0sec-system -f $B/values/gen0sec-infra-values-onprem.yaml \ | yq eval-all 'select(.kind == "ClusterRole" or .kind == "ClusterRoleBinding")' ``` ## Verifying after install Confirm nothing holds more than it should: ```bash kubectl get clusterrolebinding -o json \ | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | .metadata.name + " -> " + ([.subjects[]?.name] | join(","))' ``` No entry in that output should reference a `gen0sec` service account. If one does, tell us: that is a defect, not a configuration choice. ## What an uninstall leaves behind | | Removed | Reason | |---|---|---| | Namespaced resources | Yes | Helm owns them | | Objects the operators created outside the releases | Yes | The script removes them; `helm uninstall` alone does not | | CRDs | Yes, unless in use elsewhere or `--keep-crds` | Removal is cluster-wide destructive, so the script checks before it acts | | Cluster roles and bindings | Depends on the chart that created them | Check what remains and remove by hand if nothing else uses the operators | [Uninstall](./../operate/uninstall.mdx) reports what it removed and what it left. --- # Provenance, signatures and SBOM Source: https://docs.gen0sec.com/docs/cerebellum/security/supply-chain # Provenance, signatures and SBOM Everything we publish is signed, and every signature is verifiable by you without asking us for a key. This page also includes the one case where verification cannot follow the artifacts. ## What is signed, and how | Artifact | Signed | Attested | |---|---|---| | Install kit tarball | Detached cosign bundle, published beside it | | | Container images | `cosign sign --recursive`, so the index **and** each per-architecture manifest | SPDX JSON SBOM, one per architecture | | Helm charts | `cosign sign` on the OCI chart artifact | SPDX JSON SBOM | **Keyless, with GitHub OIDC.** There is no long-lived private key to steal, and no public key for you to fetch and trust. Each signature is bound to a workflow identity recorded in a public transparency log, so verification asserts *who built it*, not *who holds a key*. SBOMs are generated with syft and attached as SPDX JSON attestations. :::note[SLSA provenance is deliberately off] BuildKit's provenance attestations add manifests to the multi-architecture index carrying a placeholder platform. That breaks tools which walk an index by platform, including our own mirroring. We ship the SPDX SBOM attestation instead, which is the thing customers actually verify. If you require SLSA provenance, tell us: it is a build-flag change plus mirroring work, not a redesign. ::: ## Verify the install kit Do this before you unpack it. It is step 3 of [Prepare the artifacts](./../04-prepare-artifacts.mdx). ```bash cosign verify-blob \ --bundle cerebellum-install-kit.tar.gz.cosign.bundle \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/release\.yaml@refs/tags/' \ cerebellum-install-kit.tar.gz ``` **Both certificate flags are required.** `cosign` refuses to verify without an expected issuer and an expected identity, so a shortened command fails rather than passing weakly. The identity is pinned to **one workflow, on a tag reference**. A looser pattern would accept a signature produced by any workflow, on any branch, in our organization. That is the difference between "built by our release pipeline from a tag" and "built by somebody at Gen0Sec". ## Verify an image or a chart Against our registry, or against yours after mirroring: ```bash cosign verify \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/gen0sec/cerebellum/\.github/workflows/' \ registry.gen0sec.com/gen0sec/auth-api:0.2.3 ``` Two images are built and signed in their own repositories, so their identity names that repository: | Image | Identity regexp | |---|---| | `spilo` (Postgres) | `^https://github\.com/gen0sec/spilo/\.github/workflows/build-images\.yaml@refs/tags/` | | `synapse-operator` | `^https://github\.com/gen0sec/synapse-operator/\.github/workflows/build-images\.yaml@refs/tags/` | Both are signed by cosign v3, which stores signatures where cosign before 2.6.3 cannot find them. An older cosign reports `no signatures found`. ## Read the SBOM ```bash cosign download attestation \ --predicate-type https://spdx.dev/Document \ registry.gen0sec.com/gen0sec/auth-api:0.2.3 \ | jq -r '.payload | @base64d | fromjson | .predicate' ``` The predicate is an SPDX JSON document. Feed it to whatever consumes SBOMs in your pipeline. ## Verification across the air gap This is the part nobody else documents, so it gets its own section. ### What crosses, and what does not Signature verification needs the public transparency log, and an isolated network does not have one. So the bundler verifies **once, on the connected staging host, before packaging**, and protects everything downstream with checksums. ``` connected staging host isolated network ────────────────────── ──────────────── pull every artifact by digest cosign verify each one (cannot re-verify: no transparency log) write SHA256SUMS -> verify-package re-checks SHA256SUMS print the bundle fingerprint -> mirror re-checks it before pushing package ``` | Stage | Guarantee | |---|---| | Before packaging | Full cryptographic signature verification, every image and every chart | | The tarball | `SHA256SUMS` over every file, plus a **fingerprint**, which is a sha256 of that list | | On the isolated side | `verify-package` and `mirror` both re-check `SHA256SUMS` and refuse to proceed on a mismatch | **Record the fingerprint** that verification prints on the connected side, out of band. Comparing it on the isolated side is what closes the loop between the two networks. :::note[Signature re-verification inside the isolated network is not supported] It would need the transparency log, or a pinned trust bundle carried across and kept current. We do not pretend otherwise. If your policy requires cryptographic verification at the destination rather than checksum verification, tell us. That is a different design, and we would rather discuss it now than have you find the gap during an audit. ::: ### The one case where attestations are lost `--arch amd64` or `--arch arm64` halves the transfer. It also extracts **one manifest out of each multi-architecture index**, which changes every image digest by definition. | | Multi-architecture bundle (default) | Single-architecture bundle | |---|---|---| | Image signatures | Verify | **Verify.** Each per-architecture manifest is signed individually | | SBOM attestations | Verify | **Lost.** They hang off the index, and the index is what is left behind | | Digests match the release manifest | Yes | No, by definition | The bundler warns at the time and annotates the bundle rather than letting this be a silent difference. **If your acceptance process checks attestations, stay on the default.** ## Digest pinning `release-manifest.yaml` in the install kit pins **every image and chart to a digest**, not to a tag. Tags move. Digests do not. That is what makes a release reproducible: the bundler resolves nothing at build time, it fetches exactly the digests the manifest names, and it fails if one is missing. ```bash yq '.spec.images[] | [.image, .imageTag, .digest] | @tsv' $KIT/release-manifest.yaml ``` ## If verification fails | Symptom | Cause | |---|---| | `no matching signatures` | Wrong identity pattern, or the artifact is not ours | | Certificate fetch errors | The Sigstore hosts are blocked. See [Network and connectivity](./../03-network.mdx#egress-list) | | The image verifies, the SBOM does not | Single-architecture bundle. Expected, see above | | `SHA256SUMS` mismatch on the isolated side | The transfer altered the bundle. Re-transfer, do not proceed | **A failed verification is not a step to skip.** The bundler has a flag that drops the check, and a bundle built with it is annotated as unverified. Use it only if cosign genuinely cannot be installed on your staging host, and record that you did. --- # Collect diagnostics Source: https://docs.gen0sec.com/docs/cerebellum/troubleshoot/diagnostics # Collect diagnostics Collect this before you contact us. Everything here is something we would otherwise ask for, so sending it means the first reply is an answer rather than a request for more output. There is no single collection command at this release. This page is the list. ## Set up Run the commands from a directory you can archive afterwards. ```bash NS_SYS=gen0sec-system NS_APP=gen0sec OUT=gen0sec-diagnostics mkdir -p $OUT && cd $OUT ``` Cluster and storage: ```bash kubectl version -o yaml > cluster-version.yaml kubectl get nodes -o wide > nodes.txt kubectl describe nodes > nodes-describe.txt kubectl get storageclass -o yaml > storageclasses.yaml kubectl get pvc -A -o wide > pvc.txt ``` Releases and workloads: ```bash helm list -A > helm-releases.txt for ns in $NS_SYS $NS_APP; do kubectl -n $ns get all -o wide > workloads-$ns.txt kubectl -n $ns get events --sort-by=.lastTimestamp > events-$ns.txt kubectl -n $ns get pods -o yaml > pods-$ns.yaml done ``` Logs from anything not ready: ```bash for ns in $NS_SYS $NS_APP; do for p in $(kubectl -n $ns get pods -o json \ | jq -r '.items[] | select(any(.status.containerStatuses[]?; .ready == false)) | .metadata.name'); do kubectl -n $ns logs $p --all-containers --tail=2000 > logs-$ns-$p.txt 2>&1 kubectl -n $ns logs $p --all-containers --previous --tail=2000 > logs-$ns-$p-previous.txt 2>&1 done done ``` Operators and custom resources: ```bash kubectl -n $NS_SYS logs deploy/g0s-infra-postgres-operator --tail=2000 > logs-postgres-operator.txt kubectl -n $NS_SYS logs deploy/strimzi-cluster-operator --tail=2000 > logs-kafka-operator.txt kubectl -n $NS_SYS logs deploy/g0s-infra-synapse-operator --tail=2000 > logs-ingress-operator.txt kubectl -n $NS_SYS get postgresql,kafka,kafkanodepool,kafkatopic -o yaml > custom-resources.yaml ``` The migration job, which is the most common single cause: ```bash kubectl -n $NS_APP logs job/g0s-db-migrate --tail=2000 > logs-db-migrate.txt 2>&1 ``` Images actually in use, including init containers: ```bash for ns in $NS_SYS $NS_APP; do kubectl -n $ns get pods -o yaml | grep -E '^\s+image:' | sort -u > images-$ns.txt done ``` Routing and the relay: ```bash kubectl -n $NS_APP get ingress -o yaml > ingress.yaml kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' > relay-healthz.json 2>&1 ``` Then pack it: ```bash cd .. && tar -czf gen0sec-diagnostics-$(date +%Y%m%d-%H%M%S).tar.gz gen0sec-diagnostics ``` :::danger[Read the archive before you send it] It is plain text and it is yours. Nothing here redacts anything, so check it, and remove any sensitive data it might have, particularly if you added `helm get values`, which prints whatever you set. None of the commands above read secret contents, database rows or object store objects. Logs can contain request paths, key prefixes, up to 400 bytes of a telemetry body that failed to decode, and the full content of any block event the pipeline parked. ::: :::note[There should be one command for this] A single `collect-diagnostics.sh` in the install kit would replace this page with two lines, and the list above is the specification for it. A single diagnostics tool can accidentally collect data that's sensitive and may fail to redact that information, so that's why we ask you to assemble your own report. Once we're 100% sure we can provide a diagnostics tool that doesn't leak any information of your sensitive data, we'll do so. ::: ## What to tell us alongside it Four lines, and they save a round trip: 1. Which install method you used. 2. The release version. 3. The last command you ran, verbatim. 4. What you expected to happen. --- # Known limitations in this release Source: https://docs.gen0sec.com/docs/cerebellum/troubleshoot/known-limitations # Known limitations If you hit something that is not on this page, we'd like to hear about it! ## No zero-egress deployment **What it means.** Your cluster must reach `api.gen0sec.com` on port 443. That connection carries threat intelligence, GeoIP, ML models and IDS rules, which are licensed upstream data that is in no artifact we ship. **What you can remove.** All dependency on Gen0Sec-hosted registries. The [offline bundle method](./../04-prepare-artifacts.mdx) mirrors every image and chart into your own registry, so your cluster pulls nothing from us. **What you cannot remove.** The runtime connection. There is no supported configuration with zero cluster egress. **If your policy forbids all egress**, please contact us. See [Network and connectivity](./../03-network.mdx). ## Our registry challenges pulls that send no user agent **By design on our side, and it can decide which install method you use.** **Symptom.** Every image pull from `registry.gen0sec.com` returns **403 with a CAPTCHA challenge**, and pods sit in `ImagePullBackOff`. A manual `docker pull` or `skopeo copy` from the same node **succeeds**. That discrepancy is the whole diagnosis. If the CLI works and the kubelet does not, this is it. **Cause.** On some Kubernetes distributions, the component that services the kubelet's pull sets no `User-Agent` header. Go's `net/http` then sends its own default, `Go-http-client/1.1`, and our edge treats that as a scanner signature and issues a challenge. A kubelet cannot solve a CAPTCHA, so the pull never succeeds and never will. It is not your credentials, not the pull secret, not TLS, and not an allowlist. ### What passes and what does not Measured against our registry: | User-Agent | Result | |---|---| | `containerd/1.7.0` | 200 | | `containerd/v2.1.4` | 200 | | `docker/29.4.0` | 200 | | `curl/8.7.1` | 200 | | `skopeo/1.24.0` | 200 | | `Go-http-client/1.1` | **403, challenged** | | `Go-http-client/2.0` | **403, challenged** | **containerd's own user agent is fine.** Any real one is. The failure is specifically the absence of a user agent, which Go turns into `Go-http-client`. ### Confirming it The blocked response identifies what it matched: ``` "user_agent": "Go-http-client/1.1", "ja4_fingerprint": null, "captcha_validated": false ``` Compare a manual pull against what the kubelet gets: ```bash # on the node: succeeds, because the CLI sends its own user agent skopeo inspect docker://registry.gen0sec.com/gen0sec/auth-api:0.2.3 ``` ```bash # what the kubelet sees kubectl -n gen0sec describe pod | grep -A3 Failed ``` Success from the CLI plus 403 from the kubelet confirms it. ### Two ways forward **Option 1: use the offline bundle method.** Nothing to configure on your nodes, and your own registry does not inspect user agents. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). **Option 2: make the pull path send a user agent**, on every node. :::note[Settle this before you choose an install method] If you cannot change container runtime configuration on your nodes, whether by policy or because your distribution manages it, use the offline bundle method. This constraint forces that choice, and it is better made before you start than at the first failed pull. ::: ## The relay cannot use an explicit egress proxy **Symptom.** You set `HTTPS_PROXY` on the relay. Nothing changes, and nothing errors. **Cause.** Every HTTP client in the platform builds its own transport without proxy support, so `HTTP_PROXY` and `HTTPS_PROXY` are ignored inside the cluster. This is by design. **What works instead.** | Option | Notes | |---|---| | Allow `api.gen0sec.com:443` directly | One firewall exception that bypasses the proxy | | Redirect transparently at the network layer | The client does not need to be proxy-aware | A TLS-intercepting proxy also cannot work: the charts expose no way to mount a CA bundle into the platform containers, and the relay rejects the intercepted certificate. ## The object store bucket is not created for you **Symptom.** Services start and then fail to read or write the object store. `download-api` answers `500 NoSuchBucket`. **Cause.** Nothing in the charts creates the bucket. **What to do.** [Install](./../05-install.mdx#5-create-the-object-store-bucket) step 5 creates it. It is a first-class step, not an optional one. ## Three services report readiness from a weaker probe `download-api`, `ids-rules-api` and `service-graph-api` serve a richer `/status/healthz` and, in doing so, never register `/status/readyz`. Requesting it returns 404, so a readiness probe pointed at it fails forever while liveness stays healthy, and the pod sits at `0/1` with zero restarts. **What that means for you.** Do not repoint those services at `/status/readyz`. It will fail. ## Helm never upgrades CRDs Not our defect, but it will bite you on the first upgrade that contains CRDs, so it is here. Helm installs a chart's CRDs on first install and ignores them from then on. An upgrade that ships a new CRD schema fails with a server-side-apply error, or silently rejects new fields. [Upgrade](./../operate/upgrade.mdx) applies them by hand as step 1. Do not skip it. ## Kafka volumes survive an uninstall `deleteClaim: false` is deliberate, so `helm uninstall` leaves the Kafka volumes behind. Reinstalling against them produces `Invalid cluster.id ... Expected X, but read Y`, because the new cluster generates a new identity. Delete the volume claims first, or restore the original cluster. See [Kafka will not start](./symptoms.mdx#kafka-will-not-start). ## The audit log needs new partitions after two years The upgrade to v0.2.0 creates monthly audit log partitions for the previous month and the next 24. Nothing creates later ones yet. Past that, audited actions fail with `no partition of relation "audit_log" found for row`, and rule changes made through the API are lost. Create partitions ahead of time, as the database superuser: ```sql SELECT create_audit_log_partition(date '2028-11-01'); ``` ## The audit log is never pruned It is append-only and kept indefinitely, and the application cannot delete from it. To remove old history, back up and then drop a monthly partition (`audit_log_YYYY_MM`) as the superuser. ## Emails are sent as "My App" Invitation and password-reset emails carry the sender name `My App`, whatever you configure. Only the address, `HOST_ALIAS_EMAIL`, is yours. ## New WAF rule functions need current agents `regex_replace()` and `http.request.raw_path` validate from v0.2.0. The rule language is generated from Synapse 0.8.5, and agents on an older version may not evaluate a rule that uses them. Update agents before you rely on either. ## Connection-state rules need current agents A firewall rule with a connection-state condition, such as `allow established,related`, is only safe on an agent that understands the condition. One that does not ignores it, and the rule then matches every packet. So when a workspace has such a rule, its config is published only if every agent that reported in the last hour advertises support: Synapse 0.8.0 or later, with firewall rules enabled. If one does not, the workspace **keeps its previous config** and no rule change reaches its agents. The config generator names the agents holding it back: ```bash kubectl -n gen0sec logs deploy/g0s-config-generator | grep conn_state ``` Upgrade the listed agents, or remove the rule. Removing only its condition widens it to every packet. An agent silent for an hour stops counting; set `CAPABILITY_STALE_AFTER_SECONDS` to change that. See [Values](./../configure/values.mdx#services). ## Migration `force` means the opposite of what people expect `migrate force ` marks that version **applied** and continues with the next one. To re-run a migration that failed, force the version **before** it. Migrations are not idempotent. Forcing to the wrong version re-runs DDL that already exists and fails with `already exists`. Example: if `20261004150000` fails, run `force 20261004120000`. --- # Symptom index Source: https://docs.gen0sec.com/docs/cerebellum/troubleshoot/symptoms # Symptom index Everything here was reproduced on a real cluster. ## Images ### ImagePullBackOff Three different causes, and the first thing to establish is which. ```bash kubectl -n get pod -o jsonpath='{.spec.containers[*].image}' kubectl -n describe pod | tail -20 ``` | The image reference is | Cause | Fix | |---|---|---| | Correct, and the error mentions authorization | The `gen0sec-registry` secret is missing from that namespace | It is needed in **both** namespaces. See [Install step 2](./../05-install.mdx#the-image-pull-secret) | | Pointing at the wrong registry | A values file was not passed with `-f` | Re-run with `-f $B/values/-values-onprem.yaml` | | A path that does not exist in your registry | Mirror path mismatch, on the offline bundle method | See the next entry | ### 403 with a CAPTCHA challenge from our registry All images, not one, and only on the registry install method. **The tell:** a manual `docker pull` or `skopeo copy` from the same node succeeds while the kubelet's pull returns 403. ```bash # succeeds: the CLI sends its own user agent skopeo inspect docker://registry.gen0sec.com/gen0sec/auth-api:0.2.3 ``` ```bash # 403: what the kubelet gets kubectl -n gen0sec describe pod | grep -A3 Failed ``` On some distributions the component servicing the kubelet's pull sets no `User-Agent`, so Go sends `Go-http-client/1.1`, which our edge challenges as a scanner signature. A kubelet cannot solve a CAPTCHA, so it never recovers. Not credentials, not TLS, not the pull secret. Fix: set a user agent in the container runtime on every node, or use the offline bundle method. Both in [Known limitations](./known-limitations.mdx#our-registry-challenges-pulls-that-send-no-user-agent). ### ImagePullBackOff on a pod you did not configure Operator-launched pods (Postgres, the connection pooler, Kafka brokers, the entity operator) get their image reference from the **operator**, not from the chart. If one of those is stuck, the mirror path is wrong, not the pull secret. Confirm the path exists in your registry: ```bash curl -su : https://$REGISTRY/v2/_catalog?n=200 | tr ',' '\n' ``` Two paths look surprising and are not negotiable: | Image | Must be mirrored to | |---|---| | Kafka broker | `/gen0sec/strimzi/kafka:1.0.0-kafka-4.1.1` | | Entity operator | `/gen0sec/strimzi/operator:1.0.0` | Everything is under `gen0sec/`. Kafka additionally keeps the upstream `strimzi/` path inside it, because the operator's `defaultImageRegistry` replaces only the registry host and keeps the repository path. A Kafka pod asking for `/strimzi/kafka` with no `gen0sec/` means that setting lost the namespace. ### 401 Unauthorized pulling into a Postgres or pooler pod The Postgres operator does not copy `imagePullSecrets` onto the database pods it creates. They inherit them from their ServiceAccount, which `values-onprem.yaml` configures through `configKubernetes.pod_service_account_definition`. If you overrode `configKubernetes` wholesale, you dropped it. Merge instead of replacing. ### manifest unknown A chart is asking for a tag that was never published. Check what it rendered: ```bash helm get manifest -n | grep -E '^\s+image:' | sort -u ``` Then compare against [Component versions](./../configure/components.mdx). ## Postgres ### Postgres will not start `postgresql/core` sits in `CreateFailed`. ```bash kubectl -n gen0sec-system logs deploy/g0s-infra-postgres-operator | grep -i 'could not create' ``` | Log line contains | Cause | Fix | |---|---|---| | `endpoints is forbidden` | The operator's own ClusterRole omits `endpoints`, but it creates a master Endpoints object per cluster | The infra chart adds the missing rule. Confirm: `kubectl get clusterrole g0s-infra-postgres-operator-endpoints` | | `Unsupported value: "18"` | The CRDs in the cluster are older than the operator chart | Helm never upgrades CRDs. Apply them first. See [Upgrade](./../operate/upgrade.mdx) | | Nothing, and no volume is bound | No default StorageClass, or not enough capacity | See [Sizing](./../02-requirements.mdx#sizing-a-production-install) | After fixing the cause the operator does not retry promptly. Force it: ```bash kubectl -n gen0sec-system rollout restart deploy/g0s-infra-postgres-operator ``` ### Migration failures The migration job runs as a Helm hook. On failure Helm aborts and deploys nothing, so the cluster is in a safe state. ```bash kubectl -n gen0sec logs job/g0s-db-migrate ``` | Log line contains | Cause | Fix | |---|---|---| | `relation "..." does not exist` | The chain is running against a database that is not at the version it expects. Usually a forced version, or a `db-migrate` image older than the chart | Read the version table in the next entry before you force anything, then [collect diagnostics](./diagnostics.mdx) | | `role "..." does not exist` | A database role is missing. Older migrations `GRANT` to roles by name and roll back if one is absent. From v0.2.0 a missing role skips the grant with a `NOTICE`, and the service fails later with `permission denied` | The data chart creates them. Confirm: `kubectl -n gen0sec-system get secret \| grep credentials` | | `already exists` | A migration was forced to the wrong version and re-ran DDL | See the next entry | | `canceling statement due to lock timeout` | `20261004150000` could not lock the block events table within 10 seconds. Nothing changed | See [Upgrade](./../operate/upgrade.mdx#5-upgrade-the-platform) | | `no migration found for version` | The `db-migrate` image is older than the schema, typically after a rollback | Use the image of the newest release that ran | ### The schema is marked dirty The migration tool marks the schema dirty when a migration fails halfway, and refuses to continue until you say what happened. ```bash kubectl -n gen0sec-system exec -it core-0 -c postgres -- \ psql -U postgres -d ax_prod -c 'select * from schema_migrations' ``` `force ` means "**this version is applied**, continue with the next one". So to re-run a failed migration, force the version **before** it. Migrations are not idempotent. ## Kafka ### Kafka will not start Broker crash loops with `Invalid cluster.id ... Expected X, but read Y`. The volume survived a previous `helm uninstall`, which is deliberate, and the recreated cluster generated a new identity. Either restore the original cluster or wipe the volumes: ```bash kubectl -n gen0sec-system delete kafkanodepool kafka kubectl -n gen0sec-system delete pvc -l strimzi.io/cluster=core helm upgrade g0s-data ... ``` **This deletes the message data on those volumes.** ### No KafkaTopic resources exist although the chart rendered them Helm can skip re-applying objects that were deleted out of band. Re-apply the stored manifest: ```bash helm get manifest g0s-data -n gen0sec-system | kubectl apply -f - ``` ### Rule changes made through the signal API never appear The request is accepted, but the rule is not created. The signal worker writes an audit record in the same transaction as the rule, and drops the change if that fails. ```bash kubectl -n gen0sec logs deploy/g0s-signal | grep 'Failed to process v2 signal' ``` `permission denied for table audit_log` means the grant was skipped during the migration. `no partition of relation "audit_log"` means the audit log ran out of partitions. See [Known limitations](./known-limitations.mdx#the-audit-log-needs-new-partitions-after-two-years). ### Block events are missing from the dashboard Rows the database refused are parked on the `block_events_dlq` Kafka topic, with the reason in a `dlq_reason` header. Only the refused row is parked, not the batch it arrived in. telemetry-api also logs what it dropped or stored without a field, once a minute per organization: ```bash kubectl -n gen0sec logs deploy/g0s-telemetry-api | grep 'OTLP block events rejected' ``` ## Object store and cache ### Services cannot reach the object store The object store Service is `rustfs-svc`, with the suffix. `S3_ENDPOINT` and `ARXIGNIS_DATA_URL` must match it. ```bash kubectl -n gen0sec-system get svc | grep rustfs kubectl -n gen0sec get deploy g0s-download-api \ -o jsonpath='{.spec.template.spec.containers[0].env}' | tr ',' '\n' | grep -E 'S3_ENDPOINT|DATA_URL' ``` See [Install step 7](./../05-install.mdx#7-install-the-platform). ### 500 NoSuchBucket The bucket does not exist. [Install step 5](./../05-install.mdx#5-create-the-object-store-bucket) creates it. Nothing in the charts does. ### Everything works but reads are slow The shared cache is never fatal. Missing or unreachable, each service logs a warning once and runs on its in-process cache only. Confirm from the service side, not the pod list: ```bash kubectl -n gen0sec logs deploy/g0s-auth-api | grep -i "L2 cache" ``` `L2 cache (Redis) connected to server` means the cache is in use. `falling back to L1-only cache` means `REDIS_URL` is wrong or unreachable. ```bash kubectl -n gen0sec-system get svc dragonfly ``` The Service is deliberately not prefixed with the release name, because `REDIS_URL` names it directly. ### The cache pod crash loops ```bash kubectl -n gen0sec-system logs deploy/dragonfly --previous ``` | Log shows | Cause | Fix | |---|---|---| | `io_uring` errors | Blocked by the host kernel or a seccomp profile | Add `--force_epoll` to `dragonfly.extraArgs` in the infra values | | Read-only filesystem errors | The pod runs with a read-only root, which is why the chart disables snapshotting | Remove any flag you added that re-enables it | ### The cache is being evicted or OOM-killed `maxMemory` is the eviction threshold and the memory limit is the kill threshold. Keep the first comfortably below the second, or the kubelet kills the pod before it gets a chance to evict. ## Install and workstation ### The bundler fails with a yq syntax error You have the wrong `yq`. Two unrelated tools share the name. ```bash yq --version ``` The output must contain `mikefarah`. If not, install the Go version and point the bundler at it with `MIKEFARAH_YQ_PATH`. See [Requirements](./../02-requirements.mdx#check-your-yq). ### zsh: no matches found, before helm runs `zsh` expands `[0]` in a `--set` argument as a glob. Single-quote the whole argument: ```bash --set 'services.ui.ingress.hosts[0].host=example.internal' ``` ### The namespace hangs in Terminating Strimzi puts a finalizer on all 23 KafkaTopics, and once the operator is gone nothing clears it. ```bash kubectl get kafkatopics.kafka.strimzi.io -n gen0sec-system -o name \ | xargs -r -n1 kubectl patch -n gen0sec-system --type=merge \ -p '{"metadata":{"finalizers":null}}' ``` `uninstall.sh` does this for you, in the right order. See [Uninstall](./../operate/uninstall.mdx). ### A pod reports Running but nothing works A pod in `CrashLoopBackOff` still reports `status.phase: Running`. Read the `READY` column, never a field selector on phase. ```bash kubectl -n gen0sec get pods ``` ### Agents get 401 on every relayed artifact Your Gen0Sec API key is wrong, or the secret holds a trailing newline. ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` ``` artifact relayed from upstream path=/v1/ids-rules/version status=401 ``` Recreate the secret with `printf`, not `echo`. `--from-file` stores the bytes exactly, so a newline becomes part of the key. See [Data relay](./../configure/relay.mdx#getting-the-key-right). A missing secret does not produce this: the relay pod fails to start instead. See the next entry. ### CreateContainerConfigError on the relay pod The pod references a secret that does not exist. Almost always `gen0sec-download-proxy`: `downloadProxy.apiKeySecret.name` is set but the secret was never created. ```bash kubectl -n gen0sec describe pod -l app.kubernetes.io/name=download-api | tail -20 kubectl -n gen0sec get secret gen0sec-download-proxy ``` Create it. See [Install step 2](./../05-install.mdx#your-gen0sec-api-key). This failure is the visible one. The dangerous case is the opposite: leaving `apiKeySecret.name` unset renders no secret reference at all, the pod starts, and the failure is the silent 401 above. ### The relay reports 502 on every relayed route The upstream is unreachable. A 502 is deliberate rather than a 404 or 500: a polling agent reads it as "ask again later" and keeps the artifact it already holds. ```bash kubectl -n gen0sec exec deploy/g0s-download-api -- curl -fsS localhost:9999/status/healthz ``` Check that `api.gen0sec.com:443` is allowed. See [Ports and egress](./../security/egress.mdx). If you route egress through an explicit proxy, that is the cause: proxies are not honoured in-cluster. See [Known limitations](./known-limitations.mdx#the-relay-cannot-use-an-explicit-egress-proxy). ## Routing ### An agent gets HTML, or a redirect, from an API The request reached the dashboard's catch-all instead of the intended service. Routing is by longest matching prefix and the dashboard owns `/`, so any path missing from the route table lands there. ```bash kubectl -n gen0sec get ingress -o wide ``` Compare against [Publish the endpoints](./../06-publish.mdx#the-route-table). The failures this causes are quiet by design of HTTP, not by ours: | Missing route | What the agent sees | |---|---| | `/v1/authcheck` | A redirect and HTML. The agent cannot distinguish a bad key from a routing mistake and continues unverified | | `/v1/agents/events` | `SSE connect returned HTTP 404`, then silent fallback to polling, which looks like a slow platform | | `/v1/ban-ledger` | A redirect the agent reads as success. The contribution is lost silently | ### kubectl get ingress shows an empty HOSTS column The ingress controller ignores host-less rules: the object is created, nothing is logged, and no traffic is served. Set `entrypointHost`. See [Publish the endpoints](./../06-publish.mdx). ## Dashboard | Symptom | Cause | |---|---| | Components says it could not read infrastructure versions | The infra chart's `components.ui.*` values do not match the platform release, or a NetworkPolicy blocks the dashboard from the API server. See [Component versions](./../configure/components.mdx#verifying-what-you-are-actually-running) | | Components says data versions are unavailable | The relay or `ids-rules-api` is unreachable from the dashboard | | The online update check cannot reach `releases.gen0sec.com` | Expected when the cluster has no route out. Turn the check off | | No **Forgot your password?** link | SMTP is not configured. See [Invitations and email](./../configure/identity.mdx#invitations-and-email) | | Invitations show a link instead of being emailed | SMTP is not configured, or sending failed. Check `kubectl -n gen0sec logs deploy/g0s-ui` | ## Air gap and mirroring ### Confirming the cluster really is isolated From a pod, not from a node. ```bash kubectl -n gen0sec run netcheck --rm -it --restart=Never \ --image=$REGISTRY/gen0sec/busybox:1.36.1-glibc -- wget -T5 -qO- https://ghcr.io/ ; echo "exit=$?" ``` A non-zero exit is the expected result. If it succeeds, the cluster still has egress and an install that appears to work may be pulling from the internet. ### An image is missing for one architecture You built the bundle with `--arch`. A single-architecture bundle fails when the kubelet tries to pull on a node of the other architecture, which on an isolated cluster is after the transfer. Rebuild without `--arch`, or with the architecture your nodes actually run. See [Prepare the artifacts](./../04-prepare-artifacts.mdx). ## Still stuck [Collect diagnostics](./diagnostics.mdx), then send us the archive. --- # Verify the install Source: https://docs.gen0sec.com/docs/cerebellum/verify # Verify the install Running pods does not mean a working platform. A worker with no Service, a misconfigured credential, or a missing route all leave every pod `Ready` while no data moves. This page defines what "installed successfully" means. ## Before you start - [ ] [First sign-in](./07-first-signin.mdx) complete - [ ] An agent API key was created ## 1. Every pod is ready ```bash kubectl -n gen0sec-system get pods kubectl -n gen0sec get pods ``` Every pod reports `Running` and `n/n` in the `READY` column. On a correct install there are no exceptions: nothing is expected to sit unready. :::note[Check readiness, not phase] A pod in `CrashLoopBackOff` still reports `status.phase: Running`. A `--field-selector` on phase reports a broken install as clean, which is why the command above reads the `READY` column instead. ::: ## 2. The migration job is gone ```bash kubectl -n gen0sec get job g0s-db-migrate ``` The output is similar to this: ``` Error from server (NotFound): jobs.batch "g0s-db-migrate" not found ``` `NotFound` is the pass. Helm deletes the job the moment it succeeds. If it is still there, it failed: `kubectl -n gen0sec logs job/g0s-db-migrate`. ## 3. The relay is connected ```bash kubectl -n gen0sec get --raw \ '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' ``` The output is similar to this: ```json { "status": "ready", "mode": "proxy", "upstream": "https://api.gen0sec.com", "external_relays": ["geoip", "ids-rules"] } ``` `"mode": "proxy"` confirms the relay came up. If the upstream is unreachable, relayed routes answer 502, which a polling agent reads as "ask again later" rather than "the artifact is gone". Then confirm it is actually authorized upstream, which the mode alone does not tell you: ```bash kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5 ``` Any `status=401` means your Gen0Sec API key is wrong. ## 4. The platform serves an agent This is the check that matters. It makes the same requests an agent makes, from outside the cluster, over your published address. ```bash export GEN0SEC_BASE_URL=https://$ENTRYPOINT_HOST export GEN0SEC_AGENT_KEY=$(cat ~/.gen0sec/agent-key) $KIT/scripts/verify-deployment.sh ``` It checks four things: 1. The API refuses unauthenticated callers. 2. Configuration is served. 3. Threat intelligence and IDS rule bundles are published. 4. A downloaded bundle matches the SHA-256 its manifest advertises. It is read-only and safe to run against production. It exits non-zero and names the failing check. **Run it after every install and every upgrade.** It is the cheapest way to catch a deployment that looks healthy and serves nothing. ## Offline bundle only: confirm nothing pulls from outside ```bash kubectl -n gen0sec-system get pods -o yaml | grep -E '^\s+image:' | sort -u kubectl -n gen0sec get pods -o yaml | grep -E '^\s+image:' | sort -u ``` Every image listed is under your registry. Anything else means a values file was not passed with `-f`. :::note[Why both namespaces, and why read the YAML] The application runs in `gen0sec`; only the operators, object store and cache run in `gen0sec-system`. Checking one namespace covers well under half the images. Reading the YAML rather than a container template also catches init containers, which is where the object store's init image is. ::: ## Definition of installed successfully All five are true: 1. Every pod in both namespaces reports `Running` and `n/n` ready. 2. `kubectl -n gen0sec get job g0s-db-migrate` returns `NotFound` or appears as `Completed`. 3. `download-api` reports `"mode": "proxy"` with your upstream, and its log shows no `status=401` or `status=402` on relayed paths. 4. `verify-deployment.sh` exits zero. 5. On the offline bundle method, every image reference is under your registry. If all five hold, the install is done. If any fail, start at the [symptom index](./troubleshoot/symptoms.mdx). ## You may be interested in... - [Upgrade](./operate/upgrade.mdx), so you know how to move before you need to - [Roll back](./operate/rollback.mdx), and where the data boundary is - [Known limitations in this release](./troubleshoot/known-limitations.mdx) --- # Amygdala — Smart Firewall Source: https://docs.gen0sec.com/docs/components/amygdala/ # Amygdala **Detection becomes action, automatically.** When something gets flagged — a bot, a credential-stuffer, a known-bad fingerprint, a fresh threat-intel hit — Amygdala turns it into a block. It is the rule engine and enforcement layer inside every Synapse agent: you author a rule once, the platform distributes it, and each agent's Amygdala evaluates and enforces it locally against its own traffic.
Signals network + ports TLS, HTTP, DNS fingerprints ML verdicts IDS context threat intel workload identity flow behaviour ~80 rule fields Rule engine — runs inside every agent Rules, evaluated in order rule 1 → rule 2 → rule 3 → … the FIRST rule that matches decides an allow does not outrank a drop — order does allow let it through log record what it would have done prove a rule safely drop + persistent block IP · IP+port · fingerprint TLS ClientHello blocking needs an intercept path — the handshake must be parsed before there is a verdict Enforcement backends nftables · iptables (Linux) WFP · Packet Filter (Windows) blocks land in the kernel Hillock kernel enforcement see the Hillock diagram Inputs Evaluation Action Backends Signals network · ports · TLS · HTTP fingerprints · ML verdicts IDS · threat · k8s identity ~80 rule fields Rules, evaluated in order rule 1 → rule 2 → rule 3 → … the FIRST match decides allow let it through log record, enforce nothing prove a rule safely drop + persistent block IP · IP+port · fingerprint Enforcement backends nftables · iptables blocks land in the kernel Inputs Evaluation Action Backends
Rules are walked in order — the first match decides.
## What Amygdala is for - **Auto-respond to detections.** A rule that says *"if the ML classifier calls this traffic C2, block the source for 24 hours"* runs on every agent that receives it. - **Match on far more than addresses.** Rules are written in wirefilter and can reach ML verdicts, IDS context, threat intelligence, Kubernetes workload identity and flow behaviour — not just IP and port. - **Enforce in the kernel.** Blocks land in nftables, iptables or, on Windows, NDIS. Nothing is enforced in userspace on the hot path. - **Try before you enforce.** A `log` action records what a rule *would* have done without acting on it, so a rule can be proven on live traffic first. ## What rules can match on The rule scheme carries around eighty fields. The families matter more than the individual names: | Family | Examples | What it gives you | |---|---|---| | Network | `ip.src`, `ip.dst.asn`, `ip.dst.country`, `ip.proto` | Address, ASN, and geo scoping | | Transport | `tcp.dst_port`, `tcp.window`, `tcp.ttl`, `tcp.mss` | Port and TCP-stack characteristics | | Application | `tls.sni`, `tls.alpn`, `http.host`, `http.path`, `http.user_agent`, `dns.fqdn` | Protocol-level targeting | | Fingerprint | `ja4l.rtt_us`, `ja4ssh.c2s_bytes`, `ja4ls.ttl` | Latency, distance, and SSH session shape | | ML verdicts | `ml.traffic.c2`, `ml.traffic.scan`, `ml.flow.malicious`, `ml.jepa.similarity` | Act on what [Cortex](../cortex/) concluded | | IDS context | `ids.score`, `ids.sids`, `ids.alert_count`, `ids.src_home_net` | Act on what [Thalamus](../thalamus/) matched | | Threat intel | `threat.score`, `threat.advice`, `threat.labels` | Act on feed reputation | | Workload identity | `identity.k8s.src_namespace`, `identity.k8s.dst_workload`, `identity.k8s.src_label` | Rules that name a Kubernetes workload, not an ephemeral pod IP | | Flow behaviour | `flow.burstiness`, `flow.unique_dst_ports`, `flow.flows_per_min` | Catch scanning and beaconing by shape | Because ML, IDS and threat-intel signals are ordinary fields, one rule can combine them — a source is blocked when the classifier calls it a scanner **and** the IDS has already alerted on it, rather than on either alone. ## Actions and precedence A rule resolves to one of three actions: | Action | Effect | |---|---| | `allow` | Let the traffic through | | `log` | Record the match, enforce nothing | | `drop` | Drop it, and install a persistent block at the scope you chose | :::warning[Rules are evaluated in order — first match wins] Amygdala walks the rule set top to bottom and takes **the first rule that matches**. An `allow` does not automatically outrank a `drop`: whichever appears earlier decides, and everything after it is never consulted. Put your exceptions **above** the broad blocks they are meant to escape. A tenant allow-list placed below a global deny will never fire. If no rule matches at all, traffic is allowed. ::: ### What a drop actually blocks | Scope | Blocks | |---|---| | IP | The source address, `/32` for IPv4 or `/128` for IPv6 unless you widen the prefix | | IP + source port | Just that address and port pair | | Fingerprint | Every connection presenting that fingerprint, whatever the address | ## How enforcement lands | Backend | Platform | Notes | |---|---|---| | nftables | Linux | Blocked sets plus per-fingerprint rules in a `prerouting` chain | | iptables | Linux | For hosts that have not moved to nftables | | NDIS | Windows | Drop-verdict interception, the role NFQUEUE plays on Linux | | noop | Any | Evaluate and report without touching the host firewall | Fingerprints derived from the TCP handshake can be enforced as ordinary firewall rules. Blocking on **JA4 — the TLS ClientHello — is different**: the handshake has to be parsed before a verdict exists, so it needs an intercept-for-verdict path. Amygdala offers two, AF_XDP and NFQUEUE, with AF_XDP built for line-rate handshake throughput where NFQUEUE tops out. ## How it fits Amygdala is the *enforcement layer* of the Gen0Sec platform. [Cortex](../cortex/) and [Thalamus](../thalamus/) decide what looks malicious; Amygdala decides what to do about it and makes the kernel do it. [Hillock](../hillock/) is the kernel-level firewall it drives on Linux hosts. ## Use cases - **Block a credential-stuffing tool by fingerprint** so rotating its IPs does not help it. - **Write Kubernetes-aware rules** that name a namespace or workload instead of a pod IP that changes on every restart. - **Require two signals before acting** — an ML verdict *and* an IDS hit — to keep false positives off the block list. - **Ship a rule in `log` mode first**, read what it would have caught, then promote it to `drop`. - **Maintain a per-tenant allow-list** — placed above the global blocks, so it takes effect. ## Next steps - [Synapse](../../client/) — every Synapse agent ships Amygdala configured and ready - [Hillock](../hillock/) — the kernel-level enforcement layer Amygdala drives - [Cortex](../cortex/) — the ML verdicts Amygdala rules can match on - [Thalamus](../thalamus/) — the IDS context Amygdala rules can match on --- # Cortex — Behavioural ML Classifier Source: https://docs.gen0sec.com/docs/components/cortex/ # Cortex **Catch threats no rule has ever seen.** Cortex is the machine-learning inference engine inside the Synapse agent. It scores traffic by *behaviour* — how a client negotiates a TLS handshake, how it paces requests, how it differs from a known browser. When a model's malicious probability crosses your threshold, Cortex fires the enforcement callback the agent gave it.
On the host — nothing about your traffic leaves it Signals fingerprints and flows Fingerprint is this client malicious? can enforce Flow does this flow behave badly? can enforce Traffic class what kind of traffic is this? signal only verdict verdict 7 classes Severity how bad, and for how long? score only, never a TTL Policy layer maps severity to a ban duration deterministic, auditable Enforcement callback the agent's own block path crossing the threshold fires it; the class and severity models do not only two of the models can block Amygdala → Hillock Can enforce Signal only Policy Action Signals fingerprints and flows Fingerprint classifier is this client malicious? can enforce Flow classifier does this flow behave badly? can enforce Traffic classifier what kind of traffic? signal only Severity model how bad, and how long? score only, never a TTL Enforcement callback crossing the threshold fires it only the top two do Policy layer maps severity to a ban duration deterministic, auditable Can enforce Signal only Action Policy
Two models can block. The class and severity models deliberately cannot.
## What Cortex is for - **Detection without signatures.** Find malware, bots, and scrapers before anyone has written a rule for them. - **On-device inference.** Every model is a small ONNX network running on the host's CPU. Nothing about your traffic leaves the machine — no cloud round-trip, no data exfil risk. - **Proportionate response.** A separate severity model decides *how long* a source stays penalised, so a borderline client is not treated like a confirmed one. - **Updated without downtime.** Models arrive with a signed integrity manifest and are hot-swapped at runtime — no restart, no dropped traffic. ## The models Cortex is not one classifier. It is a small stack, each part answering a different question. | Model | Question it answers | Shape | |---|---|---| | **Fingerprint classifier** | Is this JA4+ fingerprint malicious? | Binary verdict, can trigger enforcement | | **Flow classifier** | Does this flow behave maliciously? | 47 features → `benign` / `malicious`, ~2 KB | | **Traffic classifier** | What *kind* of traffic is this? | 37 features → 7 classes, ~5 KB | | **Penalty (severity)** | How bad is this, and for how long? | 21 features → severity in `[0, 1]`, ~0.6 KB | | **JA4-JEPA embedder** | What is this fingerprint *near*? | 512-dimensional embedding, no verdict | The models are deliberately tiny — the largest is about 5 KB — with preprocessing baked into the ONNX graph so the agent and the training pipeline cannot drift apart on feature scaling. ### Which ones can block The fingerprint and flow classifiers expose `classify_and_block`: cross the threshold and Cortex invokes the callback the agent supplied, which is what turns a score into an XDP or nftables ban. The traffic classifier deliberately does **not** gate traffic. Its seven-class output is a parallel signal that informs severity and enrichment; treating a multi-class label as a block/allow decision is out of scope for it by design. The severity model never emits a ban duration either. It produces a bounded score, and the policy layer maps that to a TTL through a deterministic rule — so the model's role stays auditable and the response stays proportionate. ## How it fits Cortex is the *new-threats detector* in the platform. [Thalamus](../thalamus/) catches signatures we already know; Cortex catches the rest. It scores the fingerprints [Dendrite](../dendrite/) produces, and hands its verdicts to the enforcement path. :::note[Inference runs on Linux] Full inference is Linux-only. Cortex compiles on other platforms so the rest of the agent builds and tests cross-platform, but the inference entry points return errors there rather than scoring traffic. ::: :::note[Model maturity varies, and the model cards say so] The runtime paths are production-ready, but the models are at different stages: the traffic classifier is **early access**, and the severity model currently ships **trained on synthetic data**. Each model has a card covering training data, evaluation, and limitations — read the card for any model you intend to make a policy decision on. ::: ## Use cases - **Detect bots without JavaScript challenges** — works for APIs, mobile apps, and headless clients. - **Block credential-stuffing attempts** even when the attacker rotates IPs, since the fingerprint survives the address change. - **Scale the punishment to the offence** — a scanner that trips one check gets a short penalty; a source that trips several gets a long one. - **Score every connection** so you can rank investigation queues by ML confidence rather than by alert volume. ## See also - [Synapse documentation](../../client/) — the agent that ships Cortex - [Dendrite](../dendrite/) — the fingerprint source Cortex classifies - [Thalamus](../thalamus/) — signature detection, the other half of the pair - [Amygdala](../amygdala/) — the enforcement layer Cortex triggers --- # Dendrite — Fingerprint Capture Source: https://docs.gen0sec.com/docs/components/dendrite/ # Dendrite **See every connection. Know every client.** Dendrite is the sensor layer at the heart of every Gen0Sec deployment. It captures every connection on the wire and identifies *what's on it* — what client made it, what tool sent it, what protocol it used. The fingerprints it produces feed everything else: rules, ML, alerts.
Synapse agent — capture only, no enforcement Network traffic on the wire Linux capture AF_PACKET · PF_RING TC eBPF · TC egress in-process TLS read pick at deploy time Windows capture NDIS AF_XDP handshake tracking Parse protocol fields from the frame never decrypts a session JA4+ fingerprints JA4 · JA4S TLS client / server JA4T · JA4TS TCP client / server JA4H HTTP headers JA4L latency, distance JA4SSH SSH session JA4X X.509 certificate JA4D · JA4D6 DHCP / DHCPv6 ten types, both sides of the conversation Cortex classifies Amygdala blocks on Thalamus inspects Capture Processing Fingerprints Boundary Network traffic on the wire Capture Linux: AF_PACKET, PF_RING, TC eBPF, in-process TLS Windows: NDIS, AF_XDP pick at deploy time Parse protocol fields from the frame never decrypts a session Ten JA4+ fingerprints JA4 · JA4S TLS JA4T · JA4TS TCP JA4H HTTP JA4L latency JA4SSH JA4X JA4D JA4D6 both sides of the conversation to Cortex · Amygdala · Thalamus classify · block on · inspect Capture Processing Fingerprints Consumers
Capture only — Dendrite observes and parses; it never blocks.
## What Dendrite is for - **Identify clients without IPs.** JA4+ fingerprints describe the *client*, not the address. The same tool from a hundred rotating IPs has the same fingerprint. - **The full JA4+ suite.** Ten fingerprint types across TCP, TLS, HTTP, SSH, DHCP and X.509 — both halves of each conversation, not just the client side. - **Read TLS plaintext without terminating TLS.** Dendrite hooks the library, not the connection, so it sees decrypted bytes with no proxy, no re-encryption and no private keys. - **Linux and Windows.** One API across both. The capture backends differ, because the kernels do. ## Capture only, by design Dendrite has no enforcement code, no firewall and no proxy. It observes and parses; consumers like [Amygdala](../amygdala/) and [Hillock](../hillock/) decide what to do with the events. That boundary is deliberate and enforced in the code — fingerprint blocking was deliberately removed from Dendrite and lives downstream. A bug in a sensor should cost you visibility, never availability. ## The fingerprints | Type | Covers | Example | |---|---|---| | `JA4` | TLS Client Hello | `t13d1516h2_8daaf6152771_e5627efa2ab1` | | `JA4S` | TLS Server Hello | `t1302h2_1301_a56c5b993250` | | `JA4T` | TCP SYN — the client's stack | `65535_2-4-8-1-3_1460_7` | | `JA4TS` | TCP SYN-ACK — the server's stack | `65535_2-4-8_1460_0` | | `JA4H` | HTTP request headers | `ge11cr15enus_a1b2c3d4e5f6_…` | | `JA4L` | Latency and distance | `12500_64_407` | | `JA4SSH` | SSH session behaviour | `c14s14_c14s14_c14s14` | | `JA4X` | X.509 certificate | `aae71e8db6d7_b186095e22b6_…` | | `JA4D` | DHCP (IPv4) | `disco_8_a1b2c3d4e5f6_000000000000` | | `JA4D6` | DHCPv6 | `solicit_6_a1b2c3d4e5f6_…` | Every type parses back from its string form and supports wildcard patterns, so a rule can match a family of clients rather than one exact build. ## How it captures Capture backends are selected at build time, so an agent carries only what it uses. | Platform | Backend | What it gets you | |---|---|---| | Linux | AF_PACKET | Works on any kernel, no extra modules. The default | | Linux | PF_RING | Zero-copy, for high-rate links. Needs `pf_ring.ko` | | Linux | TC eBPF | JA4TS, JA4L, JA4S and JA4SSH computed in-kernel | | Linux | uprobe | TLS plaintext from the process, before it hits the wire | | Linux | TC egress | Mirrors outbound payloads to [Thalamus](../thalamus/) for inspection | | Windows | NDIS | Packet capture through the Windows Packet Filter driver | | Windows | AF_XDP | High-rate capture path | | Windows | userspace | JA4L handshake tracking, the equivalent of the Linux TC program | ### Reading TLS without breaking it Most tooling sees TLS as opaque, or terminates it and takes on the keys. Dendrite does neither: it attaches to the TLS library inside the process and reads the plaintext the application already has. | Runtime | How | |---|---| | OpenSSL, BoringSSL, LibreSSL | uprobe on `SSL_read` / `SSL_write` | | GnuTLS | uprobe on `gnutls_record_recv` / `gnutls_record_send` | | NSS | uprobe on `PR_Read` / `PR_Write` | | Go `crypto/tls` | uprobe resolved through `.gopclntab` | | Java JSSE | JVMTI agent | | .NET `SslStream` | Covered by the OpenSSL hooks on Linux | This needs a dynamically linked TLS library. A binary that statically links its TLS stack exposes no symbol to attach to, and Dendrite falls back to wire-level fingerprints for it. ## How it fits Dendrite is the *sensor*. Everything else in the platform — [Cortex](../cortex/)'s predictions, [Amygdala](../amygdala/)'s rules, Cerebellum's correlations — operates on the fingerprints Dendrite produces. ## Use cases - **Identify a credential-stuffing tool across rotating IPs** by its TLS fingerprint. - **Detect a new bot family** by spotting fingerprints with no historic baseline. - **Audit your own client fleet** — Dendrite shows you which TLS libraries your services actually negotiate with, and `JA4TS` tells you what your servers look like from outside. - **Inspect application-layer traffic that never appears in cleartext on the wire**, without standing up a proxy. ## See also - [Synapse documentation](../../client/) — the agent that ships Dendrite - [Cortex](../cortex/) — classifies the fingerprints Dendrite emits - [Amygdala](../amygdala/) — acts on the fingerprints downstream - [Thalamus](../thalamus/) — inspects the payloads Dendrite mirrors --- # Hillock — Kernel-level Firewall Source: https://docs.gen0sec.com/docs/components/hillock/ # Hillock **Drop traffic before it touches your stack.** Hillock is the kernel-level firewall that lives inside every Synapse agent. Malicious traffic is filtered, rate-limited, or dropped before it ever reaches your application — no proxy in the path, no TLS termination, no latency penalty.
Amygdala decides installs IP · port · fp · rate Linux kernel Packet arrives on the wire XDP before routing earliest drop point pass · drop · reply · to userspace TC (ingress/egress) both directions rate limiting traffic counters falls back: nftables → iptables What the kernel compares IP address and prefix · port · protocol TCP-handshake fingerprint · per-source rate entries can carry a TTL and expire themselves only things readable straight from the packet DROP discarded PASS to your app Windows same rule model firewall backend in development Fast path TC layer Match keys Verdict Amygdala decides installs IP · port · fp · rate XDP before routing earliest drop point pass · drop · reply · to userspace TC — ingress and egress rate limiting, counters falls back: nftables → iptables What the kernel compares IP + prefix · port · protocol TCP handshake fingerprint per-source rate entries can expire on a TTL PASS to your app / DROP Fast path TC layer Match keys Verdict
XDP can pass, drop, reply to the sender, or hand a packet to userspace. It cannot forward to another host.
## What Hillock is for - **Enforce in the kernel, not in userspace.** Blocks run as eBPF programs attached to the interface, so a dropped packet never costs a context switch into your application. - **Drop as early as the kernel allows.** The XDP path runs before routing, which is the earliest point a packet can be discarded. - **Rate limiting.** Cap noisy clients in the kernel, without writing custom middleware. - **Live traffic counters.** TCP, UDP, and ICMP metrics in real time, read straight from the BPF maps. - **Best available, with fallback.** XDP/eBPF where the kernel allows it, then nftables, then iptables — the rule does not change, only the machinery under it. ## What it enforces on Hillock is the *primitive layer*. It matches on what is cheap to read in the kernel: | Match | Notes | |---|---| | IP address | Source or destination, with prefix | | Port | TCP and UDP | | Protocol | TCP, UDP, ICMP | | TCP-handshake fingerprint | JA4T-style, computed from SYN header fields | | Rate | Per-source thresholds enforced in-kernel | Entries can carry a **TTL**, expired by a BPF timer, so a temporary block clears itself without a userspace sweep. :::note[Richer matching resolves before it gets here] Hillock has no notion of ASN, country, ML verdict or threat score — none of those are readable from a packet header. They reach it as addresses, by two different routes. **Ahead of time.** Cerebellum resolves an ASN or country in your [access rules](../../features/access-rules.mdx) into the IP ranges it covers, and ships the ranges. The agent never sees the ASN. **At the moment of decision.** [Amygdala](../amygdala/) evaluates its own rules on the host — ML verdicts, IDS context, live GeoIP — and when one fires, installs an IP, port, fingerprint or rate block here. Either way the kernel only ever compares something it can read directly, which is why enforcement stays cheap no matter how elaborate the policy above it. ::: ## How it attaches | Path | Direction | Use it for | |---|---|---| | XDP | Ingress, before routing | Earliest and cheapest drop on Linux | | TC (clsact) | Ingress and egress | Egress control, rate limiting, and global metrics | | nftables | Host firewall | Fallback where the BPF path is unavailable | | iptables | Host firewall | Fallback again, for hosts without nftables | :::note[Windows is the same model, not yet the same backend] The goal is one firewall platform with one rule model across Linux and Windows. The Windows firewall backend is still in development, so treat Windows enforcement as a roadmap item rather than something to build policy on today. ::: :::warning[XDP cannot forward a packet somewhere else] The XDP stage has four outcomes, and it is worth knowing which is which: | Verdict | What it does | |---|---| | Pass | Let the packet continue up the stack | | Drop | Discard it | | **Reply** | Answer the sender in place, out of the arrival interface — used for SYN cookies and refusals | | **To userspace** | Hand the packet to an AF_XDP socket for capture | What it cannot do is send a packet to a *different* destination. There is no verdict that forwards to another host, another interface, or a scrubbing appliance — the reply path answers whoever sent the packet, and nothing else. If your policy needs traffic moved rather than dropped or answered, that belongs upstream of Hillock. ::: ## How it fits Hillock is the *muscle* in every Synapse agent. When [Amygdala](../amygdala/) decides a source should be blocked, Hillock is the layer that makes the kernel do it — and [Cortex](../cortex/) and [Thalamus](../thalamus/) are what feed Amygdala that decision. ## Use cases - **Drop bot traffic at wire speed across thousands of servers**, without taking a box out of service to do it. - **Rate-limit aggressive scrapers per source** with no application changes. - **Expire a block automatically** by giving it a TTL, instead of tracking cleanup yourself. - **Read live TCP/UDP/ICMP counters** for cost attribution and incident review. ## See also - [Synapse documentation](../../client/) — the agent that ships Hillock - [Amygdala](../amygdala/) — the rule engine that decides what Hillock enforces - [Cortex](../cortex/) — ML verdicts that can end in a Hillock block - [Thalamus](../thalamus/) — signature hits that can end in a Hillock block --- # Hippocampus — Service Graph Engine Source: https://docs.gen0sec.com/docs/components/hippocampus/ # Hippocampus **Knows what talks to what.** Hippocampus is the platform's map-builder. Every other component reasons about a single connection in front of it; Hippocampus reasons about the *relationships* — which workloads talk to each other, which are supposed to, and where reality and policy have drifted apart. It is named for the part of the brain that builds spatial maps, which is exactly its job here.
Observed real connections between workloads what IS Declared the edges you say are allowed what SHOULD BE Control plane — not in the packet path Workload identity resolve an address to the workload behind it survives a reschedule Compare declared · undeclared · violation the disagreement is the finding The map workloads and the edges between them queryable, out of band Flat lookups to the agent this address is that workload this edge is declared, or not no graph consulted per packet Amygdala matches them as ordinary rule fields Dashboard and Cerebellum query the graph itself Observed Declared Reasoning To the agent Observed real connections what IS Declared the edges you allow what SHOULD BE Workload identity an address resolves to the workload behind it survives a reschedule Compare declared · undeclared · violation the disagreement is the finding Flat lookups to the agent this address is that workload this edge is declared, or not no graph consulted per packet Observed Declared Reasoning To the agent
The graph is control-plane. Agents never consult it per packet.
## What Hippocampus is for - **Turn traffic into a map.** Observed connections become a graph of workloads and the edges between them, built from what actually happened rather than from a diagram. - **Give workloads durable names.** A pod IP is meaningless an hour later. Hippocampus resolves traffic back to the workload behind it, so policy can name a service instead of an address. - **Compare declared against observed.** You declare which services may talk; every real connection is classified against that, and the disagreements are the finding. - **Answer questions about reach.** Given the map, what could a workload get to from where it sits? ## Where it runs Hippocampus is **control-plane**. It is not in the packet path, and no agent consults a graph per packet. That split is deliberate and it is what keeps enforcement cheap. The graph is built and queried out of band; what reaches the agent is a flat lookup — this address is that workload, this edge is declared or it is not. [Amygdala](../amygdala/) matches on those as ordinary rule fields, so an east-west rule costs no more to enforce than an address block. ## How it fits | Component | Reasons about | |---|---| | [Dendrite](../dendrite/) | This connection — who is on it | | [Thalamus](../thalamus/) | This traffic — does it match a signature | | [Cortex](../cortex/) | This client — how does it behave | | **Hippocampus** | **The estate — what talks to what, and what should** | | [Amygdala](../amygdala/) | The decision | | [Hillock](../hillock/) | Making the kernel act on it | Cross-site correlation — the same workload observed from more than one cluster — is Cerebellum's job rather than the agent's. ## Use cases - **Find the edges nobody documented** — the batch job quietly reaching the production database, the service still calling a dependency that was retired. - **Microsegment without maintaining IP lists**, by declaring that only `api` may reach `payments` and letting workload identity keep that true through every reschedule. - **Scope a compromise** by asking what a workload could have reached from where it sat. - **Turn a policy violation into a block** rather than a ticket, once the declarations are trustworthy. ## See also - [Service Graph](../../features/service-graph.mdx) — the capability, and how to adopt it - [Amygdala](../amygdala/) — matches on the workload identity and edge classification it produces - [Cortex](../cortex/) — behavioural scoring on individual clients --- # Thalamus — Intrusion Detection Engine Source: https://docs.gen0sec.com/docs/components/thalamus/ # Thalamus **Match known attacks, on every protocol you actually run.** Thalamus is the intrusion detection engine inside every Gen0Sec sensor and Synapse agent. It inspects traffic at wire speed, matches against a rule corpus you control, and emits structured alerts straight to Cerebellum or your SIEM.
Synapse agent / Gen0Sec sensor Network live traffic or a pcap Capture AF_XDP AF_PACKET pcap file Windows path no sampling Decode Ethernet → IP TCP / UDP / ICMP flow tracking 1,000,000 flows Protocol parsers web · mail · files DNS, LDAP, Kerberos SSH, RDP, VNC PostgreSQL Modbus, DNP3, ENIP 30+ protocols, IT and OT Rule match Suricata-compatible signatures you supply thresholding and rate limiting your feeds, unchanged EVE JSON Your SIEM or Cerebellum ~30 event types Community ID Capture Decode Parsers Detection Network live traffic or a pcap Capture AF_XDP · AF_PACKET pcap file · Windows no sampling Decode Ethernet → IP → TCP/UDP flow tracking 1,000,000 flows 30+ protocol parsers web · mail · files · DNS LDAP · Kerberos · SSH · RDP Modbus · DNP3 · EtherNet/IP IT and OT on one engine Rule match Suricata-compatible signatures thresholding, rate limiting your feeds, unchanged EVE JSON Your SIEM or Cerebellum ~30 event types Community ID Capture Decode Parsers Detection
From the wire to your SIEM, with every packet the engine receives inspected in full.
## What Thalamus is for - **Bring your own rule feeds.** Thalamus accepts Suricata-compatible signatures — community feeds, ET Open, ET Pro, your own custom rules. No translation step, no proprietary rule language. - **Thirty-plus application protocols.** Not just HTTP and DNS. Databases, directory services, remote desktop, and industrial control protocols are parsed in the same engine. - **Suricata-compatible EVE JSON.** Around thirty event types, so the SIEM pipelines and dashboards you already have keep working. - **Inspect TLS payloads without terminating TLS.** With the uprobe path enabled, Thalamus inspects plaintext [Dendrite](../dendrite/) reads from the process — no proxy, no key escrow. ## Protocol coverage Every one of these has a dedicated parser, so rules can match on protocol fields rather than raw bytes. | Area | Protocols | |---|---| | Web | HTTP/1, HTTP/2, WebSocket, QUIC, TLS | | Mail | SMTP, IMAP, POP3 | | Files and shares | FTP, SMB, NFS, TFTP | | Infrastructure | DNS, DHCP, NTP, SNMP, LDAP, Kerberos, IKE, SIP, DCERPC | | Remote access | SSH, RDP, Telnet, VNC/RFB | | Databases | PostgreSQL | | Industrial (OT) | Modbus, DNP3, EtherNet/IP | | Other | MQTT, IRC, BitTorrent DHT | HTTP/1 parses through OISF's libhtp by default, so its quirk handling matches Suricata's. A built-in parser is used instead when libhtp is compiled out. ## Output Alerts are Suricata-compatible EVE JSON, covering `alert`, `anomaly`, `flow`, `netflow`, `stats`, `fileinfo`, `frame`, `packet`, `drop`, and a per-protocol record for most of the parsers above. Records can also carry a **Community ID** — switch it on with a seed, and a Thalamus event and an event from any other Community-ID-aware tool describe the same flow under the same key. ## How it captures | Backend | Use it for | |---|---| | AF_XDP | Highest throughput on Linux. Also where in-kernel elephant-flow bypass is available | | AF_PACKET | Works on any Linux kernel with no special setup | | pcap file | Offline analysis and rule testing against a saved capture | | Windows | Windows hosts, through [Dendrite](../dendrite/)'s eBPF/XDP backend | Flow state is tracked in a lock-free table sized at **one million concurrent flows** by default, with a configurable idle timeout. Suricata-style thresholding and rate limiting suppress alert floods without dropping the underlying detection. :::note[No sampling, but capture drops are real] Thalamus never samples: there is no 1-in-N, no flow sampling, and no percentage. Every packet the engine receives is inspected in full. Packets can still be dropped *before* the engine, by the capture layer, if traffic outruns the host. That is counted and exported as `packets_dropped` rather than hidden — if the number is not zero, you are not seeing everything, and the metric is how you find out. ::: ## How it fits Thalamus is the *known-bad detector* in the platform. Where [Cortex](../cortex/) catches new threats by behaviour, Thalamus catches the ones the security community has already signature-matched. ## Use cases - **Detect known C2 communication patterns across every site.** - **Subscribe to threat intel feeds and apply them everywhere instantly** — the rules you already licensed for Suricata run unchanged. - **Monitor OT and IT on one engine.** Modbus and DNP3 traffic is parsed by the same sensor watching HTTP, so plant and corporate networks share a rule pipeline. - **Inspect encrypted application traffic** for hosts where the uprobe path is enabled, without standing up a TLS-terminating proxy. - **Feed Cerebellum the signature hits** so it can correlate with ML output and behavioural anomalies. ## See also - [Synapse documentation](../../client/) — the agent that ships Thalamus - [Dendrite](../dendrite/) — supplies the TLS plaintext Thalamus inspects - [Cortex](../cortex/) — the ML side of detection - [Amygdala](../amygdala/) — acts on what Thalamus finds --- # Access Rules Source: https://docs.gen0sec.com/docs/features/access-rules # Access Rules Access rules block traffic by IP address at the earliest point the kernel offers. A blocked packet is discarded before routing — it never reaches your application, and it never costs a context switch into user space. This is the bluntest and cheapest control Synapse has. It answers one question — *is this address allowed?* — and answers it fast. ## What it gives you - **Blocking that costs nothing per rule.** Matching is a map lookup, so the cost does not grow as the blocklist does. A hundred thousand blocked addresses are as cheap as ten. - **Threat intelligence, applied automatically.** Gen0Sec's feeds keep the blocklist current without you curating it. - **Enforcement below your stack.** No proxy in the path, no TLS termination, no application change. - **Visibility into what it dropped**, so a block is auditable rather than a silent disappearance. ## How it works 1. A packet arrives and is inspected at the XDP hook, before routing. 2. Its source address is looked up in a kernel map. 3. It is passed or dropped there and then. 4. The drop is counted, and the address can be surfaced as an event. ## Write an ASN, ship addresses You write access rules in the terms you actually think in — **this ASN**, **this country**, this address — and the kernel still only ever compares addresses. Those are not in tension, because the translation happens before the rule reaches the agent. Cerebellum resolves an ASN or a country code into the IP ranges it currently covers, using the Gen0Sec GeoIP database, and ships the resolved ranges. The agent receives addresses and nothing else. | You write | The agent receives | |---|---| | `AS15169` | Every prefix that ASN announces | | A country code | Every range allocated to it | | An address or CIDR | Itself, unchanged | That split is what keeps enforcement free. There is no lookup on the packet path — no GeoIP query per packet, no ASN resolution in the hot path, nothing that grows with the size of your policy. Just a map lookup on the source address. It also means the resolution is **as fresh as the last config generation**, not as fresh as the packet. An ASN that announces a new prefix is covered once the config regenerates, not the instant it happens. For most policy that is the right trade; where it is not, match on something the agent evaluates itself. ### What it genuinely cannot express | Not available here | Where it lives | |---|---| | TLS SNI, HTTP host or path | [WAF](./waf.mdx) | | ML verdicts, IDS context, reputation score | [Amygdala](../components/amygdala/) rules, evaluated on the host | | Port, protocol, egress direction | [Firewall Rules](./firewall-rules.mdx) | [Amygdala](../components/amygdala/) evaluates `ip.src.asn` and `ip.src.country` itself, against a local database — so the same question can be asked either way. Pre-resolved access rules are cheaper; an Amygdala rule is evaluated live and can be combined with signals that only exist on the host. ## Where it runs By default Synapse attaches only to physical uplink interfaces. Loopback, and CNI or virtual devices such as `veth`, `cilium`, `docker` and bridges, are skipped. :::caution[That default is protecting you on Kubernetes] Attaching XDP to a CNI device puts it on your cluster's own datapath. On Cilium nodes that breaks networking. The default exists to avoid it — override it only when you have a specific reason, and know which interface you are attaching to. ::: ## Limits worth knowing - **Inbound only.** The XDP hook sees traffic arriving. Egress control is [Firewall Rules](./firewall-rules.mdx). - **It cannot forward elsewhere.** The kernel stage can pass, drop, or answer the sender in place; it has no verdict that sends a packet to another host or a scrubbing appliance. - **Requires a capable kernel and privileges** — Linux 4.18 or newer, and `CAP_BPF` plus `CAP_NET_ADMIN`. ## Use cases - **Apply a threat feed fleet-wide** without touching any application. - **Drop a scanning subnet at line rate** across thousands of hosts at once. - **Absorb a volumetric source** without it consuming application capacity. - **Keep an audit trail of what was blocked**, and by which rule. ## See also - [Fleet Bans](./fleet-bans.mdx) — bans this layer reports, corroborated across your fleet - [Smart Firewall](./smart-firewall.mdx) — the dynamic counterpart: blocks by fingerprint rather than by a static list - [Firewall Rules](./firewall-rules.mdx) — port, protocol and egress matching - [Configuration](../client/configuration.mdx) — every key, including interface selection and logging - [Hillock](../components/hillock/) — the kernel firewall that enforces these rules - [Amygdala](../components/amygdala/) — the rule engine that decides what belongs in the blocklist --- # Content Scanning Source: https://docs.gen0sec.com/docs/features/content-scanning # Content Scanning Content scanning hands a request body to a ClamAV daemon and refuses the request with **403** if it comes back with a signature. It runs in the proxy, in the request path, so infected content never reaches your storage, your queue or a downstream worker. Which requests get scanned is decided by a rule, not by a config key. ## Scanning is rule-driven Turning the scanner on scans nothing by itself. A [WAF](./waf.mdx) rule with the `content_scanning` action is what selects a request: ```text starts_with(http.request.path, "/upload") ``` Give that rule the `content_scanning` action and bodies on `/upload` get scanned. Everything else is untouched, and costs nothing. That is the important property. The expense of body inspection lands only on traffic you named, so a JSON API with one upload endpoint pays for the upload endpoint rather than for every write. :::note[The agent tells you when nothing will be scanned] `enabled: true` with no `content_scanning` rule is a configuration that looks armed and inspects nothing, so the agent logs it on every load: *content scanning is enabled but no WAF rule uses the content_scanning action*. If you see that line, the rule is missing. ::: `content_scanning` is terminal under first-match-wins, like every other action — a request that matches an earlier rule never reaches it. It is also request-phase only; by the response phase the body is gone. ## What happens to a selected request 1. The body is held while ClamAV inspects it — multipart requests field by field, anything else whole. 2. A clean verdict forwards upstream. 3. A signature hit returns **403**, logs the event, and the upstream is never called. The body is *contained* rather than streamed: it is held until the verdict arrives, so a flagged body does not reach the backend while the scan is still running. ## Bodies that cannot be inspected A rule can select a request whose body cannot actually be scanned — larger than `max_file_size`, or chunked in a way that turns out to exceed it. `unscannable` decides what happens then: | | | |---|---| | `block` *(default)* | Refuse with **413**. A rule asked for this traffic to be scanned, and forwarding it unscanned is a hole that never announces itself. | | `pass` | Forward it unscanned, logging each time. Choose this where oversized uploads must keep working and you accept that they reach the backend uninspected. | `pass` has a second effect worth knowing: it stops the proxy reading a chunked body up front, because bytes read past the cap cannot be put back on the wire. ## Memory, and why the ceiling is not a refusal Containment means holding a body in memory until ClamAV answers, so `max_file_size` times the number of simultaneous uploads is an out-of-memory condition waiting to happen. `scan_memory_budget` (256 MB by default) caps the total held across all in-flight scans. Crossing it does **not** refuse the request. The scan falls back to streaming — the client is still refused on a hit, but the bytes reach the backend while the scan runs. :::note[That asymmetry is deliberate] `unscannable` is policy: you asked for a guarantee and it could not be provided, so the default is to refuse. The memory ceiling is load shedding: refusing traffic because the proxy is briefly busy would turn memory pressure into an outage. They are separate settings because they answer different questions. ::: ## Verdict cache A verdict is reused for an identical body for `verdict_cache_ttl_secs` (300 by default), so re-uploading the same artifact skips the ClamAV round trip. `verdict_cache_entries` (8192) bounds how many are held. The TTL exists because a *clean* verdict is only as good as the signature database behind it — it caps how long one can outlive a freshclam update. Failed scans are never cached. Set the TTL to `0` to disable. ## Configuration ```yaml proxy: content_scanning: enabled: true clamav_server: "localhost:3310" max_file_size: 10485760 unscannable: block scan_memory_budget: 268435456 verdict_cache_ttl_secs: 300 verdict_cache_entries: 8192 ``` :::caution[Three keys were removed — and they still parse] `scan_expression`, `scan_content_types` and `skip_extensions` are gone. Selecting traffic is the `content_scanning` rule action's job now. The config struct does not reject unknown fields, so leaving them in place is silent: they parse, and they do nothing. If your config carries them, delete them and write the rule. ::: ## Limits worth knowing - **Request bodies only.** Responses from your application are not scanned. - **Signature-based.** It catches known malware, not a payload written for you. Pair it with the [machine learning models](./machine-learning.mdx). - **It needs a ClamAV daemon you run and keep updated.** Detection is only as current as its signatures. - **It costs latency proportional to body size,** plus a round trip — on the traffic your rule selected. - **Encrypted or archived content limits what a scanner sees.** Archive contents are inspected; a password-protected archive is opaque. :::danger[Scanning fails open when the scanner is unreachable] If the scan errors or ClamAV cannot be reached, the request **proceeds**, and the failure is logged. That keeps an outage in the scanner from becoming an outage in your application, and it also means a scanner that is quietly down stops protecting you while everything looks healthy. Monitor scanner health and alert on scan failures rather than treating silence as clean. This is distinct from `unscannable`, which covers a body that was reachable but too large. ::: ## See also - [WAF](./waf.mdx) — the rule engine, and the `content_scanning` action - [Block malware in uploads](../use-cases/block-malware-uploads.mdx) — the worked scenario - [Configuration](../client/configuration.mdx) — every key - [Security Event Export](./event-export.mdx) — the record a detection produces --- # Security Event Export Source: https://docs.gen0sec.com/docs/features/event-export # Security Event Export Synapse decides things at several layers — a packet dropped in the kernel, a request blocked by the WAF, a source flagged by threat intelligence, an IDS signature matched. Each of those is a security event, and each is emitted through **one** unified stream. That matters because the alternative is correlating seven log formats to answer "why was this client blocked?". ## One event model, whatever decided Every layer that makes a decision emits the same structured event: | Layer | Decides | |---|---| | Access rules | Address blocked in the kernel | | Smart firewall | Fingerprint-driven kernel block | | IDS | Signature matched | | WAF | HTTP request blocked or challenged | | Threat intel | Source reputation acted on | | CAPTCHA | Challenge issued or passed | | Rate limit | Client throttled | Each event carries the layer that decided and the action taken, so a single query answers "everything we blocked, and why" without joining across sources. ## Where it can go | Destination | Format | Reach for it when | |---|---|---| | Local event log | Human-readable lines | You are on the box, tailing | | JSON Lines file | One self-contained record per line | A shipper — Fluent Bit, Vector, an OTel Collector `filelog` receiver — is already running | | OTLP over HTTP | Batched OpenTelemetry log records **and metrics** | You have a collector and want push, not scrape | | CEF | ArcSight Common Event Format | Your SIEM is ArcSight, QRadar, or anything that ingests CEF natively | | syslog | Standard syslog, with facility and identifier | Central syslog is the collection point | On Windows, events can additionally go to ETW and the Windows Event Log. OTLP carries two signals, not one: the event stream as log records, and a set of counters, gauges and a histogram as metrics — so the same collector gets both "what was blocked" and "how much is being blocked" without a second exporter. See [OpenTelemetry](../observability/opentelemetry.mdx). CEF carries three streams, not just one: security decisions, IDS alerts, and access logs — so a SIEM gets the whole picture in the format it already parses, rather than a decision feed it has to reconcile with logs from elsewhere. ## Try it ### What a decision looks like Every layer emits the same record, so one line tells you who enforced, what fired it, and against whom: ```json { "ts": "2026-09-08T11:04:22.481Z", "layer": "smart_firewall", "action": "drop", "source": "fingerprint_rule", "signals": ["fingerprint_rule", "threat_intel"], "rule_id": "sf-scanner-ja4t-01", "rule_msg": "known scanner TCP fingerprint", "hits": 47, "ban_secs": 3600, "src_ip": "203.0.113.42", "src_port": 54321, "dst_ip": "10.0.0.5", "dst_port": 443, "src_country": "NL", "fingerprints": { "ja4t": "t64320_2_1-3-8-nop_nop,sackOK,ts,nop,ws,eol_", "ja4": "t13d311200_e8f1e7e78f70_d339722ba4af" } } ``` Three fields carry most of the meaning, and they are not interchangeable: | Field | Answers | Values | |---|---|---| | `layer` | **Who enforced** | `access_rules`, `smart_firewall`, `ids`, `waf` | | `action` | **What was done** | `block`, `drop`, `allow`, `log`, `notice`, `ratelimit`, `captcha` | | `source` | **What decided** | `threat_intel`, `classifier`, `traffic_classifier`, `ids`, `waf`, `fingerprint_rule`, `geoip`, `cidr`, `behavioral`, `microseg`, `access_rules` | `signals` is the one people miss. A rule that fuses two families — an IDS score *and* a threat-intel verdict — lists every contributing signal, with `source` being the first. A scalar `source` cannot represent fusion, so if you are attributing blocks to a cause, read `signals`, not `source` alone. `ban_secs` is the duration declared **at install**, not a countdown. Two events for the same address both report the full duration. ### Answer "why was this blocked?" ```bash # Everything we dropped in the last run, newest first jq -c 'select(.action == "drop") | {ts, layer, source, src_ip, rule_msg}' \ /var/log/synapse/events.log | tail -20 # One address, across every layer that touched it jq -c 'select(.src_ip == "203.0.113.42")' /var/log/synapse/events.log # Which sources are driving enforcement jq -r '.source // "unknown"' /var/log/synapse/events.log | sort | uniq -c | sort -rn ``` That last one is the useful daily query — a sudden shift in which `source` dominates is usually a feed change or a misfiring rule, not an attack. ### Ship it to a SIEM as CEF CEF is configured under `logging.cef`, and carries three independent streams. Only the first is on by default, which is deliberate: block events are per decision, IDS alerts per signature match, and access logs per request, so the third is by far the largest. ```yaml # /etc/synapse/config.yaml logging: cef: enabled: true file: "/var/log/synapse/cef.log" max_size: 104857600 # rotate at 100 MB keep: 10 streams: block_events: true # security decisions — the primary stream ids_alerts: true # including alert-only signatures that never block access_logs: false # one record per proxied request; leave off unless metered # dvchost on every record, so a SIEM can attribute events to an agent. # Defaults to the system hostname. device_host: "" ``` Leave `file` empty to disable the file sink and deliver over syslog only. :::caution[`access_logs` is a volume decision, not a detail] Block events fire per decision; access logs fire per request. On a busy proxy that is orders of magnitude more records, and most SIEM licensing is metered by ingest. Turn it on deliberately. ::: ### Structured fingerprints, separately Decisions and observations are different streams. The unified fingerprint log carries every JA4+ and BPF event, whether or not anything was enforced: ```yaml logging: eventbridge_log: enabled: true # the default # file: /var/log/synapse/eventbridge.log ``` `fingerprint_log` is an accepted alias for `eventbridge_log`. With `file` unset the path is `{log_directory}/eventbridge.log`. It is **not** `fingerprints.log`. That name survives in source comments and in the shipped `config/config.yaml`, which still gives it as the default, but nothing writes a file by that name — it is the stream's old name, and this page carried it over. :::caution[The six per-type sections are already gone, not merely deprecated] `bpf_stats`, `tcp_fingerprint`, `ssh_fingerprint`, `latency_fingerprint`, `tls_fingerprint` and `http_fingerprint` are **absent from the 0.8.8 schema**. Setting any of them has no effect: the keys land in the `Unused config options in ` warning at startup and nothing else happens. The shipped `config/config.yaml` still sets **all six**, so a host built from the sample we ship emits that warning without the operator having typed anything wrong. ::: ## Why both a readable log and a structured one They serve different readers, and neither substitutes for the other. The readable log is for a person debugging at 3am who wants to see what just happened. The structured stream is for machines — stable field names, one record per line, no parsing heuristics. Both are fed from the same event, so they cannot disagree about what occurred. ## Limits worth knowing - **Export is push or file, not a query API.** Synapse emits; it does not store a searchable history. Retention and search are your collector's job. - **OTLP batches.** Events arrive at the collector in batches rather than one at a time, so there is a small delay between decision and visibility. - **A shipper is still your responsibility.** The JSON Lines sink writes a file; something has to move it. ## Use cases - **Answer "why was this blocked?" from one query**, instead of correlating kernel, proxy and IDS logs by timestamp. - **Feed an existing SIEM without a translation layer**, by emitting CEF it already understands. - **Alert on the decision rate** — a sudden spike in blocks is a signal in its own right. - **Keep an audit trail** of enforcement actions, with the layer and reason attached to each. ## See also - [Fleet Bans](./fleet-bans.mdx) — the same decisions, aggregated across the deployment - [OpenTelemetry](../observability/opentelemetry.mdx) — OTLP export of both logs and metrics, and the config precedence - [Axiom](../observability/axiom.mdx) and [Honeycomb](../observability/honeycomb.mdx) — worked examples for specific backends - [Configuration](../client/configuration.mdx) — sinks, paths, endpoints and levels - [Threat Detection](./threat-detection.mdx) — where the reputation half of these events comes from - [WAF](./waf.mdx) — where the HTTP half comes from --- # EventBridge Source: https://docs.gen0sec.com/docs/features/eventbridge # EventBridge EventBridge is the local interface into a running Synapse agent. Your own software can subscribe to what the agent is seeing, or ask it a direct question about one connection and get an answer back. That closes a gap nothing else does. Your application knows the socket it is serving. It does not know the client's TLS fingerprint, its SNI, or how its TCP stack behaved — the agent does, and this is how you get it without implementing any capture yourself. ## Two interfaces | | What it is | Reach for it when | |---|---|---| | **Event stream** | Events pushed as they happen — HTTP requests and packets, with their fingerprints | You want a live feed to consume or display | | **Control socket** | A request/response query about a specific connection | You have a connection in hand and want to know what the agent saw | Both are **local sockets on the host** — a Unix domain socket on Linux and macOS, a loopback address on Windows. Neither is exposed to the network, so the interface is available to processes on the box and to nothing else. ## Asking about a connection You identify a connection by its source address, optionally narrowing with source port, destination address and port, or a fingerprint you already hold. The agent answers from its cache of recent connections. What comes back is everything it captured for that connection: | | | |---|---| | **TLS** | JA4 and JA4S, plus the raw JA4 string | | **TCP** | JA4T and JA4TS, with the hash form | | **Latency** | JA4L and JA4LS | | **HTTP** | JA4H | | **Certificate** | JA4X | | **Handshake detail** | SNI, ALPN, and the negotiated TLS version | | **Context** | The connection four-tuple and when it was captured | You can ask for a summary or the full event detail. The SNI and ALPN are worth calling out. Your application sees the request; it does not see what hostname the client asked for at the TLS layer, or what protocol it offered. Those are frequently the interesting part of a mismatch. ## Try it Both sockets are off unless you give them a path. Set them, and restart: ```yaml # /etc/synapse/config.yaml daemon: control_socket: "/var/run/synapse-control.sock" event_socket: "/var/run/synapse-events.sock" ``` On Windows these are loopback TCP instead — `127.0.0.1:19198` for control and `127.0.0.1:19199` for events. ### Ask about one connection The control socket speaks HTTP/1.1 with a single endpoint, `POST /query`, so `curl` is enough to test it: ```bash curl --unix-socket /var/run/synapse-control.sock \ -X POST http://localhost/query \ -H 'Content-Type: application/json' \ -d '{"src_ip":"203.0.113.42","src_port":54321}' ``` ```json { "status": "found", "event": { "src_ip": "203.0.113.42", "src_port": 54321, "dst_ip": "10.0.0.5", "dst_port": 443, "ja4": "t13d311200_e8f1e7e78f70_d339722ba4af", "ja4t": "t64320_2_1-3-8-nop_nop,sackOK,ts,nop,ws,eol_", "ja4s": "t301600_c02bc02f002fc030_h2,http/1.1_", "sni": "example.com", "alpn": "h2", "tls_version": "TLS 1.3" } } ``` Only fields actually captured for that connection appear — absent ones are omitted rather than returned as `null`. A connection the cache has evicted returns `404` with `{"status":"not_found"}`. `src_ip` and `src_port` are required. Narrow the match with `dst_ip`, `dst_port`, `ja4` or `ja4t` when you need to. :::note[Behind CGNAT, ask by address alone] Set `"aggregate": true` and drop `src_port` to get every entry seen under that address: the response becomes `{"status":"found_many","events":[…]}`. Combine it with `expect_ja4_prefix` to keep only one fingerprint family. ::: ### Enrich a request in your own handler The point of the query interface is that your application already knows the socket it is serving, and can ask what the agent saw on it: ```python import http.client, json, socket class UnixConn(http.client.HTTPConnection): def connect(self): self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) self.sock.connect("/var/run/synapse-control.sock") def fingerprint(src_ip: str, src_port: int) -> dict | None: c = UnixConn("localhost") c.request("POST", "/query", json.dumps({"src_ip": src_ip, "src_port": src_port}), {"Content-Type": "application/json"}) body = json.loads(c.getresponse().read()) return body.get("event") if body.get("status") == "found" else None # In a request handler, from the peer address you already have: fp = fingerprint("203.0.113.42", 54321) if fp: log.info("client ja4=%s sni=%s alpn=%s", fp.get("ja4"), fp.get("sni"), fp.get("alpn")) ``` The SNI and ALPN are the part worth having: your application sees the request, but not what hostname the client asked for at the TLS layer or which protocols it offered. ### Watch the stream The event socket needs no handshake — connect and events start arriving as newline-delimited JSON, one object per line: ```bash socat -u UNIX-CONNECT:/var/run/synapse-events.sock - \ | jq -c 'select(.type == "Packet") | {src_ip, ja4, sni}' ``` ```json {"src_ip":"203.0.113.42","ja4":"t13d311200_e8f1e7e78f70_d339722ba4af","sni":"example.com"} {"src_ip":"198.51.100.7","ja4":"t13d1516h2_8daaf6152771_b0da82dd1658","sni":"api.example.com"} ``` Two event types are emitted: `Packet` for TCP/TLS connection fingerprints, and `Http` for request fingerprints. In agent mode you will see `Packet` almost exclusively, because `Http` needs the proxy to have parsed a request. Up to 64 clients can be connected at once. ## What you can build with it - **Enrich a request in your own handler** — look up the connection you are serving and attach the client's fingerprint to your own logs or risk scoring. - **Step up authentication on an unfamiliar client** by treating the fingerprint as one factor. - **Confirm during an incident** that the connection you are looking at is the one you think it is, rather than inferring from timestamps. - **Build your own view** on the event stream — the terminal client that ships with Synapse is itself a consumer of this interface. ## Limits worth knowing - **The cache is bounded and recent.** A connection old enough to have been evicted returns nothing. This is a live interface, not a history — for anything retrospective use [Security Event Export](./event-export.mdx). - **Local only, by design.** There is no network listener to secure, and equally no remote access; anything off-host has to go through the export path instead. - **Answers are best-effort.** Fields are absent rather than wrong when a view was never captured — a connection that never completed a TLS handshake has no JA4, and says so by omitting it. ## See also - [Live Traffic View](./live-traffic.mdx) — the terminal client built on this interface - [Security Event Export](./event-export.mdx) — the durable, off-host counterpart - [JA4+](../client/ja4-plus.mdx) — what each fingerprint in the answer means - [Configuration](../client/configuration.mdx) — socket paths and enablement --- # Firewall Rules Source: https://docs.gen0sec.com/docs/features/firewall-rules # Firewall Rules Firewall rules filter traffic in **both directions** on address, port and protocol. Where [Access Rules](./access-rules.mdx) answer *is this address allowed?*, firewall rules answer *is this conversation allowed?* — and they can stop a host from reaching out, not just from being reached. Enforcement is stateless and runs in the kernel at the TC hook, so nothing sits in the application's path. ## What it gives you - **Egress control.** Stop a compromised workload calling home, without an outbound proxy. - **Port and protocol matching**, including port ranges, on top of address and prefix. - **Ordered rules with an explicit default.** You decide what happens to traffic no rule mentions. - **Separate IPv4 and IPv6 handling**, so a rule cannot accidentally span both. ## How it works 1. A packet is inspected at the TC hook — ingress or egress. 2. Rules are evaluated in order, lowest `order` value first. 3. **The first matching rule wins**, and the packet is allowed or dropped accordingly. 4. If nothing matches, the configured default action applies. :::caution[Order decides, so put exceptions first] Evaluation stops at the first match. An `allow` placed below a broad `drop` never runs — the drop has already decided. Give exceptions a lower `order` value than the rules they are meant to escape. ::: ## What a rule matches on | Dimension | Detail | |---|---| | Address | Source and destination, each with a CIDR prefix length | | Port | Source and destination ranges, low and high bounds; unset means any | | Protocol | IP protocol number — TCP, UDP, or any | | IP version | IPv4 or IPv6, chosen per rule | | TCP flags | A mask and the value it must equal — see below | | Direction | `ingress` or `egress` | | Action | `allow` or `drop` | Only active rules are loaded into the kernel, so disabling a rule takes it out of the datapath rather than leaving it inert but resident. ### TCP flags Flag matching is expressed as **`flags & mask == value`** — you name the flags you care about, and what they must be. That is the same primitive nftables and iptables use natively, so a rule means the same thing whichever backend enforces it. | You want | Mask | Value | |---|---|---| | SYN set, nothing else constrained | `syn` | `syn` | | An ACK that is not a SYN-ACK | `syn,ack` | `ack` | | SYN without ACK — a connection attempt | `syn,ack` | `syn` | | No RST | `rst` | *(empty)* | :::note[A bare flag list is rejected on purpose] `tcp_flags: "ack"` has three defensible readings, and firewall vendors genuinely disagree about which they mean: ACK set with everything else unconstrained, ACK set and nothing else, or any of the listed flags set. For one flag the readings coincide; for a list they diverge completely — under the third reading `syn,ack,fin,rst,psh,urg` matches *every* TCP packet, and under the second it matches essentially none. Rather than pick a convention and let you discover it in production, the mask must be stated. It is one extra field and it removes the ambiguity entirely. ::: ## Firewall rules or access rules? | | Access Rules | Firewall Rules | |---|---|---| | Hook | XDP, before routing | TC, after ingress and before egress | | Direction | Inbound only | **Inbound and outbound** | | Matches | IP address | IP, port range, protocol | | Cost | The cheapest drop available | Slightly higher, still in-kernel | | Reach for it when | Absorbing a known-bad address at volume | Expressing a specific allowed conversation | Both are enforced by [Hillock](../components/hillock/) and both are driven by decisions [Amygdala](../components/amygdala/) makes. Neither is a substitute for the other: address blocking is cheaper, conversation control is more precise. ## One rule, whichever backend runs it Rules are written once and enforced by whatever the host can actually offer. Synapse picks the best available and falls back rather than failing: **XDP / eBPF → nftables → iptables** A modern kernel gets in-kernel eBPF enforcement. A host without it falls back to nftables, and one without that to iptables. The rule you wrote does not change — only the machinery underneath it does, which is why the same policy can cover a fleet that is not uniform. :::note[Windows firewall is coming] The intent is one firewall platform across Linux and Windows, with the same rule model on both. The Windows firewall backend is still in development — today, plan Windows enforcement as a roadmap item rather than something to build policy on. ::: ## Limits worth knowing - **Stateless.** Rules match each packet on its own; there is no connection tracking, so a rule permitting a reply must exist in its own right. - **No application awareness.** Matching stops at the transport header. For TLS SNI, HTTP paths or fingerprints, use [Amygdala](../components/amygdala/) — it evaluates those on the host and installs the resulting block here. ## Use cases - **Deny egress by default** and allow only the destinations a workload legitimately needs. - **Contain a compromised host** by cutting its outbound paths while leaving inbound monitoring intact. - **Restrict a management port** to one source range without touching the service itself. - **Prove an allow-list before enforcing it** by ordering a permissive rule above a broad deny, then removing it. ## See also - [Smart Firewall](./smart-firewall.mdx) — fingerprint-driven blocks, installed into this same kernel firewall - [Access Rules](./access-rules.mdx) — faster, address-only, inbound - [Configuration](../client/configuration.mdx) — the agent's firewall settings - [Hillock](../components/hillock/) — the kernel firewall that enforces these - [Amygdala](../components/amygdala/) — evaluates richer conditions and installs the resulting blocks --- # Fleet Bans Source: https://docs.gen0sec.com/docs/features/fleet-bans # Fleet Bans One agent blocking an address is an observation. Forty agents blocking the same address is a different kind of fact — and the second one is invisible to any agent looking only at its own traffic. Fleet bans close that gap. Each agent reports what it decided to block, the platform corroborates across the deployment, and every agent pulls back the aggregated result. :::caution[Advisory by default] Out of the box a fleet ban does **not** install a block. The set is consulted, the outcome counted and logged, and traffic is unaffected. That is deliberate. Acting on what other machines concluded is a large step, and it should be justified by the false-positive rate that advisory mode exists to measure. Run it, read the numbers, then turn on [enforcement](#enforcing-the-set) for the agents where they hold up. ::: ## Two halves of one loop | | Direction | Config | Scope | |---|---|---|---| | **Ban ledger** | Upload — what *this* agent decided to block | `platform.ban_ledger` | `fleet_bans:write` | | **Fleet bans** | Download — what the *fleet* corroborated | `platform.fleet_bans` | `fleet_bans:read` | They are separate stores with opposite provenance, and kept separate on purpose. The ledger holds first-party evidence — decisions this agent made from its own events. The fleet set holds bans this agent has no local evidence for. Merging them is how a fleet ban gets re-reported upward as fresh evidence and corroboration becomes an echo of itself. ## What a fleet ban carries | Field | | |---|---| | `agents` | Distinct agents that reported banning this address | | `agent_count_total` | Live agents in the workspace — the denominator | | `confidence` | Platform confidence, 0–100 | | `score` | Platform score, 0–99, directly comparable to a threat-MMDB score | | `first_banned_unix` | When the fleet first banned this address | | `expires_unix` | When the ban lapses; absent means no expiry | | `layers` | Bitmask of the enforcing layers that reported it | **`agent_count_total` is the field that makes the rest readable.** `agents: 3` is unanimity in a three-agent fleet and noise in a three-thousand-agent one. Any threshold you write should be a ratio, not a count. The layer bitmask records *what kind* of agreement it is — `access_rules`, `smart_firewall`, `ids`, `waf`, `threat_intel`, `captcha`, `rate_limit`. Three agents agreeing via three different layers is stronger corroboration than three agreeing via the same one, which is usually the same rule firing three times. ## The platform is not trusted Both local safety predicates are re-applied to everything the platform serves, and anything failing either is dropped regardless of what arrived: | Predicate | Drops | |---|---| | Ban-safe | Anything not unambiguously global-unicast. Fails closed. | | Never-ban | Your own infrastructure — HOME_NET, and the node's own IPs | The never-ban check is the one that matters most in practice. A public load balancer VIP is global-unicast, so the first predicate would permit it; banning it blackholes every client behind it. It is protected through HOME_NET: in Kubernetes the Synapse operator puts the VIPs there, and elsewhere you list them in `ids.address_vars.HOME_NET` yourself. The asymmetry is intentional: the agent is **permissive about what it uploads** — an unsafe ban is worth reporting so the fleet can see it — and **strict about what it accepts**. Rejections are counted separately as `rejected_unsafe`, `rejected_protected` and `rejected_malformed`, so a set that is arriving but being discarded is visible rather than looking like an empty fleet. A malformed record is skipped and counted rather than failing the whole payload. ## Configuration ```yaml platform: ban_ledger: enabled: true interval_secs: 60 fleet_bans: enabled: true refresh_secs: 60 advisory: true advisory_dedup_secs: 300 enforce: false ``` Both inherit `platform.base_url`, so neither normally states a URL. `interval_secs` governs how fast a ban **propagates**, not how much traffic this generates: a tick whose snapshot has not changed sends nothing at all, and a forced resend happens only every twelfth suppressed tick. `refresh_secs` is a cheap version check — the artifact is downloaded only when the version changes. The shipped configuration sets both to 60 seconds (left out, each defaults to 300), so the loop is symmetric rather than having a slow leg. Raise `refresh_secs` if a large fleet makes the version endpoint hot. ## Enforcing the set With `enforce: true` an agent stops only advising and blocks what the fleet corroborated: - **In the kernel** (Linux, unless `firewall.mode: none`). Each fleet ban goes into the same IP filter local bans use, and comes out again when its deadline passes, when the fleet withdraws it, when your own configuration starts to allow the address, or when it no longer ranks within the fleet's share. While the agent runs, a new set reaches the kernel as soon as it is downloaded, and a ban whose deadline passes comes out within about half a minute. - **At the proxy.** Behind a load balancer the kernel sees only the balancer's address, so a kernel ban on the client never matches. With `proxy.protocol.enabled: true` and a balancer that sends PROXY protocol headers, the proxy knows the real client and refuses a fleet-banned one with `403` and `X-Block-Reason: fleet_ban`. It does not use `X-Forwarded-For` for this. The proxy refuses every enforceable fleet ban, whether or not the kernel holds it. What is never enforced, whatever the set says: - **An address you allow.** `access_rules.allow` beats a fleet ban, including when the allowed entry is a wider range than the banned address. - **Your own infrastructure.** The same never-ban checks as above — HOME_NET, which is where load-balancer VIPs belong, and the node's own IPs — and, on Unix, any address the host holds on any interface. - **An address your configuration already blocks.** A configured `access_rules.block` entry covering it drops it anyway. On nftables the fleet's entry would even keep that range from being installed, so a held fleet ban a configured range comes to cover is lifted for it. - **A ban without a deadline.** Every fleet ban the platform publishes expires unless agents keep reporting it; a ban that does not is never installed. - **More than its share, in the kernel.** Fleet bans take at most a quarter of each address family's IP filter (`firewall.max_ip_filter_entries_v4` / `_v6`), leaving at least three quarters for this agent's own bans and rules. When the set is larger, the most widely corroborated bans win the kernel's quarter, and the proxy still refuses the rest. The quarter is of the filter the kernel actually has: a resized filter takes effect only once the agent's pinned BPF state is removed. A fleet ban and a local ban on the same address do not interfere: lifting either leaves the other in place. **An enforced fleet ban is never reported as this agent's own.** The ban ledger leaves it out, so it is never uploaded, and the platform ignores it when corroborating. Without that, one agent's ban would come back as every other agent's evidence and never expire. Enforcement is Linux-only in the kernel; the proxy refusal works wherever the proxy runs. Each install is a security event with source `fleet_ban` and rule id `fleet_bans.enforce`, and so is an address's first refusal at the proxy in each `advisory_dedup_secs` window (at most 50 such events a second); every refusal is counted. Each lift is a `synapse.fleet_ban.lifted` OpenTelemetry log record, which your own collector keeps (the platform stores only block events). The `synapse.fleet_bans.*` metrics report the set's size and age, what is enforced, and what was skipped and why. ## Limits worth knowing - **Advisory means advisory.** Consulted, counted, logged. No traffic is affected until enforcement is turned on. - **An unloaded set is not an empty one.** A miss before the first set has been published means *we do not know yet*, not *not banned*. Enforcement never lifts a ban for that reason: until a set arrives, only bans past their own deadline come out. - **A stopped agent lifts nothing.** While it is down, what it installed stays in the filter, past its deadline if need be; only nftables expires an entry by itself, within the hour. The agent records what it installed, and on the next start it lifts whatever the fleet withdrew or what expired in the meantime. - **Turning enforcement off takes a restart.** `enforce` is read at startup. The first start with it off lifts everything the agent installed. - **A ban the kernel will not take waits.** An address the IP filter refuses (a full filter, for one) is tried again after five minutes. After three refusals in one address family, the rest of that family waits for the next pass; bans already held, and the other family, are unaffected. - **Corroboration is not proof.** Agents in one deployment often run one policy, so agreement can mean the same rule fired everywhere rather than independent confirmation. Read the layer mask before treating a count as consensus. - **It needs both scopes.** `fleet_bans:write` to contribute, `fleet_bans:read` to receive. An agent with only read is a passenger. ## Use cases - **Measure before enforcing** — run advisory for a period and read the false-positive rate against your own traffic, then enforce where it holds up. - **Close the window on a scanner** — once enforcing, an address the fleet agrees on is blocked on agents it has not reached yet. - **Spread a detection across a fleet** — one agent seeing a scanner means every agent knows about it within about a minute. - **Corroborate a marginal signal** — a source your local threshold nearly caught is easier to judge when forty other agents already did. - **Spot a bad rule** — a sudden spike in reported bans across one layer is usually a rule, not an attack. ## See also - [Access Rules](./access-rules.mdx) and [Smart Firewall](./smart-firewall.mdx) — layers that report bans - [Threat Detection](./threat-detection.mdx) — the platform-wide feed, distinct from your own fleet's decisions - [Security Event Export](./event-export.mdx) — the local record of what this agent blocked - [Configuration](../client/configuration.mdx) — the full `platform.ban_ledger` and `platform.fleet_bans` blocks --- # Live Traffic View Source: https://docs.gen0sec.com/docs/features/live-traffic # Live Traffic View Logs tell you what happened. This tells you what is happening — a terminal view of fingerprints, traffic and threat data as the agent sees them, and a way to ask a running agent about one specific connection. It is the fastest way to answer "is this thing actually working?" during a deployment, and "what is that client?" during an incident. ## The terminal view Synapse ships an interactive terminal interface. It renders live JA4+ fingerprints, traffic, and threat context — no dashboard, no browser, no shipping logs somewhere first. There are two ways to run it: | Mode | Command | What it does | |---|---|---| | **Embedded** | `synapse --mode agent --terminal` | The view runs in the same process as the agent | | **Standalone** | `synapse --terminal` | Connects to an already-running daemon over a local socket | Standalone is the one you usually want in production: the agent keeps running untouched, and you attach a viewer to it and detach again. Embedded suits a laptop or a test box where you are starting the agent anyway. The transport differs by platform — a Unix domain socket on Linux and macOS, a loopback TCP address on Windows — but the view is the same. ## Asking about one connection The same interface answers direct questions: give a running agent a connection's four-tuple and it returns the fingerprints, SNI, ALPN and TLS version it captured for it. That is a programmatic interface rather than a view, so it has its own page — see [EventBridge](./eventbridge.mdx). ## Limits worth knowing - **It is a live view, not history.** The terminal shows what is happening now, and the query answers from a cache of recent connections. For anything retrospective, use [Security Event Export](./event-export.mdx). - **The cache is bounded.** A connection old enough to have been evicted will not be found. - **Attaching a viewer is an operator action.** It is not a substitute for monitoring — nobody is watching a terminal at 3am. ## Use cases - **Verify a deployment in seconds** — start the view, generate traffic, watch fingerprints appear. - **Watch a rule take effect** rather than waiting for a log pipeline to confirm it. - **Demonstrate what the agent sees** to someone who does not believe fingerprinting works. - **Debug a client you cannot reproduce** by watching it live instead of reconstructing it afterwards. ## See also - [Security Event Export](./event-export.mdx) — the durable, machine-readable counterpart - [Configuration](../client/configuration.mdx) — socket paths and terminal mode settings - [JA4+](../client/ja4-plus.mdx) — what the fingerprints in the view mean - [Dendrite](../components/dendrite/) — the sensor producing them --- # Load Balancing & Reverse Proxy Source: https://docs.gen0sec.com/docs/features/load-balancing # Load Balancing & Reverse Proxy Synapse is a reverse proxy before it is anything else. Traffic terminates at it, is inspected, and is forwarded to whichever of your backends is healthy — and the security features are not bolted on beside that path, they *are* that path. The practical consequence: you do not run a proxy and a WAF and a fingerprinting layer. You run one hop. ## What it gives you - **Balancing across healthy backends**, with unhealthy ones taken out of rotation automatically. - **Sticky sessions** when a client needs to keep landing on the same backend. - **Client fingerprints forwarded upstream**, so your application can see *who* connected, not just from where. - **Layered headers and timeouts** — set a default once, override per host, override again per route. - **HTTP/2 and gRPC**, including cleartext, for backends that expect it. ## How traffic reaches a backend 1. A connection arrives on the HTTP or TLS listener. 2. The host decides how it is handled — Synapse either terminates TLS and inspects the request, or passes the connection straight through untouched. 3. For terminated traffic, security runs here: [threat detection](./threat-detection.mdx) enriches it, the [WAF](./waf.mdx) evaluates it, and a `content_scanning` rule can send the body to the [scanner](./content-scanning.mdx). 4. A healthy upstream is chosen and the request is forwarded. ### Terminate or pass through | Mode | What happens | Use it when | |---|---|---| | **Terminate** | Synapse holds the certificate, decrypts, inspects, then forwards | You want the WAF, content scanning, or HTTP-level rules | | **Passthrough** | The connection is forwarded encrypted, by SNI, without decryption | The backend must own TLS, or you cannot hold the key | Passthrough is chosen per host, and matches wildcards the way DNS does — one label deep, with exact names always winning over a wildcard. Inspection needs termination. A passthrough host gets connection-level controls only; there is no HTTP request to apply a WAF rule to. ## Health checking Backends are probed on an interval and removed from rotation when they stop answering. The probe method is yours to choose — a `HEAD` is cheapest, a `GET` exercises more of the path, and the right answer depends on what your backend does on each. A backend that fails its probe stops receiving traffic without you intervening, and returns when it recovers. ## Sticky sessions Off by default, because most services do not need it and it costs you even distribution. When enabled, a client is consistently hashed to the same backend across reconnects — useful for a backend holding per-session state that is not shared. Prefer shared state when you can. Stickiness makes one backend's failure visible to the specific users pinned to it. ## Fingerprints forwarded to your application This is the part that is hard to get elsewhere. The proxy forwards the client's JA4+ fingerprints upstream as request headers, so your own code can act on them: `X-JA4` · `X-JA4T` · `X-JA4H` · `X-JA4S` · `X-JA4L` · `X-JA4X` Your application can then treat a client's TLS or HTTP fingerprint as an input — rate-limit by it, require step-up authentication on an unfamiliar one, or record it alongside a transaction — without implementing any of the capture itself. See [JA4+](../client/ja4-plus.mdx) for what each covers. Forwarding is on by default and can be turned off if you would rather your backends never see it. ## Layered configuration Headers and timeouts resolve global → host → route, each layer extending the one above rather than replacing it. Set a security header once at the top and it applies everywhere; override a slow endpoint's read timeout at the route without restating the rest. ## Limits worth knowing - **Inspection requires termination.** A passthrough host cannot be WAF'd, scanned, or matched on HTTP fields. - **Health checks prove reachability, not correctness.** A backend answering `200` on `HEAD /` while failing real requests stays in rotation. - **Stickiness is by client address.** Clients behind a shared egress hash together, and a client whose address changes moves backend. ## Use cases - **Put one hop in front of a service** and get balancing, TLS, WAF and fingerprinting from it. - **Drain a backend for maintenance** by letting it fail its health check, with no config change. - **Give your application client identity it could not otherwise compute**, through forwarded fingerprint headers. - **Front a backend that must terminate its own TLS** by passing the connection through while keeping connection-level controls. ## See also - [Configuration](../client/configuration.mdx) — listeners, upstreams, timeouts and health checks - [Kubernetes Ingress](../client/kubernetes-ingress.mdx) — the same proxy as an Ingress and Gateway API controller - [WAF](./waf.mdx) — what runs on terminated traffic - [JA4+](../client/ja4-plus.mdx) — the fingerprints forwarded upstream --- # Machine Learning Models Source: https://docs.gen0sec.com/docs/features/machine-learning # Machine Learning Models Gen0Sec ships small, purpose-built models rather than one general classifier. Each answers a narrow question about traffic the agent is already fingerprinting, and each runs on the host — no traffic leaves the machine to be scored. This page is the summary. Every model has a fuller card covering training data, evaluation and limitations; read the card for any model you intend to make a policy decision on. ## What ships | Model | Question | Input → output | |---|---|---| | **Fingerprint classifier** | Is this client's JA4+ fingerprint malicious? | Fingerprint features → `benign` / `malicious`, with a probability | | **Flow classifier** | Does this flow behave maliciously? | Flow features → `benign` / `malicious`, with probabilities | | **Traffic classifier** | What *kind* of traffic is this? | Fused signals → one of seven classes | | **Severity model** | How serious is this, and for how long? | Per-source context → a bounded severity score | | **JA4-JEPA embedder** | What is this fingerprint *near*? | Fingerprint → an embedding, no verdict | All of them are small networks or trees exported to ONNX, executed on CPU, with preprocessing baked into the graph so the agent and the training pipeline cannot drift apart on feature scaling. Models arrive with a signed integrity manifest and are hot-swapped at runtime — no restart, no dropped traffic. ## The fingerprint classifier, in detail The one you are most likely to act on. It takes the JA4+ views the agent already captures — the TLS client and server fingerprints, the TCP handshake pair, and the HTTP fingerprint — encodes them into a fixed feature vector, and returns a binary verdict with a probability. Gradient-boosted trees are the default backend. They are the most accurate of the three tried, and the only one that handles a *missing* fingerprint view natively rather than imputing it to zero — which matters, because a real connection frequently presents only some views. An alternative built on self-supervised embeddings scores within a rounding error of it on the same data, which is a useful signal that the result is a property of the problem rather than of one modelling choice. ### Measured performance On a held-out test set of 3,599 connections, split so that a given fingerprint appears in training or in test but never both: | Metric | Value | |---|---| | Accuracy | 0.9964 | | Precision (malicious) | 0.9925 | | Recall (malicious) | 0.9936 | | ROC-AUC | 0.9999 | Seven false positives and six false negatives out of 3,599. :::warning[Read this before quoting that number] Those figures are a **matched-conditions upper bound, not a prediction of live performance.** In the corpus they were measured on, benign and malicious traffic largely came from *different capture pipelines*. That leaves the classes partly separable by where the traffic was collected rather than by anything about the client. We know exactly how strong that effect is, because we measured it: a model given **only** the five bits saying which fingerprint views are present — no fingerprint content whatsoever — scores 0.9999 on one split and 0.2954 on another. Worse than chance. The shortcut even inverts between splits. The fair split above removes that shortcut and proves the model *can* discriminate on fingerprint content. It does not prove it will hold up on your traffic. For that, benign and malicious samples have to be captured through the same pipeline — ideally the agent's own — so the only thing left to learn is the fingerprint. Treat 0.996 as a ceiling measured under favourable conditions, and validate against your own traffic before setting policy on it. ::: ## How to use the output responsibly - **Not a standalone block.** Use a model verdict as one signal among several — pair it with IDS context, threat reputation, or a second model. [Amygdala](../components/amygdala/) exposes all of them as ordinary rule fields precisely so a rule can require two before acting. - **Set your own threshold.** Probabilities are uncalibrated. The right cut-off depends on how common malicious traffic actually is in your environment, not on 0.5. - **Expect distribution shift.** On deliberately out-of-distribution malware samples the classifier does misclassify some as benign. That is the same capture-source effect, seen from the other side. - **A fingerprint is not a person.** Many legitimate clients share one. Treat it as a strong grouping signal, not proof of identity. ## Where each one runs Inference is Linux-only. [Cortex](../components/cortex/) compiles on other platforms so the rest of the agent builds and tests cross-platform, but the inference entry points return errors there rather than scoring traffic. Of the five, only the fingerprint and flow classifiers can trigger enforcement directly. The traffic classifier is deliberately a parallel signal, and the severity model produces a score that the policy layer — not the model — turns into a ban duration. ## See also - [Smart Firewall](./smart-firewall.mdx) — turn a flagged fingerprint into a kernel block - [Cortex](../components/cortex/) — the engine that runs these models - [Threat Detection](./threat-detection.mdx) — reputation signals to combine them with - [JA4+](../client/ja4-plus.mdx) — the fingerprints they consume - [Amygdala](../components/amygdala/) — where a verdict becomes a decision --- # Service Graph Source: https://docs.gen0sec.com/docs/features/service-graph # Service Graph Most network security watches the edge: what comes in, what goes out. The service graph watches the traffic that never crosses it — service to service, pod to pod, inside your own estate. It answers three questions: 1. **What is actually talking to what?** Observed from real traffic, not from a diagram someone drew. 2. **What is *supposed* to talk to what?** Declared by you. 3. **Where do those two disagree?** That difference is the finding. ## Identity, not addresses In a cluster, an IP address is a poor name for a workload. Pods are rescheduled, addresses are recycled, and a rule written against `10.42.3.17` is wrong the moment something restarts. The service graph resolves traffic back to the workload behind it, so policy can be written against names that survive a restart: | Instead of | Write | |---|---| | `10.42.3.17` | the `payments` workload | | A CIDR that happens to cover a namespace | the `production` namespace | | A pod IP range | a label selector | [Amygdala](../components/amygdala/) exposes these as ordinary rule fields — source and destination namespace, workload, and labels — so a firewall rule can name a service instead of an address. ## Declared edges You declare which services are allowed to reach which. Every observed connection is then classified against that declaration: | | Meaning | |---|---| | **Declared** | This edge is in your policy. Expected traffic | | **Undeclared** | Real traffic with no matching declaration. Either a gap in the policy or something that should not be happening | | **Violation** | Traffic that contradicts the declaration | The classification is available to rules, so a policy violation is not just a dashboard entry — it can be the condition that triggers a block. :::note[Start by watching, not blocking] Declared-edge policy is most useful in the order: observe real traffic first, promote what is legitimate into declarations, then enforce. Writing declarations before you have seen the traffic tends to produce an allow-list that is missing exactly the edges you forgot about. Amygdala's `log` action is built for this — a rule can record what it *would* have blocked without acting, so you can read the consequences before accepting them. ::: ## Enforcement stays on the fast path The graph itself is not consulted per packet. Agents enforce using flat lookups on the packet path; the graph is built and queried out of band, by the platform. That keeps east-west policy enforcement as cheap as any other rule, no matter how large the graph gets. ## Querying it The graph is served by the platform's service-graph API, reachable at `/v1/service-graph` on a Cerebellum deployment, and surfaced in the dashboard. Cross-site correlation — the same workload seen from more than one cluster — is Cerebellum's job rather than the agent's. ## Use cases - **Find the edges nobody documented** — the batch job that quietly reaches the production database, the service still calling a dependency that was supposed to be retired. - **Microsegment without maintaining IP lists.** Declare that only `api` may reach `payments`, and let workload identity keep that true through every reschedule. - **Turn a policy violation into a block** rather than a ticket, once you trust the declarations. - **Scope the blast radius of a compromise** by knowing what a workload could have reached from where it sat. ## See also - [Amygdala](../components/amygdala/) — the rule engine that acts on workload identity and edge classification - [Access Rules](./access-rules.mdx) — address-level filtering, the north-south counterpart - [Architecture overview](../architecture/overview.mdx) — where the platform pieces sit --- # Smart Firewall Source: https://docs.gen0sec.com/docs/features/smart-firewall # Smart Firewall The Smart Firewall blocks by **who the client is**, not by which address it happens to be using today. Rules match JA4+ fingerprints and address expressions, and the result is installed into the kernel firewall — so enforcement costs a kernel map lookup, not a request. It sits between the two things either side of it: | | Matches on | Where | |---|---|---| | [Access rules](./access-rules.mdx) | A static address, CIDR, country or ASN list | XDP, before routing | | **Smart Firewall** | **Fingerprints and expressions over them** | **Kernel, after evaluation** | | [WAF](./waf.mdx) | The HTTP request | In the proxy, after decryption | ## What a rule can match Rules are [wirefilter](./waf.mdx) expressions over the fingerprint scheme — the whole JA4+ family, plus the source address: ```text ja4t == "64240_2-4-8-1-3_1460_10" ja4 in {"t13d3112h2_e8f1e7e78f70_..." "t13d301100_..."} and ip.src in 5.0.0.0/8 ``` Sets use `in { }`, addresses take CIDR ranges, and `any()` / `all()` reduce an array comparison to a boolean. ## Supported fields The scheme carries **87 fields**. Every one is **optional** — a single packet rarely has all of them, and a rule referencing a field that was never populated simply does not match. ### Network and transport | Field | Type | | |---|---|---| | `ip.src`, `ip.dst` | IP | Source and destination address | | `ip.proto` | Bytes | IP protocol | | `tcp.src_port`, `tcp.dst_port` | Int | TCP ports | | `udp.src_port`, `udp.dst_port` | Int | UDP ports | | `tcp.ttl`, `tcp.window`, `tcp.mss`, `tcp.window_scale` | Int | The raw inputs JA4T is built from, for when you want one component rather than the whole fingerprint | ### Application | Field | Type | | |---|---|---| | `tls.sni`, `tls.alpn` | Bytes | From the TLS ClientHello | | `http.method`, `http.host`, `http.path`, `http.user_agent` | Bytes | Request line and headers | | `dns.fqdn` | Bytes | Destination hostname resolved from a snooped DNS answer — this is what makes egress-by-domain rules possible: `dns.fqdn matches "(^|\.)evil\.com$"` | ### Fingerprints All nine JA4+ identifiers are Bytes, so they take `eq`, `in { }`, `contains`, `wildcard` and `matches`: `ja4` · `ja4s` · `ja4t` · `ja4ts` · `ja4h` · `ja4l` · `ja4ls` · `ja4x` · `ja4ssh` The latency and SSH fingerprints are also decomposed into integers, so you can express a band rather than match a literal: | Family | Fields | |---|---| | Latency | `ja4l.rtt_us`, `ja4l.ttl`, `ja4ls.rtt_us`, `ja4ls.ttl` | | Latency delta | `ja4l.delta_us`, `.delta_abs_us`, `.delta_ms` | | App-handshake timing | `ja4l.app_rtt_us`, `ja4ls.app_rtt_us`, and `tcp_app_delta_us` / `_abs_us` / `_ms` on both | | SSH counts | `ja4ssh.c2s_pkts`, `.s2c_pkts`, `.c2s_acks`, `.s2c_acks`, `.c2s_bytes`, `.s2c_bytes` | The latency delta is worth understanding rather than skipping: it is client RTT minus server RTT, so a genuinely distant client shows a large positive delta while a spoofed or colocated bot shows one near zero. ### Enrichment These are populated by the agent rather than read off the packet, so they are present only when the corresponding feature is enabled. | Family | Fields | |---|---| | GeoIP | `ip.src.country`, `ip.dst.country`, `ip.src.asn`, `ip.dst.asn`, `ip.src.asn_org`, `ip.dst.asn_org` | | Threat intel | `threat.score` (0–100), `threat.advice`, `threat.labels` | | IDS | `ids.alert_count`, `ids.block_count`, `ids.notice_count`, `ids.score`, `ids.severity`, `ids.sids` | | Network scope | `ids.src_home_net`, `ids.dst_home_net`, `ids.src_external_net`, `ids.dst_external_net`, `ids.src_pod_net`, `ids.dst_pod_net` | | Flow behaviour | `flow.flows_per_min`, `flow.unique_dst_ports`, `flow.unique_src_ports`, `flow.dst_port_entropy`, `flow.burstiness` | :::caution[`ids.severity` runs the opposite way to `ids.score`] `ids.score` is 0–100 where **higher is worse**. `ids.severity` is the raw Suricata-style value where **1 is the most severe**. So `ids.severity > 3` selects the *least* serious alerts, which is almost never what someone writing it means. Prefer `ids.score` for thresholds. Reach for `ids.severity` only when you specifically want the raw scale, and read it as a rank rather than a magnitude. ::: `ids.sids` is an array of the signature IDs that fired, which is how a rule enforces on an IDS detection that has no dedicated field of its own: ```text any(ids.sids[*] in {2054155 2047703}) ``` ### ML verdicts Alert-first by design: the models emit a verdict, and a rule opts in to acting on it. | Family | Fields | |---|---| | Traffic classifier | `ml.traffic.class`, `.confidence`, and a per-class probability for `benign`, `scan`, `bruteforce`, `dos`, `exfil`, `c2`, `malware` | | Flow classifier | `ml.flow.label`, `.benign`, `.malicious` | | JA4-JEPA | `ml.jepa.label`, `.similarity` | Per-class probabilities matter because they let a rule fire on a class that was *not* the argmax — `ml.traffic.c2 ge 50` catches a flow the classifier ranked as something else but still considered plausibly C2. ### Kubernetes identity | Field | Type | | |---|---|---| | `identity.k8s.src_workload`, `dst_workload` | Bytes | Workload name, resolved from the identity feed | | `identity.k8s.src_namespace`, `dst_namespace` | Bytes | Namespace | | `identity.k8s.src_label`, `dst_label` | Map | Pod labels by key: `identity.k8s.src_label["role"] eq "database"` | | `edge.declared` | Int | 1 when this conversation is allowed by a NetworkPolicy | | `edge.policy_violation` | Int | 1 when the destination **is** governed by policy and this edge is not in the allow-list | `edge.policy_violation == 1` is the microsegmentation signal — a real policy-deny rather than merely an undeclared edge. See [Service Graph](./service-graph.mdx). :::note[Fields carry no value unless something fills them] The TLS fields need a parsed ClientHello, the HTTP fields need a parsed request, and the enrichment families need their feature enabled and its feed loaded. An unset field does not error — the rule just does not match, which is why a rule that appears to do nothing is usually a rule referencing a field nothing populated. ::: Field names deliberately mirror the [WAF](./waf.mdx) scheme where they overlap, so an expression can move between the two without translation. Rules arrive in three lists — `block`, `allow`, and standalone expressions — and are delivered from the platform rather than written into the agent's local config. The [Signal API](../api/signal-api.mdx) is how an automation or SOAR playbook adds one. :::note[Two actions only] A Smart Firewall rule is `block` or `allow`. There is no challenge and no rate limit here, because the decision happens **before a connection is established** — there is no request yet to challenge. Those actions belong to the [WAF](./waf.mdx). ::: ## Precedence **The first matching rule wins**, in order. An `allow` does not outrank a `block` by virtue of being an allow — it outranks it by being earlier. :::caution[Order your exceptions above the rule they escape] This is the same evaluation model as [firewall rules](./firewall-rules.mdx), and it catches people the same way: an `allow` placed below a broad `block` never runs, because the block already decided. ::: An expression that does not parse is logged and skipped at load. A bad rule does not stop the agent from starting, and it does not stop the rest of the ruleset from applying — but it also does not tell you at write time, so check the log after shipping one. ## How enforcement lands Two paths, and the difference matters: | Rule | Enforcement | |---|---| | `ja4t` block | Kernel BPF, at **SYN time** — the connection is dropped before the handshake completes | | Everything else | The reactor installs an address block into the kernel firewall, applied from then on | The reason for the split is what is available when. JA4T is computed from TCP options present in the SYN, so the kernel can decide immediately. A JA4 needs the TLS ClientHello, which arrives after the handshake — by then the connection exists, so what a match buys you is *that address blocked going forward*, not that packet dropped. Blocks are direction-aware: the remote end of the flow is the block target, so a match on traffic you initiated does not blocklist your own host. Blocks carry a **TTL** and are swept when they expire, so a fingerprint seen once does not become a permanent entry in a kernel map that only grows. ## Platform coverage | | | |---|---| | Linux, eBPF build | XDP, with nftables and iptables as fallbacks | | Windows | The NDIS path, same rule model | The `ja4t` SYN-time drop needs an eBPF build; without one, rules still apply through the reactor and the address-block path. ## Limits worth knowing - **A fingerprint identifies a population, not a person.** Every client of a given TLS stack and build shares one. Blocking a common browser fingerprint blocks everyone using that browser — pair it with an address condition, or use it as one signal among several. - **Only `ja4t` stops the first connection.** For every other fingerprint the first connection completes; you are blocking the second onwards. - **Rules come from the platform.** This is not a file you hand-edit on the host, which is deliberate — but it means an agent that cannot reach the platform keeps the ruleset it last received. - **Invalid expressions fail quietly.** They are skipped with a log line, not rejected at submission. ## Use cases - **Block a scanner that rotates addresses** — the fingerprint is stable even when the address is not. - **Neutralise a botnet family** by the TLS stack its malware links against, in one rule. - **Act on a classifier verdict** — take the fingerprint the [ML models](./machine-learning.mdx) flagged and install it as a kernel block. - **Automate response from a playbook** by posting a `smart_firewall_rule` to the [Signal API](../api/signal-api.mdx) and having every agent enforce it. ## See also - [JA4+](../client/ja4-plus.mdx) — what each fingerprint captures and how stable it is - [Amygdala](../components/amygdala/) — the engine that evaluates these rules - [Hillock](../components/hillock/) — the kernel firewall that enforces the result - [Access Rules](./access-rules.mdx) — the cheaper, static, address-only layer - [Signal API](../api/signal-api.mdx) — submitting rules programmatically --- # Threat Detection Source: https://docs.gen0sec.com/docs/features/threat-detection # Threat Detection Threat detection enriches every request with what is already known about its source — reputation from Gen0Sec's feeds, geographic and network origin, and the client's own fingerprint — and makes all of it available as conditions you can write rules against. It does not decide anything on its own. It supplies the evidence; [rules](./waf.mdx) decide what to do with it. ## What it gives you - **Reputation without running your own feeds.** Scores are aggregated from a large set of sources and kept current for you. - **Identity that survives IP rotation.** A client's fingerprint stays the same when its address changes, so a blocked tool does not come back simply by moving. - **Network and geographic context** — ASN, the organisation behind it, and country. - **One vocabulary across features.** The same fields are available to the WAF, to access policy, and to logging. ## What you can act on | Field | Meaning | |---|---| | `threat.score` | Reputation score, 0–100 | | `threat.advice` | Recommended action: `allow`, `block` or `challenge` | | `ip.src.country` | Country of the source address | | `ip.src.asn` | Autonomous System Number | | `ip.src.asn_org` | Organisation that owns the ASN | | `ip.src.asn_country` | Country the ASN is registered in | The full field list is on [WAF](./waf.mdx#supported-fields) — kept there rather than duplicated here so the two cannot drift. :::note[Two rule surfaces, two field sets] These are the fields available to **WAF rules**, which run in the proxy on HTTP requests. The agent's firewall rules use a different and larger scheme — it reaches IDS, ML and Kubernetes identity, and names HTTP fields differently (`http.host` rather than `http.request.host`). A field that works in one will not necessarily parse in the other. See [Amygdala](../components/amygdala/) for the firewall side. ::: Combining them is the point: ```text # Reputation says block, and the source agrees it is worth acting on threat.score > 80 and threat.advice == "block" # Somewhere between suspicious and hostile — challenge instead of blocking threat.score > 50 and threat.score <= 80 # Reputation, narrowed to one network ip.src.asn == 64500 and threat.advice == "block" ``` ## Advice, and why it exists alongside the score `threat.score` is a number; `threat.advice` is Gen0Sec's recommendation given everything known about that source. Acting on advice keeps your policy stable when scoring changes underneath it — you are asking "should this be blocked?" rather than pinning a threshold that may mean something slightly different next month. Thresholds remain available when you want to set the bar yourself. ## Fingerprints as identity Threat detection also attaches the client's JA4+ fingerprint, which describes *how* the client speaks rather than where it speaks from — its TLS handshake, its HTTP header order, its TCP stack. Two connections from different addresses running the same tool share a fingerprint. That makes a fingerprint the durable half of an identity, and an address the disposable half. See [JA4+](../client/ja4-plus.mdx) for what each type covers, and [Dendrite](../components/dendrite/) for how they are captured. ## Limits worth knowing - **Reputation is about the source, not the request.** A clean address can still send a malicious payload — that is the [WAF](./waf.mdx)'s and [Content Scanning](./content-scanning.mdx)'s job. - **Shared addresses carry shared reputation.** A CGNAT or corporate egress address represents many users; prefer fingerprint or a combination of signals before blocking one outright. - **A fingerprint identifies a client, not a person.** Many legitimate clients share one, so treat it as a strong grouping signal rather than proof. ## Use cases - **Block sources the feeds already know are hostile**, before writing a single custom rule. - **Challenge rather than block** in the ambiguous middle, so a false positive costs a user friction instead of an outage. - **Stop a credential-stuffing tool by fingerprint**, so rotating its addresses does not help it. - **Require two signals before acting** — reputation and fingerprint — to keep false positives off the block list. ## See also - [Fleet Bans](./fleet-bans.mdx) — what your own agents corroborated, as distinct from the platform-wide feed - [WAF](./waf.mdx) — the rule engine and the full field list - [Configuration](../client/configuration.mdx) — platform credentials and telemetry settings - [Cortex](../components/cortex/) — behavioural scoring for threats with no reputation yet - [Thalamus](../components/thalamus/) — signature detection on the same traffic --- # Web Application Firewall (WAF) Source: https://docs.gen0sec.com/docs/features/waf # Web Application Firewall (WAF) Synapse WAF provides comprehensive protection for your web applications and APIs against a wide range of attacks and security threats. ## Overview Our WAF solution offers: - **Real-time protection** against OWASP Top 10 threats - **API security** with specialized protection - **Custom rule creation** for specific needs - **Performance optimization** with minimal latency - **Cloud-native deployment** across multiple platforms ## Core Protection Features ### OWASP Top 10 Protection Comprehensive protection against the most critical web application security risks: 1. **Injection** - SQL, NoSQL, OS command injection 2. **Broken Authentication** - Session management flaws 3. **Sensitive Data Exposure** - Data leakage prevention 4. **XML External Entities (XXE)** - XML processing attacks 5. **Broken Access Control** - Authorization bypasses 6. **Security Misconfiguration** - Configuration vulnerabilities 7. **Cross-Site Scripting (XSS)** - Client-side attacks 8. **Insecure Deserialization** - Object injection attacks 9. **Known Vulnerabilities** - Component security issues 10. **Insufficient Logging** - Security event visibility ## Rule expressions Synapse WAF evaluates rules using a Wirefilter-compatible expression language (Cloudflare Ruleset Engine style). Expressions run against each request and may trigger an action when matched. - **Syntax**: Boolean expressions with fields, literals, operators, grouping: `()`, `and`, `or`, `not`, `==`, `!=`, `<`, `<=`, `>`, `>=`, `in`, `contains`, `matches`, `starts_with()`, `ends_with()`, etc. - **Evaluation**: First matching rule short-circuits. Invalid expressions are skipped and logged. - **Source**: Rules are stored in the database per organization; expired rules are ignored. ### Operators and grouping | Operator | Description | | --- | --- | | `and` | Logical AND; both operands must evaluate to true. | | `or` | Logical OR; at least one operand must evaluate to true. | | `not` | Logical negation; inverts the boolean value of the operand. | | `()` | Groups expressions to control precedence. | | `==` | Equality comparison between operands of the same type. | | `!=` | Inequality comparison between operands of the same type. | | `<`, `<=`, `>`, `>=` | Relational comparison for numeric or lexicographic types. | | `in` | Membership test against a set or range. | | `contains` | True if left string contains the right substring. | | `matches` | True if left string matches the right regular expression. | Precedence: `not` > `and` > `or` (use parentheses to make precedence explicit). :::note[`starts_with` and `ends_with` are functions, not operators] `contains` and `matches` sit between their operands. `starts_with` and `ends_with` do not — they are called, like every entry in the function table below: ```text starts_with(http.request.path, "/admin") # correct http.request.path starts_with "/admin" # does not parse ``` ::: ### Supported fields These fields are provided to expressions per request. | Field | Type | Description | | --- | --- | --- | | `http.request.method` | Bytes | HTTP method (GET, POST, etc.). | | `http.request.scheme` | Bytes | URL scheme (http, https). | | `http.request.host` | Bytes | Request host (authority). | | `http.request.port` | Int | Destination port. | | `http.request.path` | Bytes | Path component only (no query). | | `http.request.uri` | Bytes | Full URI including path and query. | | `http.request.query` | Bytes | Raw query string. | | `http.request.user_agent` | Bytes | User-Agent header value. | | `http.request.content_type` | Bytes | Content-Type header value. | | `http.request.content_length` | Int | Content length in bytes. | | `http.request.body` | Bytes | Request body (as text/bytes). | | `http.request.body_sha256` | Bytes | Hex-encoded SHA-256 of body. | | `http.request.headers` | Map(Array(Bytes)) | Map of all headers (header name → array of values). | | `ip.src` | Ip | Source client IP address. | | `ip.src.country` | Bytes | ISO country code from threat intelligence. | | `ip.src.asn` | Int | Autonomous System Number. | | `ip.src.asn_org` | Bytes | ASN organization name. | | `ip.src.asn_country` | Bytes | ASN country code. | | `threat.score` | Int | Threat intelligence score (0-100). | | `threat.advice` | Bytes | Threat intelligence advice (allow, block, challenge). | ### Supported functions The following functions are recognized and can be used in expressions. Only these are currently allowed; unsupported functions cause validation failure. | Function | Signature | Returns | Description | | --- | --- | --- | --- | | `all` | `all(array, predicate)` | bool | True if all elements satisfy the predicate. | | `any` | `any(array, predicate)` | bool | True if any element satisfies the predicate. | | `cidr` | `cidr(ip, cidr)` | bool | Checks if an IP is within the CIDR range. | | `concat` | `concat(array)` | string | Concatenates array elements into a single string. | | `decode_base64` | `decode_base64(bytes)` | bytes | Decodes a base64-encoded string. | | `ends_with` | `ends_with(bytes, suffix)` | bool | Checks if a string ends with suffix. | | `json_lookup_integer` | `json_lookup_integer(json_bytes, json_path)` | int | Extracts integer via JSON path. | | `json_lookup_string` | `json_lookup_string(json_bytes, json_path)` | bytes | Extracts string via JSON path. | | `len` | `len(array)` | int | Number of elements in array. | | `lower` | `lower(bytes)` | bytes | Converts string to lowercase. | | `remove_bytes` | `remove_bytes(bytes, remove)` | bytes | Removes occurrences of `remove` from input. | | `remove_query_args` | `remove_query_args(url, args)` | bytes | Removes query parameters from URL. | | `starts_with` | `starts_with(bytes, prefix)` | bool | Checks if a string starts with prefix. | | `substring` | `substring(bytes, start, length)` | bytes | Extracts substring. | | `to_string` | `to_string(any)` | bytes | Converts value to string representation. | | `upper` | `upper(bytes)` | bytes | Converts string to uppercase. | | `url_decode` | `url_decode(bytes)` | bytes | URL-decodes a string. | | `uuid4` | `uuid4(seed_bytes)` | bytes | Generates UUIDv4 with optional seed. | | `wildcard_replace` | `wildcard_replace(bytes, pattern, replacement)` | bytes | Replaces wildcard pattern in string. | Examples: ```text # Block requests with high threat scores threat.score > 80 and threat.advice == "block" # Block SQL injection attempts in query string http.request.query contains "union select" or http.request.query contains "1=1" # Block suspicious user agents http.request.user_agent contains "scanner" or http.request.user_agent contains "bot" # Allow only specific countries ip.src.country not in {"US", "CA", "GB"} # Check request body hash against known malicious payloads http.request.body_sha256 == "a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3" # Block POST requests to sensitive paths http.request.method == "POST" and starts_with(http.request.path, "/admin") ``` ### Actions Each rule carries an `action`, and there are four. The first match short-circuits further evaluation, so rule order decides which one applies. | Action | Effect | |---|---| | `block` | Refuse the request. The default. | | `challenge` | Issue a CAPTCHA. The client can proceed by solving it. | | `rate_limit` | Allow until the rule's threshold is crossed, then return **429** for a configured period. | | `content_scanning` | Send the body to the [content scanner](./content-scanning.mdx), and refuse with **403** on a signature hit. | | `allow` | Permit and stop evaluating — an explicit exception above broader rules. | `challenge` also accepts the spelling `captcha`, which is what telemetry reports it as. `content_scanning` is request-phase only — by the response phase the body is gone. ### Rate limiting `rate_limit` counts **per rule and per client address**, so two rules maintain independent counters and one client crossing a threshold does not affect another. Three settings, all carried on the rule: | Field | Meaning | |---|---| | `requests` | How many requests are allowed in the window | | `period` | Length of the window, in seconds | | `duration` | How long the client stays limited after crossing, in seconds | `duration` is the one worth reading twice: it is not the window, it is the penalty. A rule with `period: 60` and `duration: 600` allows a burst to be counted over a minute and then holds the client at 429 for ten. Requests that pass under the threshold are not marked as remediated — only the ones that actually received a 429 are, so your event stream counts enforcement rather than evaluation. :::caution[Behind a load balancer, the counter keys on the wrong address] The bucket key is the **L4 socket peer**, which behind a reverse proxy is the proxy, not the client. Two failure modes, both observed in production: - A load balancer that **rotates its source addresses** spreads one client across many buckets, so no bucket ever reaches the threshold and the rule silently never fires. - A load balancer that **collapses clients onto a few addresses** puts unrelated users in one bucket, so they rate-limit each other. The fix is `proxy.trusted_proxies` — see [Configuration](../client/configuration.mdx#client-address-behind-a-proxy). Until it is set, a rate-limit rule behind an LB is not doing what it appears to. ::: ### Response - Response payload includes reason, action, and details about the matched rule. Successful response (default, when `originalEvent` is not requested): ```json { "success": true, "reason": "Request blocked due to WAF rule ", "action": "block", "details": { "blocked_by_waf": true, "matched_rule": "", "matched_rule_action": "block", "rules_matched": 1, "rules_processed": 3, "processed_at": "" } } ``` Notes: - Only idempotent HTTP methods (GET, PUT, DELETE, HEAD, OPTIONS, TRACE) are cached for 15s. - `originalEvent=true` query returns the full event instead of the summarized response. ## Rule Engine ### Pre-built Rules Comprehensive rule sets for common threats: - **SQL Injection** - Database attack prevention - **XSS Protection** - Cross-site scripting defense - **CSRF Protection** - Request forgery prevention - **File Upload Security** - Malicious file prevention - **Bot Protection** - Automated attack prevention ## Performance Optimization ### Edge Computing - **Global edge deployment** for minimal latency - **Local rule evaluation** when possible - **Caching** of frequently used rules - **Connection pooling** for API calls ### Rule Optimization - **Rule ordering** for optimal performance - **Conditional evaluation** to skip unnecessary checks - **Caching** of rule results - **Batch processing** for multiple requests ## Monitoring and Analytics ### Real-time Dashboard - **Attack visualization** with geographic mapping - **Threat trends** and patterns - **Performance metrics** and latency monitoring - **Rule effectiveness** analysis ### Logging and Alerting - **Detailed attack logs** with full context - **Real-time alerts** for critical threats - **SIEM integration** for security teams - **Custom dashboards** for specific needs ## Configuration Management ### Rule Management - **Version control** for rule changes - **A/B testing** for rule effectiveness - **Rollback capabilities** for quick recovery - **Environment promotion** from dev to production ### Policy Templates Pre-built policy templates for common use cases: - **E-commerce** - Online store protection - **SaaS Applications** - Multi-tenant security - **APIs** - API-first application protection - **Content Management** - CMS security ## Best Practices ### Rule Development - **Start with monitoring** before blocking - **Test rules thoroughly** in staging - **Document rule purposes** and business justification - **Regular rule reviews** and updates ### Performance - **Monitor latency impact** of rules - **Optimize rule order** for efficiency - **Use caching** where appropriate - **Scale resources** based on traffic ### Security - **Regular threat intelligence** updates - **Monitor for new attack vectors** - **Keep rules current** with latest threats - **Conduct security reviews** regularly --- # Data Flow Source: https://docs.gen0sec.com/docs/architecture/data-flow # Data Flow ## Network Traffic Flow ### Agent Mode In agent mode, Synapse operates as a kernel-level firewall with no HTTP proxying.
Inbound packet Linux kernel XDP — access rules before routing address lookup in a map cheapest possible drop TC — firewall rules address · port · protocol TCP flags · direction first match wins Where eBPF is unavailable the same rules run through nftables, then iptables the rule does not change, only the machinery Kernel network stack then your application Dropped never reaches user space XDP stage TC stage Fallback Verdict Inbound packet XDP — access rules before routing address lookup in a map cheapest possible drop TC — firewall rules address · port · protocol TCP flags · direction first match wins Kernel stack, then your app or dropped, never reaching user space where eBPF is absent nftables, then iptables the rule does not change XDP stage TC stage Fallback Verdict
Two kernel stages. Either can drop; neither can redirect.
1. **XDP filtering** — Packets are evaluated at the earliest point the kernel offers, before routing. Known-bad addresses from threat intelligence and access rules are dropped there, without a context switch into user space. 2. **Firewall backend** — Synapse auto-selects the best available backend: XDP > nftables > iptables > userland. Rules from the Gen0Sec API (allow/block lists) are enforced here. 3. **TCP fingerprinting** — JA4T and JA4L fingerprints are captured from SYN packets for OS identification and latency measurement. 4. **Clean traffic** reaches the application directly — no proxying overhead. ### Proxy Mode In proxy mode, Synapse adds full reverse proxy capabilities on top of agent features.
Request Synapse, proxy mode XDP address filter before routing TLS terminate, or pass through inspection needs this Inspect WAF · CAPTCHA · content scan threat score · fingerprint any of these can stop the request Upstream selection healthy backend, balanced sticky sessions optional Forwarded X-JA4 headers to your backend client identity, free Your backend Blocked 403, or a challenge page Kernel Proxy Decision To your app Request XDP address filter before routing TLS terminate, or pass through inspection needs termination Inspect WAF · CAPTCHA · content scan threat score · fingerprint any of these can stop it Upstream selection healthy backend, balanced sticky sessions optional with X-JA4 headers Your backend gets the client fingerprint client identity, free Kernel Proxy Decision To your app
Anything that stops the request never reaches your backend — and what does carries the client's fingerprint.
1. **XDP filtering** — Same kernel-level packet filtering as agent mode. 2. **TLS termination** — Terminates with your certificates, or passes the connection through by SNI. JA4 (ClientHello) and JA4S (ServerHello) fingerprints are captured during the handshake. 3. **WAF engine** — Wirefilter-compatible expression language evaluates requests against security rules. GeoIP and ASN data enrich rule evaluation. 4. **CAPTCHA** — Suspicious requests can be challenged with hCaptcha, reCAPTCHA, or Turnstile. 5. **Content scanning** — Bodies selected by a `content_scanning` WAF rule are scanned via ClamAV. 6. **Reverse proxy** — Clean requests are forwarded to upstream servers with weighted load balancing. 7. **JA4H fingerprinting** — HTTP request headers are fingerprinted at the application layer. ## Threat Intelligence Flow Synapse continuously syncs threat data from the Gen0Sec API.
Your Kubernetes cluster Cerebellum resolves ASN and country to IP ranges here so the agent never does Gen0Sec API threat intel, GeoIP, models and rules rules · models · ranges MMDB refresh The agent Synapse applies what it receives Access rule maps XDP filter maps WAF rule context ML models hot-swapped Scored here verdicts computed on this host nothing about your traffic leaves it inference is local Control plane Agent Kernel maps Models Cerebellum resolves ASN and country to IP ranges here so the agent never does rules · models · ranges Synapse applies what it receives Kernel maps access rules · XDP filters WAF rule context ML models hot-swapped, no restart Scored on this host verdicts computed locally nothing about your traffic leaves Control plane Agent Kernel maps Models
Models are distributed. Verdicts are computed on the host and stay there.
| Data Source | Update Method | Used By | |-------------|--------------|---------| | **Gen0Sec Threat API** | Periodic API polling | Access rules, IP reputation | | **Threat MMDB** | File-based, hot-reloadable | GeoIP, ASN, threat scoring | | **GeoIP MMDB** | File-based, hot-reloadable | Country/city/ASN lookups | | **Access Rules** | API sync + SIGHUP reload | Firewall allow/block lists | | **Cerebellum Verdicts** | Real-time push | Threat classification updates | ## Fingerprinting Pipeline JA4+ fingerprints are captured at different layers and included in every access log entry. | Layer | Fingerprints | Capture Point | |-------|-------------|---------------| | **TCP** | JA4T, JA4TS, JA4L, JA4LS | SYN/SYN-ACK packets | | **TLS** | JA4, JA4S, JA4X | TLS handshake | | **HTTP** | JA4H | Request headers | | **SSH** | JA4SSH | SSH session traffic | | **DHCP** | JA4D, JA4D6 | DHCP messages | | **Scan** | JA4TScan | Active TCP probes | Fingerprints flow into: - **Access logs** — Every request includes all available fingerprints - **Event queue** — Batched for downstream analysis - **Cerebellum** — ML correlation across all sensors ## Where decisions are made, and where they are enforced The single most useful thing to understand about the data flow is that **the packet path only ever compares things it can read from a packet header.** Everything richer is resolved somewhere else, before the packet arrives. | Condition | Resolved | By | |---|---|---| | ASN, country in access rules | Ahead of time, into IP ranges | Cerebellum, at config generation | | ML verdict, IDS context, threat score, workload identity | On the host, at decision time | [Amygdala](../components/amygdala/) | | Address, port, protocol, TCP flags, fingerprint, rate | On the packet path | [Hillock](../components/hillock/) in the kernel | So "block this ASN" never becomes an ASN lookup per packet — it becomes a set of address blocks generated upstream. And "block when the classifier says C2 and the IDS already alerted" is evaluated on the host, which then installs a plain address block. Two consequences worth planning around: - **Enforcement cost is flat.** An elaborate policy costs the same per packet as a simple one, because the kernel is comparing the same primitives either way. - **Pre-resolved conditions are as fresh as the last config generation**, not as fresh as the packet. A network that announces a new prefix is covered at the next regeneration. Where that matters, match on it live through Amygdala instead. ## Event & Log Processing Synapse uses a unified event queue for all telemetry. Every layer that reaches a verdict — access rules, the smart firewall, the IDS, the WAF, threat intelligence, CAPTCHA, rate limiting — emits the same structured event. One stream, whichever layer decided, which is what makes "why was this client blocked" a single question rather than a correlation exercise.
Every layer that decides access rules smart firewall IDS WAF threat intel CAPTCHA rate limit one event shape, whichever decided Unified event stream block · drop · allow · notice with the layer that decided and the reason one query, not a correlation Readable log for tailing on the box JSON Lines one record per line CEF ArcSight, QRadar OTLP / HTTP batched to a collector Your shipper Fluent Bit, Vector Your SIEM Your collector Sources One stream Formats Your tooling Every layer that decides access rules · smart firewall IDS · WAF · threat intel CAPTCHA · rate limit one event shape Unified event stream block · drop · allow · notice with the layer and the reason one query, not a correlation Readable log for tailing on the box JSON Lines one record per line CEF ArcSight, QRadar OTLP / HTTP batched to a collector Your shipper, SIEM or collector Sources One stream Formats Your tooling
One event shape, whichever layer decided, in whichever format your tooling reads.
See [Security Event Export](../features/event-export.mdx) for what each sink is for. ### Event Types | Event | Description | Interval | |-------|-------------|----------| | **Access log** | Per-request log with fingerprints, GeoIP, threat data | Real-time | | **TCP fingerprint stats** | Aggregated SYN fingerprint statistics | Configurable (default 60s) | | **Fingerprint events** | Individual fingerprint observations | Configurable (default 30s) | | **BPF stats** | eBPF program statistics and dropped IPs | Configurable (default 60s) | | **Threat intel updates** | Access rule and threat feed sync results | On sync | ### Log Output Access logs include full request context with all fingerprints: ```json { "event_type": "http_access_log", "schema_version": "1.0.0", "timestamp": "2025-10-29T12:34:56.789Z", "request_id": "req_1730203496789000000", "network": { "src_ip": "192.168.1.100", "src_port": 54321 }, "geoip": { "country": "United States", "iso_code": "US", "asn": 13335 }, "http": { "method": "GET", "path": "/api/data" }, "response": { "status": 200 }, "fingerprints": { "ja4": "t13d1516h2_8daaf6152771_b186095e22b6", "ja4h": "ge11cr15enus_a1b2c3d4e5f6_123456789abc_def012345678", "ja4t": "65535_2-4-8-1-3_1460_7", "ja4s": "t1302h2_1301_a56c5b993250", "ja4x": "aae71e8db6d7_b186095e22b6_c1a4f9e7d8b3", "ja4l": "12500_64" } } ``` The client address is `network.src_ip`, the status is `response.status`, the GeoIP object is `geoip` and carries no city, and every fingerprint is a field of `fingerprints`. There is no `client_ip`, `geo`, `tcp` or `tls.ja4` field. ## Full Platform Data Flow When deployed as a full stack (Cerebrum + Synapse + Jailer + Cerebellum), data flows across all components.
Internet Cerebrum XDP + IDS inline at the edge north-south filtered Synapse on your servers agent or proxy east-west too Jailer process sandboxing Applications Your Kubernetes cluster Cerebellum cross-site correlation · policy and model distribution the only place that sees more than one site events events policy policy Edge Host Cluster Isolation Internet Cerebrum XDP + IDS inline at the edge north-south filtered Synapse on your servers east-west too Jailer process sandboxing Applications events up, policy down Cerebellum cross-site correlation policy and model distribution the only place seeing >1 site Edge Host Cluster Isolation
Telemetry up, policy down. Your traffic itself never leaves the host.
1. **Cerebrum** captures wire-speed metadata (JA4+ fingerprints, IDS alerts) and sends it to Cerebellum 2. **Synapse** sends access logs, fingerprint events, and threat data to Cerebellum 3. **Cerebellum** runs ML inference across all sensor data, correlates cross-site patterns, and pushes updated threat verdicts back to Cerebrum and Synapse in milliseconds 4. **Jailer** enforces per-process sandboxing — file, network and exec control — through BPF LSM hooks, independently of the traffic path ## Service Discovery Upstreams are declared in a file, `upstreams.yaml`, and hot-reloaded when it changes — a rewrite is picked up without a restart and without dropping connections. | | | |---|---| | Source | A file on the host, watched for changes | | Reload | Automatic on write; no restart, no dropped connections | | Backends | One per route, or several with weights | | Health | Probed on an interval; failed backends leave rotation and return when they recover | File discovery is the only provider implemented. On Kubernetes, use the [Ingress and Gateway API controller](../client/kubernetes-ingress.mdx) instead — it discovers backends from Ingress and Gateway resources, which is the idiomatic path there. --- # Data Sources Source: https://docs.gen0sec.com/docs/architecture/data-sources # Data Sources Gen0Sec leverages a comprehensive collection of over 30 threat intelligence sources to provide accurate and up-to-date security scoring. Our data collection strategy combines multiple source types to ensure comprehensive coverage and high-quality threat intelligence. ## Source Categories ### Community Sources Community-driven threat intelligence feeds that are freely available and maintained by security researchers and organizations: - IP reputation databases and blocklists - Open source threat intelligence platforms - Community-maintained malware and botnet intelligence - Brute force and attack pattern detection feeds - Spam and malicious activity tracking - Network scanning and reconnaissance data - Proxy and anonymization service detection - Tor network infrastructure tracking - Banking trojan and financial threat intelligence - High-confidence threat intelligence feeds ### Free Sources Open source and freely available threat intelligence feeds: - Commercial-grade threat intelligence with free tiers - Regional security community blocklists - Community intelligence scoring systems - Open source security research data ### Paid Sources Premium threat intelligence sources that require authentication and provide high-quality, curated data: - Advanced threat detection and response platforms - Proprietary threat intelligence feeds - Commercial security vendor data - Enterprise-grade threat intelligence services ## Data Collection Architecture ### Source Management Each data source is configured with: - **Source Metadata**: Name, URL, threat level, type, and expiration settings - **Parser Configuration**: Specific parsing logic for different data formats - **Authentication**: Support for both public and authenticated sources - **Caching Strategy**: Intelligent caching to optimize performance and reduce API calls ### Parser Types The system supports multiple parsing formats to handle diverse data sources: - **TXT Parser**: Simple text format with one indicator per line - **CSV Parser**: Comma-separated value format - **JSON Parser**: Structured JSON data format - **CrowdSec Parser**: Specialized parser for CrowdSec data format - **DangerRulez Parser**: Custom parser for DangerRulez format - **Flexible Parser**: Adaptive parser that can handle various formats ### Data Processing Pipeline 1. **Collection**: Automated data fetching from all configured sources 2. **Parsing**: Format-specific parsing to extract threat indicators 3. **Validation**: IP address validation and deduplication 4. **Scoring**: Threat level assignment based on source reputation 5. **Tagging**: Automatic categorization using threat intelligence tags 6. **Storage**: Efficient storage in PostgreSQL with relationship mapping ## Threat Intelligence Tags The system categorizes threats using a comprehensive tagging system: - **Malware** (Weight: 9) - Malicious software and code - **Botnet** (Weight: 8) - Botnet command and control infrastructure - **C2** (Weight: 9) - Command and control servers - **Phishing** (Weight: 7) - Phishing and social engineering attacks - **Proxy** (Weight: 4) - Proxy and anonymization services - **Tor** (Weight: 5) - Tor network infrastructure - **Scanner** (Weight: 3) - Network scanning and reconnaissance - **VPN** (Weight: 2) - VPN services and infrastructure - **Spam** (Weight: 2) - Spam and unwanted communications - **Brute Force** (Weight: 6) - Brute force attack patterns - **Unknown** (Weight: 1) - Unclassified or unknown threats ## Data Quality and Reliability ### Source Validation - **Threat Level Scoring**: Each source is assigned a threat level (1-15) based on reliability and accuracy - **Expiration Management**: Automatic data refresh based on source-specific expiration times - **Error Handling**: Robust error handling and retry mechanisms for failed data collection - **Status Monitoring**: Real-time monitoring of source collection status ### Data Freshness - **Real-time Updates**: Continuous data collection and processing - **Cache Management**: Intelligent caching to balance freshness and performance - **Source Health**: Monitoring and alerting for source availability and data quality ## Integration and Scalability ### API Integration - **RESTful APIs**: Standardized API endpoints for data access - **Authentication**: Secure API access with authentication requirements - **Rate Limiting**: Intelligent rate limiting to respect source API limits ### Performance Optimization - **Parallel Processing**: Concurrent data collection from multiple sources - **Caching Strategy**: Multi-level caching for optimal performance - **Database Optimization**: Efficient database design with proper indexing ## Proprietary Data Gen0Sec also incorporates proprietary threat intelligence data: - **Internal Threat Data**: Data collected from Gen0Sec security infrastructure - **Custom Indicators**: Proprietary threat indicators and patterns - **Behavioral Analysis**: Advanced behavioral threat detection - **Machine Learning Models**: AI-powered threat classification and scoring This comprehensive approach to data collection ensures that Gen0Sec provides the most accurate and up-to-date threat intelligence available, combining the best of community, commercial, and proprietary sources. ## Score Each source carries a threat level reflecting how reliable it has proven, and that weighting feeds the per-address score you act on. What reaches a rule is `threat.score` on a 0–100 scale, alongside `threat.advice` — Gen0Sec's recommendation of `allow`, `block` or `challenge` given everything known about that address. [Threat Detection](../features/threat-detection.mdx) covers both, and how to write rules against them. --- # Data Usage & Privacy Source: https://docs.gen0sec.com/docs/architecture/data-usage # Data Usage & Privacy This document outlines how Gen0Sec handles data, our privacy policies, and information about Data Processing Agreements (DPA). ## Data Collection & Processing Gen0Sec collects and processes data to provide threat intelligence and security scoring services. Our data handling practices are designed with privacy and security at the core. ### Types of Data We Collect - **Security Logs**: Access logs, error logs, and security event data - **Performance Metrics**: Response times, throughput, and system health data - **Threat Indicators**: IP addresses, user agents, and behavioral patterns - **Configuration Data**: Service settings and integration parameters ### Data Processing Principles - **Minimal Collection**: We only collect data necessary for service delivery - **Purpose Limitation**: Data is used solely for security analysis and threat detection - **Data Retention**: Limited retention periods with automatic deletion - **Access Control**: Strict access controls and audit logging ## Data Processing Agreement (DPA) If your organization requires a Data Processing Agreement (DPA) for compliance with GDPR, CCPA, or other privacy regulations, please contact us or join our [Discord community](https://discord.gg/jzsW5Q6s9q) for assistance. ### DPA Requirements We can provide DPAs that address: - Data processing purposes and duration - Data subject rights - Security measures and data protection - Sub-processor relationships - Data breach notification procedures - International data transfers ### Contact Information **For DPA requests and privacy inquiries:** - **Email**: privacy@gen0sec.com - **Legal Team**: legal@gen0sec.com - **Support**: support@gen0sec.com - **Community**: [Discord](https://discord.gg/jzsW5Q6s9q) ### Response Time We typically respond to DPA requests within 5-7 business days and can execute agreements within 2-3 weeks depending on your specific requirements. ## Data Security ### Technical Safeguards - **Encryption**: Data encrypted in transit and at rest - **Access Controls**: Role-based access with multi-factor authentication - **Network Security**: Secure connections and firewall protection - **Regular Audits**: Security assessments and penetration testing ### Organizational Safeguards - **Employee Training**: Regular privacy and security training - **Background Checks**: Comprehensive screening for all personnel - **Confidentiality Agreements**: Binding agreements for all staff - **Incident Response**: Documented procedures for security incidents ## Data Subject Rights We support the following data subject rights: - **Access**: Request information about personal data we hold - **Rectification**: Correct inaccurate or incomplete data - **Erasure**: Request deletion of personal data - **Portability**: Receive data in a structured format - **Objection**: Object to certain types of processing ## International Data Transfers When data is transferred internationally, we ensure appropriate safeguards are in place: - **Standard Contractual Clauses**: EU-approved data transfer mechanisms - **Adequacy Decisions**: Recognition of equivalent protection levels - **Certification Schemes**: Industry-recognized privacy frameworks We're committed to transparency and will respond to all inquiries promptly. --- # Fingerprinting Source: https://docs.gen0sec.com/docs/architecture/ja4-fingerprinting # JA4+ Fingerprinting Synapse implements the complete JA4+ fingerprinting suite for advanced threat detection and behavioral analysis. Fingerprints identify clients, servers, and connections by their protocol behavior — not just IP addresses. ## Fingerprint Suite | Fingerprint | Layer | Source | Available In | |-------------|-------|--------|-------------| | **JA4** | TLS | ClientHello | Proxy | | **JA4S** | TLS | ServerHello | Proxy | | **JA4H** | HTTP | Request Headers | Proxy | | **JA4L** | Network | Client-to-Server Latency | Both | | **JA4LS** | Network | Server-to-Client Latency | Both | | **JA4T** | TCP | SYN Packet Options | Both | | **JA4TS** | TCP | SYN-ACK Response | Proxy | | **JA4TScan** | TCP | Active TCP Scan | Both | | **JA4X** | TLS | X.509 Certificate | Proxy | | **JA4SSH** | SSH | SSH Traffic Patterns | Both | | **JA4D** | DHCP | DHCPv4 Fingerprinting | Both | | **JA4D6** | DHCP | DHCPv6 Fingerprinting | Both | ## JA4 — TLS Client Fingerprint Fingerprints the TLS ClientHello message to uniquely identify client applications. **Format:** `{proto}{version}{sni}{cipher_count}{ext_count}{alpn}_{cipher_hash}_{ext_hash}` **Example:** `t13d1516h2_8daaf6152771_b186095e22b6` | Segment | Meaning | |---------|---------| | `t` | TCP (or `q` for QUIC) | | `13` | TLS 1.3 | | `d` | SNI present (`d` = domain, `i` = IP) | | `15` | 15 cipher suites | | `16` | 16 extensions | | `h2` | ALPN protocol (HTTP/2) | | `_8daaf6152771` | SHA256 truncated hash of sorted cipher suites | | `_b186095e22b6` | SHA256 truncated hash of sorted extensions | **Detects:** Bot traffic, TLS library tampering, client application identification, impersonation attempts. ## JA4S — TLS Server Fingerprint Fingerprints the TLS ServerHello response to identify server infrastructure. **Format:** `{proto}{version}{ext_count}{alpn}_{cipher}_{extensions_hash}` **Example:** `t1302h2_1301_a56c5b993250` | Segment | Meaning | |---------|---------| | `t13` | TLS 1.3 over TCP | | `02` | 2 extensions | | `h2` | ALPN HTTP/2 | | `1301` | Cipher: TLS_AES_128_GCM_SHA256 | | `_a56c5b993250` | Extension hash | **Detects:** Backend infrastructure mapping, load balancer identification, server configuration changes, CDN fingerprinting. ## JA4H — HTTP Header Fingerprint Fingerprints HTTP request headers to identify clients at the application layer. **Format:** `{method}{version}{cookie}{referer}{count}{lang}_{headers_hash}_{cookie_names_hash}_{cookie_values_hash}` **Example:** `ge11cr15enus_a1b2c3d4e5f6_123456789abc_def012345678` | Segment | Meaning | |---------|---------| | `ge` | First 2 chars of HTTP method (GET) | | `11` | HTTP/1.1 | | `c` | Cookie header present | | `r` | Referer header present | | `15` | 15 headers (excluding Cookie/Referer) | | `enus` | Accept-Language (en-US) | | Three hashes | Headers, cookie names, cookie values | **Detects:** Browser fingerprinting, User-Agent spoofing, header manipulation, automation tools, scrapers. ## JA4T — TCP Client Fingerprint Fingerprints the TCP SYN packet to identify the client's OS and network stack. **Format:** `{window_size}_{tcp_options}_{mss}_{window_scale}` **Example:** `65535_2-4-8-1-3_1460_7` | Segment | Meaning | |---------|---------| | `65535` | TCP window size | | `2-4-8-1-3` | TCP option kinds (MSS, SACK, Timestamps, NOP, Window Scale) | | `1460` | Maximum Segment Size | | `7` | Window scale factor | **Common patterns:** | Pattern | OS | |---------|-----| | TTL=64, Window=65535 | Linux | | TTL=128, Window=65535 | Windows | | TTL=64, Window=65535 | macOS | These are rules of thumb about the IP and TCP headers, **not** about a JA4T value: a JA4T carries no TTL, and the window size alone does not separate the three rows above. To tell these stacks apart from a fingerprint, compare the TCP option order and the MSS - see [OS Fingerprinting](/docs/client/ja4-plus#os-fingerprinting). **Detects:** Operating system fingerprinting, network stack behavior, NAT/proxy presence, device type identification. ## JA4TS — TCP Server Fingerprint Fingerprints the TCP SYN-ACK response from the server side. **Format:** Same structure as JA4T but captured from the server's SYN-ACK packet. **Detects:** Server OS identification, load balancer detection, backend infrastructure mapping. ## JA4L — Client-to-Server Latency Measures client-to-server latency from packet timing for light distance estimation. **Format:** `{rtt_microseconds}_{ttl}` **Example:** `12500_64` (12.5ms RTT, TTL 64) **Calculation:** `(ACK - SYNACK) / 2` **Detects:** Network distance estimation, geographic location approximation, proxy/VPN detection, CDN usage patterns. ## JA4LS — Server-to-Client Latency Measures server-to-client latency from packet timing. **Format:** `{rtt_microseconds}_{ttl}` **Calculation:** `(SYNACK - SYN) / 2` **Detects:** Server distance measurement, hosting provider identification, latency anomaly detection. ## JA4X — X.509 Certificate Fingerprint Fingerprints X.509 certificates by their structural properties. **Format:** `{issuer_hash}_{subject_hash}_{extensions_hash}` **Example:** `aae71e8db6d7_b186095e22b6_c1a4f9e7d8b3` Three 12-char SHA256 truncated hashes of: - **Issuer** — RDN OIDs of the certificate issuer - **Subject** — RDN OIDs of the certificate subject - **Extensions** — Certificate extension OIDs **Detects:** Certificate authority patterns, certificate structure anomalies, organization identification, PKI mapping. ## JA4TScan — Active TCP Fingerprint Scanner Active TCP fingerprint scanning tool that sends crafted packets to identify remote systems. **Detects:** Remote OS identification, network service discovery, infrastructure mapping. ## JA4SSH — SSH Traffic Fingerprint Fingerprints SSH sessions by analyzing traffic patterns without decrypting the session. **Detects:** SSH client identification, automated SSH tools, lateral movement detection, tunneling activity. ## JA4D — DHCPv4 Fingerprint Fingerprints DHCP clients by analyzing DHCPv4 Discover and Request messages. **Detects:** Device type identification (IoT, workstation, mobile), rogue device detection, network inventory, unauthorized device discovery. ## JA4D6 — DHCPv6 Fingerprint Fingerprints DHCP clients by analyzing DHCPv6 Solicit and Request messages. **Detects:** IPv6 device identification, dual-stack device correlation, rogue device detection on IPv6 networks. ## Access Log Integration A proxy records every fingerprint it collected for a request under one `fingerprints` object, with the client address under `network`: ```json { "event_type": "http_access_log", "timestamp": "2025-10-29T12:34:56.789Z", "http": { "method": "GET", "path": "/api/data" }, "network": { "src_ip": "192.168.1.100", "src_port": 54321 }, "fingerprints": { "ja4": "t13d1516h2_8daaf6152771_b186095e22b6", "ja4h": "ge11cr15enus_a1b2c3d4e5f6_123456789abc_def012345678", "ja4t": "65535_2-4-8-1-3_1460_7", "ja4s": "t1302h2_1301_a56c5b993250", "ja4x": "aae71e8db6d7_b186095e22b6_c1a4f9e7d8b3", "ja4l": "12500_64" } } ``` There is no `client_ip`, no `tcp` object and no `tls.ja4`: every JA4 value is a field of `fingerprints`. The access log is a proxy file and needs `logging.file_logging_enabled: true`; an agent reports its fingerprints through `eventbridge.log` instead, where the same values sit at the top level of each line. See [Viewing Fingerprints](/docs/client/testing#viewing-fingerprints). ## Configuration The fingerprint and BPF-statistics streams share **one** switch, and it is on by default: ```yaml # /etc/synapse/config.yaml logging: eventbridge_log: enabled: true # the default # file: /var/log/synapse/eventbridge.log ``` `fingerprint_log` is an accepted alias for `eventbridge_log`, and one line per event covers every fingerprint type and the BPF statistics together. With `file` unset the path is `{log_directory}/eventbridge.log` — not `fingerprints.log`, which is the stream's old name and names no file anything writes. `LOGGING_EVENTBRIDGE_LOG_ENABLED` is the environment equivalent of the key. :::caution[The per-type switches no longer exist] `logging.tcp_fingerprint`, `logging.bpf_stats` and the other four per-type sections are **absent from the 0.8.8 schema**, and this page used to tell you to set two of them. Setting any of them has no effect: the keys land in the `Unused config options in ` warning at startup and nothing else happens. `TCP_FINGERPRINT_ENABLED`, `TCP_FINGERPRINT_LOG_INTERVAL`, `BPF_STATS_ENABLED` and `BPF_STATS_LOG_INTERVAL` have no reader anywhere in the code either. ::: ## Analyzing Fingerprints ### Extract Unique Fingerprints Every JA4 value is a field of `fingerprints`, and the client address is `network.src_ip`: ```bash AL=/var/log/synapse/access.log # -R with `fromjson?` skips any line that is not JSON instead of aborting the run: # two appenders write this path, and only one of them guarantees JSON. # Unique JA4 (TLS client) fingerprints jq -R -r 'fromjson? | select(.fingerprints.ja4) | .fingerprints.ja4' "$AL" | sort | uniq -c | sort -rn # Unique JA4T (TCP) fingerprints jq -R -r 'fromjson? | select(.fingerprints.ja4t) | .fingerprints.ja4t' "$AL" | sort | uniq -c | sort -rn # Unique JA4H (HTTP) fingerprints jq -R -r 'fromjson? | select(.fingerprints.ja4h) | .fingerprints.ja4h' "$AL" | sort | uniq -c | sort -rn # Correlate address with all fingerprints jq -R -r 'fromjson? | [.network.src_ip, .fingerprints.ja4 // "-", .fingerprints.ja4t // "-", .fingerprints.ja4h // "-"] | @tsv' "$AL" ``` On a proxy without `logging.file_logging_enabled` there is no `access.log`, but the record is not lost: it goes to the console instead, printed raw as one JSON object per line with no prefix, interleaved with non-JSON tracing lines. Pipe it through `jq -R 'fromjson? | select(.http)'` so those lines are skipped rather than fatal. On an agent there is no access log at all - read `eventbridge.log`, where each observation carries the same fingerprint values at the top level of its own line, without the request correlation. ### Bot Detection Combine multiple fingerprints to identify automated traffic: ```bash # Group by JA4 fingerprint, find addresses sharing the same fingerprint jq -R 'fromjson? | select(.fingerprints.ja4) | {ip: .network.src_ip, ja4: .fingerprints.ja4, ja4h: .fingerprints.ja4h}' \ /var/log/synapse/access.log | jq -s 'group_by(.ja4) | map({ja4: .[0].ja4, count: length, ips: map(.ip) | unique})' ``` ### Find Rare Fingerprints ```bash # Rare fingerprints often indicate attack tools or unusual clients jq -R -r 'fromjson? | select(.fingerprints.ja4) | .fingerprints.ja4' \ /var/log/synapse/access.log | sort | uniq -c | sort -n | head -20 ``` ## Why JA4+ Matters Traditional IP-based blocking is easily bypassed. JA4+ fingerprints provide: - **Persistent identification** — Fingerprints remain consistent across IP changes - **Evasion resistance** — Harder to spoof than User-Agent headers or IP addresses - **Granular detection** — Distinguish between different client types sharing the same IP - **Zero-day detection** — Identify previously unknown attack tools by their protocol behavior - **Cross-layer correlation** — Combine TLS, TCP, HTTP, and latency signals for high-confidence identification ## References - [Official JA4+ Repository](https://github.com/FoxIO-LLC/ja4) - [JA4+ Technical Details](https://github.com/FoxIO-LLC/ja4/tree/main/technical_details) - [FoxIO Blog — JA4+ Network Fingerprinting](https://blog.foxio.io/ja4%2B-network-fingerprinting) ## License - **JA4** (TLS Client) — BSD 3-Clause License - **JA4S, JA4H, JA4L, JA4LS, JA4T, JA4TS, JA4TScan, JA4X, JA4SSH, JA4D, JA4D6** — FoxIO License 1.1 See [JA4 License](https://github.com/FoxIO-LLC/ja4/blob/main/LICENSE) for details. --- # Architecture Overview Source: https://docs.gen0sec.com/docs/architecture/overview # Architecture Overview Gen0Sec is a multi-layer Active NDR platform that protects infrastructure from kernel level to application layer. The platform consists of four products that work standalone or together for full-stack protection. ## Platform Components ### Synapse - Firewall & Reverse Proxy High-performance eBPF-powered firewall and reverse proxy built with Rust. Runs on your servers in two modes: - **Agent Mode** - Kernel-level firewall with XDP packet filtering, threat intelligence, TCP fingerprinting, and access rules enforcement. No HTTP proxying. - **Proxy Mode** - Full reverse proxy with all agent features plus WAF, JA4+ fingerprinting, CAPTCHA, content scanning, TLS management, and upstream load balancing. ### Cerebrum - Inline Hardware Sensors A line of two appliances, both running stock Synapse in native XDP mode. **Cerebrum Edge** — 1U distributed sensor for branch, SASE and SD-WAN edges. Single NXP LX2160A (16 ARM Cortex-A72 cores, 8× 1G + 8× 10G) or dual (32 cores, 12× 1G + 12× 10G), plus one mandatory OCP NIC — 4× SFP+ 10G, or 2× QSFP28 100G on the dual board. Forwarding capacity is **100 Gb/s maximum**. That ceiling is the platform's, not the front panel's: fitting the 100G OCP NIC gives you more port bandwidth than the box forwards. Size against the 100 Gb/s figure. **Cerebrum Max** — 2U core and aggregation sensor. NVIDIA Grace C1 on MGX with 72 Neoverse V2 cores and 256 GB LPDDR5, three PCIe Gen5 x16 slots taking BlueField-3 DPUs or ConnectX-7 NICs, **800 Gb/s** wire speed. Both carry the IDS/IPS engine and JA4+ fingerprinting inline. Both are edge appliances for north-south traffic — neither runs inside Kubernetes. For cloud and Kubernetes sites, Synapse in proxy mode covers north-south instead. ### Cerebellum - Self-hosted Platform The enterprise Gen0Sec platform, running in your own Kubernetes cluster. Three Helm charts install the operators, the data plane and the dashboard. - **Runs entirely in your cluster** — Postgres, Kafka, object store and dashboard included - **Cross-site correlation** across every sensor and agent reporting to it - **Policy and rule distribution** — access rules, WAF rules and ML models reach agents from here - **ASN and country resolution** — policy written against a network or a country is resolved to IP ranges here, so agents only ever receive addresses - Install from the registry, or build an offline bundle See [Cerebellum](../cerebellum/) for the full installation and operations guide. ### Jailer - Process Sandboxing eBPF LSM-powered application isolation: - **Mandatory Access Control** per process - **File access control** - restrict read/write/execute per role - **Network control** - limit connections and listeners - **Exec control** - prevent unauthorized program execution - Zero performance overhead, no code changes needed ## How It All Connects
Internet north-south traffic Network edge Cerebrum inline hardware sensor Edge 100 Gb/s Max 800 Gb/s runs Synapse in native XDP Or nothing cloud and Kubernetes sites skip the appliance Synapse proxy covers north-south Your servers Synapse agent or proxy mode six engines: capture, detect, classify, map, decide, enforce see the component pages Jailer per-process sandboxing via BPF LSM Your applications web, API, database Processes file, network, exec control Cerebellum — your Kubernetes cluster rules, models and resolved IP ranges down · events and fingerprints up Hardware Software Control plane Isolation Internet north-south traffic Cerebrum inline hardware sensor Edge 100 Gb/s · Max 800 Gb/s cloud sites skip this Synapse agent or proxy mode six engines in one process on your servers Jailer per-process sandboxing Your applications web · API · database policy down, events up Cerebellum your Kubernetes cluster rules, models, resolved ranges Hardware Software Control plane Isolation
Traffic runs left to right. Policy and models come down from the cluster; events and fingerprints go up.
### Traffic Flow 1. **Inbound traffic** hits the Cerebrum sensor at the network edge 2. **XDP filtering** drops known-bad traffic before routing, ahead of the kernel network stack 3. **Cerebrum** performs JA4+ fingerprinting and IDS/IPS inspection inline 4. **Clean traffic** flows to your servers where Synapse applies access rules, WAF, and threat intelligence 5. **Jailer** sandboxes application processes with eBPF LSM — file, network, and exec control 6. **Cerebellum** receives metadata from all sensors and servers, runs ML inference, and pushes updated threat verdicts back ### Deployment Models | Model | Components | Use Case | |-------|-----------|----------| | **Agent Only** | Synapse (agent mode) | Kernel-level firewall on existing servers | | **Proxy** | Synapse (proxy mode) | Reverse proxy with WAF, TLS, load balancing | | **Edge + Server** | Cerebrum + Synapse | Hardware sensor at edge + server protection | | **Full Stack** | Cerebrum + Synapse + Jailer + Cerebellum | Complete Active NDR with AI-driven correlation | ## Synapse Internals Synapse is not one program with a feature list. It is six engines, each named for a part of the brain, arranged in the order a packet meets them. Every one has its own page; this is the map. | | Engine | What it does | |---|---|---| | 1 | [Dendrite](../components/dendrite/) | Captures traffic and computes the ten JA4+ fingerprint types. Capture only — it never blocks | | 2 | [Thalamus](../components/thalamus/) | Matches Suricata-compatible signatures across thirty-plus application protocols, and emits EVE JSON | | 3 | [Cortex](../components/cortex/) | Scores behaviour with five small on-device models, so threats with no signature are still caught | | 4 | [Hippocampus](../components/hippocampus/) | Builds the east-west service graph — what talks to what, and what should | | 5 | [Amygdala](../components/amygdala/) | The rule engine. Roughly eighty match fields, evaluated in order, first match wins | | 6 | [Hillock](../components/hillock/) | Makes the kernel act on the decision — XDP, falling back to nftables then iptables | The split matters for one reason above all: **only what is readable from a packet header is compared on the packet path.** ASN, country, reputation, ML verdicts and workload identity are resolved either ahead of time by Cerebellum or on the host by Amygdala, and what reaches the kernel is an address, a port, a fingerprint or a rate. That is why an elaborate policy costs no more to enforce than a simple one. ### Around them - **Reverse proxy** — TLS termination or SNI passthrough, upstream selection with health checks, and JA4+ fingerprints forwarded to your backends. See [Load Balancing & Proxy](../features/load-balancing.mdx) - **WAF** — a wirefilter expression language over HTTP requests, with allow, block, challenge and rate-limit actions - **Threat intelligence and GeoIP** — local MaxMind databases, refreshed in the background, so lookups never leave the host - **Event export** — every block, drop, allow and notice from any layer as one stream, in JSON Lines, CEF or OTLP. See [Security Event Export](../features/event-export.mdx) ### Configuration Reload Synapse supports runtime reload via `SIGHUP`: ```bash kill -HUP $(cat /var/run/synapse.pid) ``` **Hot-reloadable:** log level, API credentials, upstreams, security rules, GeoIP databases. **Requires restart:** listen addresses, network interfaces, firewall mode, runtime threads, Redis, CAPTCHA, and daemon settings. ## Performance | | | |---|---| | Forwarding capacity | Cerebrum Edge: 100 Gb/s maximum. Cerebrum Max: 800 Gb/s | | Where packets are dropped | At the XDP hook, before routing — no context switch into user space | | Cost per blocked address | A kernel map lookup. It does not grow with the size of the blocklist | | Fingerprint coverage | Ten JA4+ types across TLS, TCP, HTTP, SSH, X.509 and DHCP | | Encrypted traffic analysis | No decryption — classification runs on fingerprints and behaviour | | ML inference | On the host CPU, on models measured in kilobytes. Nothing leaves the machine | Figures are hardware specifications and architectural properties. Per-packet latency depends on your traffic mix, ruleset and hardware, so measure it on your own workload rather than taking a number from a table. --- # Signal API Source: https://docs.gen0sec.com/docs/api/signal-api # Signal API The Signal API allows you to submit security events and signals for threat detection and analysis. ## Base URL ``` https://api.gen0sec.com ``` ## Authentication All API requests require authentication using a Bearer token in the Authorization header: ```bash Authorization: Bearer ``` ## Endpoints ### Submit Signal Event Submit a security signal event for processing. **Endpoint:** `POST /v1/signal` **Request Body:** The endpoint accepts a batch of signal events as an array. You can send one or multiple events in a single request (up to 1000 events per batch). **Single Event Examples:** Block by IP address: ```json [ { "type": "access_rules", "action": "block", "ip": "192.168.1.100", "expiration": 3600, "description": "Blocked due to suspicious activity", "name": "suspicious-ip-block" } ] ``` Block by ASN: ```json [ { "type": "access_rules", "action": "block", "asn": "AS12345", "expiration": 3600, "description": "Blocked ASN due to malicious activity", "name": "suspicious-asn-block" } ] ``` Block by Country: ```json [ { "type": "access_rules", "action": "block", "country": "US", "expiration": 3600, "description": "Blocked country due to threat intelligence", "name": "suspicious-country-block" } ] ``` **Batch Example (Multiple Events):** ```json [ { "type": "access_rules", "action": "block", "ip": "192.168.1.100", "description": "Blocked IP" }, { "type": "access_rules", "action": "block", "asn": "AS12345", "description": "Blocked ASN" }, { "type": "access_rules", "action": "block", "country": "US", "description": "Blocked country" } ] ``` **Parameters:** - `type` (string, required): Event type. Currently supports: `"access_rules"` - `action` (string, required): Action to take. Valid values: `"block"` or `"unblock"` - `ip` (string, optional): IP address to block/unblock (IPv4 or IPv6, CIDR notation supported) - `asn` (string, optional): ASN to block/unblock (format: `AS12345`, must exist in database) - `country` (string, optional): Country code to block/unblock (ISO-3166 Alpha-2 format, e.g., `US`, `GB`, `JP`) - `expiration` (integer, optional): Expiration time in seconds - `description` (string, optional): Description of the signal event - `name` (string, optional): Name identifier for the signal **Note:** - Exactly one of `ip`, `asn`, or `country` must be provided and non-empty per event. You cannot provide multiple fields simultaneously in a single event. - The endpoint accepts a batch of events (array format). Maximum batch size is 1000 events per request. **Response:** The endpoint processes events in batch and returns a summary of the processing results. **Success (200) - All events processed successfully:** ```json { "success": true, "message": "Processed 3 entries, 0 failed" } ``` **Partial Success (206) - Some events failed:** ```json { "success": false, "message": "Processed 2 entries, 1 failed", "errors": [ "Entry 1: Schema validation failed: country: Country must be a valid ISO-3166 Alpha-2 code (e.g., US, GB, JP)" ] } ``` **Error (400):** ```json { "error": "ValidationError", "message": "Validation failed", "details": [ { "field": "action", "message": "must be one of: block, unblock", "value": "invalid" } ], "code": 400 } ``` **Error (401):** ```json { "error": "Unauthorized", "message": "Invalid or missing API key", "code": 401 } ``` ## Signal API v2 `POST /v1/signal` (the flat `access_rules` shape above) is unchanged and remains supported. **v2 adds `POST /v2/signal`**, a typed envelope that also supports **WAF** and **Smart Firewall** rules and can be consumed by Synapse. ### Why v2 The v1 body flattened rule fields to the top level and overloaded `action` (`block`/`unblock`) to mean both *what to do with the rule* and *the rule's enforcement verb*. v2 separates three concerns: | Concern | v1 | v2 | | --- | --- | --- | | Rule family | `type` (always `access_rules`) | `kind`: `access_rule` \| `waf_rule` \| `smart_firewall_rule` | | Operation | `action: block/unblock` | `op`: `upsert` \| `delete` | | Enforcement verb | conflated into `action` | `rule.action` (per kind) | | Identity | implicit (by value) | `rule_ref` (caller-stable id) | ### Endpoint **Endpoint:** `POST /v2/signal` — Bearer auth, JSON array, up to 1000 envelopes per batch. Per-entry results: `200` if all succeed, `206` if any entry fails (the response lists per-entry errors), identical to v1. ### Envelope ```jsonc { "schema_version": 2, // required (defaulted to 2 on /v2/signal) "kind": "waf_rule", // access_rule | waf_rule | smart_firewall_rule "op": "upsert", // upsert | delete "rule_ref": "soar-pb12-sqli-login", // caller-stable id; REQUIRED for waf/smart_firewall "expires_in": 86400, // seconds; 0/omitted = no expiry; ignored on delete "reason": "SOAR playbook PB-12 matched SQLi", "labels": { "playbook": "PB-12" }, // optional structured metadata "rule": { /* kind-specific; omit when op=delete */ } } ``` `rule_ref` is the upsert/delete key and the idempotency key: re-sending the same `rule_ref` updates the existing active rule rather than creating a duplicate. It must match `^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$`. ### Access rule (`kind: access_rule`) Exactly one of `ip`, `asn`, `country` in `target`. `op: delete` replaces v1's `action: "unblock"`. `expires_in` replaces v1 `expiration` (same policy: a `block` with no expiry defaults to one day, any positive value is floored at 60s). `rule_ref` is optional here — without it, v1-style match-by-value applies. ```json [ { "schema_version": 2, "kind": "access_rule", "op": "upsert", "rule_ref": "soar-block-203-0-113-10", "expires_in": 3600, "reason": "Brute force from this host", "rule": { "target": { "ip": "203.0.113.10/32" }, "action": "block" } } ] ``` Remove it later: ```json [ { "schema_version": 2, "kind": "access_rule", "op": "delete", "rule_ref": "soar-block-203-0-113-10" } ] ``` ### WAF rule (`kind: waf_rule`) and Smart Firewall rule (`kind: smart_firewall_rule`) Both use the same expression-rule body. `expression` is a Wirefilter expression and is **syntactically validated at ingest** by the rules-validator service against the scheme for that `kind` (WAF = HTTP L7 fields; Smart Firewall = L3/L4 + JA4 fields, no `http.*`); an unparseable expression — or a WAF field used in a Smart Firewall rule — rejects that entry with a `206` and a precise error. `rule_ref` is required. `rule.type` is **required** for both `waf_rule` and `smart_firewall_rule` and must be one of: | `type` | Routes to | Use for | | --- | --- | --- | | `custom` | **Custom** WAF / Smart Firewall tab | Rules authored by a human or a SOAR playbook | | `generated` | **AI** WAF / Smart Firewall tab | Rules produced by the classifier / automated generation | The UI splits rules across tabs by this value. It is matched case-insensitively and stored lowercase. An empty or missing `rule.type` rejects the entry with a `400` (`"rule.type is required and must be one of custom, generated"`) — there is no silent default, so a forgotten field no longer routes a Custom rule into the AI tab by accident. Action and phase are kind-specific, matching the Synapse enforcement engine: | | `action` (allowed values) | `phase` | | --- | --- | --- | | `waf_rule` | `block`, `challenge`, `ratelimit`, `allow` | optional: `request` \| `response`; auto-detected by Synapse from the expression when omitted | | `smart_firewall_rule` | `block`, `allow` | not allowed (L3/L4 is pre-connection) | `phase` is carried through into `waf_rules.config` (under the `phase` key) — there is no separate column. `action` and `phase` are matched case-insensitively and stored lowercase. ```json [ { "schema_version": 2, "kind": "waf_rule", "op": "upsert", "rule_ref": "soar-pb12-sqli-login", "reason": "SOAR playbook PB-12 matched SQLi", "rule": { "name": "Block SQLi on /login", "type": "custom", "expression": "http.request.uri.path eq \"/login\" and http.request.body.raw contains \"union select\"", "action": "block", "phase": "request", "order": 100, "short_description": "SQLi guard", "config": { "sensitivity": "high" } } } ] ``` `smart_firewall_rule` uses the same body shape — including the required `type` (`custom`/`generated`) — but with `action` limited to `block`/`allow`, no `phase`, and an L3/L4 + JA4 expression (e.g. `ip.src in 192.0.2.0/24 and ja4t == "64240_2-4-8-1-3_1460_10"`) — `http.*` fields are rejected for this kind. ### Response Same structure as v1: ```json { "success": true, "message": "Processed 1 entries, 0 failed", "errors": [] } ``` A failed entry (e.g. bad expression, missing `rule_ref`, invalid target) yields HTTP `206` with the per-entry reason in `errors`. ### Backward compatibility `/v1/signal`, its body shape, and its Kafka format are untouched — existing Synapse and SOAR integrations are unaffected. v2 flows on the same Kafka topic with a `schema_version` header; the consumer routes v1 (absent/`1`) to the legacy access-rule path and v2 to the typed path. WAF / Smart Firewall upserts land in `waf_rules` / `smart_firewall_rules`; access rules continue to land in `access_rules`. ## Status codes The two success codes matter most, and are covered above: `200` when every entry in the batch was accepted, `206` when some were and the rest come back with per-entry reasons. The rest are the ones a client has to handle. | Code | Meaning | What a client should do | |---|---|---| | `200` | Every entry processed | Continue | | `206` | Some entries failed | Read `errors` for the per-entry reason; do not retry the whole batch | | `400` | Invalid JSON, data, or batch size | Fix the payload. Retrying will not help | | `401` | Invalid API key, or missing org/workspace context | Check credentials | | `402` | Subscription required | A billing state, not a transient one. Stop and resolve it | | `403` | Forbidden | The key is valid but not permitted here | | `405` | Method not allowed | Use `POST` | | `429` | Too many requests | **Back off and retry.** This is the one worth implementing properly | | `500` | Server error | Retry with backoff | `402` and `429` are the two that catch people out. A `402` will not clear on its own — retrying a batch against an inactive subscription just burns the batch. A `429` will clear, so a client without backoff turns a brief limit into sustained failure and dropped telemetry. `v2` returns `200`, `206`, `400` and `401`. ## Interactive Documentation Interactive API documentation is available at: ``` https://api.gen0sec.com/docs/signal/swagger/ ``` ## Example Usage ### Block by IP Address ```bash curl -X POST https://api.gen0sec.com/v1/signal \ -H "Authorization: Bearer your-api-key" \ -H "Content-Type: application/json" \ -d '[ { "type": "access_rules", "action": "block", "ip": "192.168.1.100", "description": "Blocked due to suspicious activity" } ]' ``` ### Block by ASN ```bash curl -X POST https://api.gen0sec.com/v1/signal \ -H "Authorization: Bearer your-api-key" \ -H "Content-Type: application/json" \ -d '[ { "type": "access_rules", "action": "block", "asn": "AS12345", "description": "Blocked ASN due to malicious activity" } ]' ``` ### Block by Country ```bash curl -X POST https://api.gen0sec.com/v1/signal \ -H "Authorization: Bearer your-api-key" \ -H "Content-Type: application/json" \ -d '[ { "type": "access_rules", "action": "block", "country": "US", "description": "Blocked country due to threat intelligence" } ]' ``` ### Batch Request (Multiple Events) ```bash curl -X POST https://api.gen0sec.com/v1/signal \ -H "Authorization: Bearer your-api-key" \ -H "Content-Type: application/json" \ -d '[ { "type": "access_rules", "action": "block", "ip": "192.168.1.100", "description": "Blocked IP" }, { "type": "access_rules", "action": "block", "asn": "AS12345", "description": "Blocked ASN" }, { "type": "access_rules", "action": "block", "country": "US", "description": "Blocked country" } ]' ``` ### Python ```python import requests url = "https://api.gen0sec.com/v1/signal" headers = { "Authorization": "Bearer your-api-key", "Content-Type": "application/json" } # Block by IP (single event in batch) data = [ { "type": "access_rules", "action": "block", "ip": "192.168.1.100", "description": "Blocked due to suspicious activity" } ] # Block by ASN (single event in batch) data = [ { "type": "access_rules", "action": "block", "asn": "AS12345", "description": "Blocked ASN due to malicious activity" } ] # Block by Country (single event in batch) data = [ { "type": "access_rules", "action": "block", "country": "US", "description": "Blocked country due to threat intelligence" } ] # Batch request (multiple events) data = [ { "type": "access_rules", "action": "block", "ip": "192.168.1.100", "description": "Blocked IP" }, { "type": "access_rules", "action": "block", "asn": "AS12345", "description": "Blocked ASN" }, { "type": "access_rules", "action": "block", "country": "US", "description": "Blocked country" } ] response = requests.post(url, json=data, headers=headers) print(response.json()) ``` ### Go ```go package main import ( "bytes" "encoding/json" "net/http" ) func main() { url := "https://api.gen0sec.com/v1/signal" // Block by IP (single event in batch) data := []map[string]interface{}{ { "type": "access_rules", "action": "block", "ip": "192.168.1.100", "description": "Blocked due to suspicious activity", }, } // Block by ASN (single event in batch) data = []map[string]interface{}{ { "type": "access_rules", "action": "block", "asn": "AS12345", "description": "Blocked ASN due to malicious activity", }, } // Block by Country (single event in batch) data = []map[string]interface{}{ { "type": "access_rules", "action": "block", "country": "US", "description": "Blocked country due to threat intelligence", }, } // Batch request (multiple events) data = []map[string]interface{}{ { "type": "access_rules", "action": "block", "ip": "192.168.1.100", "description": "Blocked IP", }, { "type": "access_rules", "action": "block", "asn": "AS12345", "description": "Blocked ASN", }, { "type": "access_rules", "action": "block", "country": "US", "description": "Blocked country", }, } jsonData, _ := json.Marshal(data) req, _ := http.NewRequest("POST", url, bytes.NewBuffer(jsonData)) req.Header.Set("Authorization", "Bearer your-api-key") req.Header.Set("Content-Type", "application/json") client := &http.Client{} resp, _ := client.Do(req) defer resp.Body.Close() } ``` ## Rate Limits API rate limits apply to prevent abuse. Contact support if you need higher limits. ## Support For API support, visit: - Discord: https://discord.com/invite/jzsW5Q6s9q - Email: support@gen0sec.com --- # Threat API Source: https://docs.gen0sec.com/docs/api/threat-api # Threat API The Threat API provides comprehensive threat intelligence analysis for IP addresses, including risk scores, confidence levels, threat categories, and remediation advice. ## Base URL ``` https://api.gen0sec.com ``` ## Authentication All API requests require authentication using a Bearer token in the Authorization header: ```bash Authorization: Bearer ``` ## Endpoints ### Get Threat Analysis Analyze an IP address and retrieve comprehensive threat intelligence. **Endpoint:** `GET /v1/threat` **Query Parameters:** - `ip` (string, required): IP address to analyze (IPv4 or IPv6) - Example: `57.141.0.51` **Response:** **Success (200):** ```json { "schema_version": "1.0", "tenant_id": "public", "ip": "57.141.0.51", "intel": { "score": 99, "confidence": 0.96, "score_version": "2025-09-01", "categories": ["botnet", "brute_force", "c2", "malware", "scanner", "spam"], "tags": ["default"], "first_seen": "2025-08-31T01:08:03Z", "last_seen": "2025-09-08T02:56:57Z", "source_count": 3, "reason_code": "MULTI_RECENT_SIGNALS", "reason_summary": "Indicators >=70 from 3 sources in last 7 days", "rule_id": "00000000-0000-0000-0000-000000000000" }, "context": { "asn": 64500, "org": "ExampleNet", "ip_version": 4, "geo": { "country": "United States", "country_code": "US" } }, "advice": "challenge", "ttl_s": 43200, "generated_at": "2025-09-08T03:00:00Z" } ``` **Response Fields:** - `intel.score` (integer, 0-100): Threat risk score - `intel.confidence` (float, 0-1): Confidence level of the analysis - `intel.categories` (array): Threat categories (e.g., botnet, malware, scanner) - `intel.tags` (array): Free-form labels distinct from categories - `intel.first_seen` (string): ISO 8601 timestamp of first detection - `intel.last_seen` (string): ISO 8601 timestamp of most recent detection - `intel.source_count` (integer): Number of threat intelligence sources - `intel.reason_code` (string): Code explaining the threat assessment - `intel.reason_summary` (string): Human-readable summary of the threat - `context.asn` (integer): Autonomous System Number - `context.org` (string): Organization name - `context.geo.country` (string): Country name - `context.geo.country_code` (string): ISO country code - `advice` (string): Recommended action: `"allow"`, `"block"`, or `"challenge"` - `ttl_s` (integer): Cache TTL in seconds **No Data Found (200):** When no threat data is available, the API returns: ```json { "schema_version": "1.0", "tenant_id": "public", "ip": "192.168.1.1", "intel": { "score": 0, "confidence": 0.0, "score_version": "2025-09-01", "categories": [], "tags": [], "first_seen": "", "last_seen": "", "source_count": 0, "reason_code": "NO_DATA", "reason_summary": "No threat data available", "rule_id": "none" }, "context": { "asn": 0, "org": "", "ip_version": 4, "geo": { "country": "", "country_code": "" } }, "advice": "allow", "ttl_s": 60, "generated_at": "2025-09-08T03:00:00Z" } ``` **Error (400):** ```json { "success": false, "error": "Missing IP address parameter", "details": { "required_param": "ip", "example": "/v1/threat?ip=192.168.1.1" } } ``` **Error (401):** ```json { "success": false, "error": "Unauthorized - invalid or missing API key" } ``` **Error (402):** ```json { "success": false, "error": "Payment required - subscription required" } ``` **Error (429):** ```json { "success": false, "error": "Too many requests - rate limit exceeded" } ``` ### Get Access Rules Retrieve the access rules configured for your workspace — one by ID, or all of them. **Endpoint:** `GET /v1/access-rules/{id}` The `id` is optional. Omit it to list every rule. **Query Parameters:** - `resolve` (boolean, optional): resolve country codes and ASN numbers to the IP ranges they currently cover. **Why `resolve` matters.** A rule you wrote against a country or an ASN is stored that way, but the agent only ever enforces on addresses — the ranges are resolved for it upstream. Setting `resolve=true` shows you what a rule expands to, which is the difference between "block AS15169" and the several hundred prefixes that actually becomes. With `resolve=false` (the default) the `country` and `asn` fields come back as the plain values you set. **Response:** a paginated envelope — `success`, `data`, `page`, `limit`, `total`. | Status | Meaning | |---|---| | `200` | The rule, or the list | | `400` | Malformed request | | `404` | No rule with that ID | | `500` | Server error | ```bash # every rule, as written curl -u "api:$API_KEY" "https://api.gen0sec.com/v1/access-rules" # one rule, expanded to the addresses it currently covers curl -u "api:$API_KEY" "https://api.gen0sec.com/v1/access-rules/{id}?resolve=true" ``` See [Access Rules](../features/access-rules.mdx) for how these are enforced, and why the resolution happens before the rule reaches the agent. ## Interactive Documentation Interactive API documentation is available at: ``` https://api.gen0sec.com/docs/threat/swagger/ ``` ## Example Usage ### cURL ```bash curl -X GET "https://api.gen0sec.com/v1/threat?ip=57.141.0.51" \ -H "Authorization: Bearer your-api-key" ``` ### Python ```python import requests url = "https://api.gen0sec.com/v1/threat" headers = { "Authorization": "Bearer your-api-key" } params = { "ip": "57.141.0.51" } response = requests.get(url, params=params, headers=headers) threat_data = response.json() print(f"Threat Score: {threat_data['intel']['score']}") print(f"Categories: {', '.join(threat_data['intel']['categories'])}") print(f"Advice: {threat_data['advice']}") ``` ### Go ```go package main import ( "encoding/json" "fmt" "net/http" "net/url" ) func main() { baseURL := "https://api.gen0sec.com/v1/threat" params := url.Values{} params.Add("ip", "57.141.0.51") req, _ := http.NewRequest("GET", baseURL+"?"+params.Encode(), nil) req.Header.Set("Authorization", "Bearer your-api-key") client := &http.Client{} resp, _ := client.Do(req) defer resp.Body.Close() var threatData map[string]interface{} json.NewDecoder(resp.Body).Decode(&threatData) fmt.Printf("Threat Score: %v\n", threatData["intel"].(map[string]interface{})["score"]) } ``` ## Threat Categories Common threat categories include: - `botnet` - IP is part of a botnet - `brute_force` - IP involved in brute force attacks - `c2` - Command and control server - `malware` - Associated with malware distribution - `scanner` - Port scanning activity - `spam` - Spam source - `phishing` - Phishing activities - `exploit` - Exploit attempts ## Rate Limits API rate limits apply to prevent abuse. Contact support if you need higher limits. ## Caching Responses include a `ttl_s` (time-to-live in seconds) field indicating how long you should cache the result. For IPs with no threat data, the TTL is typically 60 seconds. For IPs with threat data, the TTL is typically 43200 seconds (12 hours). ## Support For API support, visit: - Discord: https://discord.com/invite/jzsW5Q6s9q - Email: support@gen0sec.com --- # Threat Feeds API Source: https://docs.gen0sec.com/docs/api/threat-feeds-api # Threat Feeds API The Threat Feeds API provides a plain text feed of IP addresses and IP ranges from block rules (threats). Countries and ASNs are automatically resolved to IP ranges, making it suitable for integration with firewalls, intrusion detection systems, and other security tools. ## Base URL ``` https://api.gen0sec.com ``` ## Authentication The API supports two authentication methods: ### Bearer Token Authentication ```bash Authorization: Bearer ``` ### Basic Authentication Basic Authentication uses HTTP Basic Auth with: - **Username**: `api` (literal string) - **Password**: Your API key The credentials are base64-encoded in the format `api:your-api-key`. **Manual Encoding:** ```bash # Encode credentials manually echo -n "api:your-api-key" | base64 # Output: YXBpOnlvdXItYXBpLWtleQ== ``` **Using cURL:** cURL automatically handles Basic Auth encoding when using the `-u` flag: ```bash curl -u "api:your-api-key" "https://api.gen0sec.com/v1/threat/feeds" ``` **Using Authorization Header:** ```bash # Base64 encode: api:your-api-key Authorization: Basic YXBpOnlvdXItYXBpLWtleQ== ``` ## Endpoints ### Get Threat Feed Retrieve a workspace's access rules as an appliance-consumable list. **Endpoint:** `GET /v1/threat/feeds` **Response Format:** Plain text (`text/plain`) by default; `text/csv` for the Check Point dialect. The feed includes: - **Direct IP addresses** from rules (normalized to `/32` for IPv4 or `/128` for IPv6) - **IP ranges** resolved from countries - **IP ranges** resolved from ASNs Networks are deduplicated, **coalesced** (adjacent and overlapping prefixes are merged into the smallest equivalent set) and sorted numerically, IPv4 first. Coalescing is lossless: the merged list covers exactly the same addresses. **Query parameters:** | Parameter | Values | Default | Description | |---|---|---|---| | `format` | `plain`, `cisco`, `checkpoint`, `nftables` | `plain` | Output dialect | | `family` | `v4`, `v6`, `both` | `both` | Address family to publish | | `action` | `block`, `allow` | `block` | Publish the deny list or the never-block list | | `checksum` | `md5` | *(unset)* | Return the feed's checksum instead of the feed | `plain` also accepts `txt`, `fortios` and `fortigate` as aliases. **Response Example:** ``` 2.0.0.0/8 10.0.0.0/8 192.168.1.0/24 2001:db8::/32 ``` ### Get a Named Feed Publish a single access-rules list rather than every rule in the workspace, so one workspace can serve several feeds to different appliances. **Endpoint:** `GET /v1/threat/feeds/{list}` `{list}` is the list's UUID or its name. All the query parameters above apply unchanged. ```bash curl -u api:$API_KEY https://api.gen0sec.com/v1/threat/feeds/edge-blocklist curl -u api:$API_KEY "https://api.gen0sec.com/v1/threat/feeds/edge-blocklist?format=cisco" ``` A list can store its own defaults (`format`, `family`, `action`, `max_entries`, `confidence`, `severity`), which keeps the appliance URL bare. An explicit query parameter always wins over a stored default. If a name matches more than one list the request returns **409** rather than guessing; address the list by UUID to disambiguate. ## Conditional Requests Every response carries an `ETag` and a `Last-Modified`. Send them back and an unchanged feed costs a single empty round trip instead of a full re-download — which matters when a fleet of appliances polls on a timer. ```bash # First fetch curl -sD headers.txt -u api:$API_KEY https://api.gen0sec.com/v1/threat/feeds -o feed.txt ETAG=$(grep -i '^etag:' headers.txt | cut -d' ' -f2 | tr -d '\r') # Subsequent fetches: 304 Not Modified, empty body curl -s -o /dev/null -w '%{http_code}\n' \ -u api:$API_KEY -H "If-None-Match: $ETAG" \ https://api.gen0sec.com/v1/threat/feeds ``` `If-None-Match` takes precedence over `If-Modified-Since`; if you send both, the entity-tag decides. **Response headers:** | Header | Description | |---|---| | `ETag` | Strong validator over the response body. Differs per format, family, action and content encoding. | | `Last-Modified` | When the current feed content first appeared | | `Cache-Control` | `private, max-age=30, must-revalidate` | | `Vary` | `Accept-Encoding, Authorization` | | `X-Feed-Entry-Count` | Number of networks in the feed | | `X-Feed-Entry-Limit` | The feed's entry cap | ## Compression Send `Accept-Encoding: gzip` to receive a gzip-encoded body. Without the header the response is uncompressed and byte-identical to what a plain client has always received, so existing integrations are unaffected. Because a compressed body is a distinct representation, it carries its own `ETag`. Do not compare an entity-tag obtained with gzip against one obtained without it. ## Checksum Sidecar `?checksum=md5` returns the MD5 digest of the feed body instead of the feed. This is what Cisco Secure Firewall's optional MD5 URL expects: the appliance fetches the small digest and skips the feed download entirely when it has not changed. ```bash curl -u api:$API_KEY "https://api.gen0sec.com/v1/threat/feeds?checksum=md5" # 5d41402abc4b2a76b9719d911017c592 ``` The digest is computed over the exact bytes the feed endpoint returns for the **same parameters**, so keep them consistent between the two URLs — `?format=cisco&checksum=md5` describes `?format=cisco`, not the default feed. ## Size Limits A feed is capped at **131,072 entries** by default, matching the ceiling FortiOS enforces on external resources. A single country rule can expand well past that: `US` alone is roughly 420,000 networks before coalescing. If a feed exceeds its cap the API returns **413** with no feed content, rather than a truncated list. This is deliberate — the appliance keeps the last list it fetched successfully, so enforcement stays complete instead of silently becoming partial. ```json { "success": false, "error": "Feed exceeds the maximum number of entries", "details": { "entries": 421824, "limit": 131072, "hint": "narrow the rules (a single country rule can expand to hundreds of thousands of networks) or raise the feed's max_entries" } } ``` `X-Feed-Entry-Count` and `X-Feed-Entry-Limit` are set on the 413 as well as on successful responses, so you can monitor headroom before a feed hits the wall. Raise a specific feed's ceiling with `max_entries` in its list configuration. ## Output Formats ### `plain` (default) One network per line. Suitable for FortiGate external resources, Palo Alto DBL/EBL, and anything that expects a bare list. ``` 10.0.0.0/8 192.168.1.0/24 ``` ### `cisco` One entry per line with a `#` comment header. Cisco Secure Firewall ignores comment lines. ``` # edge-blocklist # perimeter deny list # action: block # family: both # generated: 2026-09-01T12:00:00Z # entries: 2 10.0.0.0/8 192.168.1.0/24 ``` ### `checkpoint` CSV for the Check Point R81+ Custom Intelligence Feed, with the columns `value, type, confidence, severity` and no header row. Host routes are emitted as `IP`; wider networks as an `IP Range` in `first-last` form. ``` 203.0.113.7,IP,high,high 10.0.0.0-10.255.255.255,IP Range,low,medium ``` `confidence` and `severity` come from the individual rule's configuration where set, then the feed's configured default, then `high`. Networks carrying different values are never merged together. ### `nftables` An element list ready to splice into a set definition. IPv4 and IPv6 need separate nftables sets, so pair this with `family`. ``` elements = { 10.0.0.0/8, 192.168.1.0/24 } ``` ## Errors **Error (400)** — unsupported `format`, `family` or `action`, or a malformed request: ```json { "success": false, "error": "Unsupported format", "details": { "supported": ["plain", "cisco", "checkpoint", "nftables"] } } ``` **Error (401)** — invalid or missing API key: ```json { "success": false, "error": "Unauthorized - invalid or missing API key", "details": {} } ``` **Error (404)** — no such access-rules list. **Error (409)** — the list name matches more than one list; use the UUID. **Error (413)** — the feed exceeds its entry limit (see [Size Limits](#size-limits)). **Error (500)**: ```json { "success": false, "error": "Internal server error", "details": {} } ``` ## Features - **Automatic Resolution**: Countries and ASNs are resolved to IP ranges using MaxMind GeoIP databases - **Coalescing**: Adjacent and overlapping networks are merged losslessly into the smallest equivalent set - **Deduplication**: Duplicate addresses and ranges are removed - **Sorted Output**: Numeric ordering, IPv4 before IPv6, for stable, diffable output - **Conditional GET**: `ETag` / `Last-Modified` support so unchanged feeds cost one empty round trip - **Compression**: Optional gzip via `Accept-Encoding` - **Checksum Sidecar**: `?checksum=md5` for appliances that check a digest before downloading - **Multiple Dialects**: Plain, Cisco Secure Firewall, Check Point IoC CSV, and nftables - **Named Feeds**: Publish several independently-configured feeds from one workspace - **Allow Feeds**: Publish a never-block list with `action=allow` - **Expiry**: Rules are filtered on `expires_at` at query time, so a lapsed rule stops being served immediately - **Caching**: Responses are cached server-side for 30 seconds - **IPv4 and IPv6 Support**: Including per-family feeds via `family` ## Interactive Documentation Interactive API documentation is available at: ``` https://api.gen0sec.com/docs/feeds/swagger/ ``` ## Example Usage ### cURL with Bearer Token ```bash curl -X GET "https://api.gen0sec.com/v1/threat/feeds" \ -H "Authorization: Bearer your-api-key" \ -H "Accept: text/plain" ``` ### cURL with Basic Authentication ```bash curl -X GET "https://api.gen0sec.com/v1/threat/feeds" \ -u "api:your-api-key" \ -H "Accept: text/plain" ``` ### Python **Using Bearer Token:** ```python import requests url = "https://api.gen0sec.com/v1/threat/feeds" headers = { "Authorization": "Bearer your-api-key", "Accept": "text/plain" } response = requests.get(url, headers=headers) if response.status_code == 200: # Process the IP list ip_list = response.text.strip().split('\n') for ip in ip_list: print(f"Block IP: {ip}") else: print(f"Error: {response.status_code} - {response.text}") ``` **Using Basic Auth:** ```python import requests from requests.auth import HTTPBasicAuth url = "https://api.gen0sec.com/v1/threat/feeds" response = requests.get( url, auth=HTTPBasicAuth("api", "your-api-key"), headers={"Accept": "text/plain"} ) if response.status_code == 200: ip_list = response.text.strip().split('\n') for ip in ip_list: print(f"Block IP: {ip}") else: print(f"Error: {response.status_code} - {response.text}") ``` **Using Basic Auth with Manual Encoding:** ```python import requests import base64 url = "https://api.gen0sec.com/v1/threat/feeds" api_key = "your-api-key" # Encode credentials credentials = f"api:{api_key}" encoded = base64.b64encode(credentials.encode()).decode() headers = { "Authorization": f"Basic {encoded}", "Accept": "text/plain" } response = requests.get(url, headers=headers) if response.status_code == 200: ip_list = response.text.strip().split('\n') for ip in ip_list: print(f"Block IP: {ip}") ``` ### Go **Using Bearer Token:** ```go package main import ( "bufio" "fmt" "net/http" "strings" ) func main() { url := "https://api.gen0sec.com/v1/threat/feeds" req, _ := http.NewRequest("GET", url, nil) req.Header.Set("Authorization", "Bearer your-api-key") req.Header.Set("Accept", "text/plain") client := &http.Client{} resp, _ := client.Do(req) defer resp.Body.Close() scanner := bufio.NewScanner(resp.Body) for scanner.Scan() { ip := strings.TrimSpace(scanner.Text()) if ip != "" { fmt.Printf("Block IP: %s\n", ip) } } } ``` **Using Basic Auth:** ```go package main import ( "bufio" "encoding/base64" "fmt" "net/http" "strings" ) func main() { url := "https://api.gen0sec.com/v1/threat/feeds" apiKey := "your-api-key" req, _ := http.NewRequest("GET", url, nil) // Encode credentials for Basic Auth credentials := fmt.Sprintf("api:%s", apiKey) encoded := base64.StdEncoding.EncodeToString([]byte(credentials)) req.Header.Set("Authorization", fmt.Sprintf("Basic %s", encoded)) req.Header.Set("Accept", "text/plain") client := &http.Client{} resp, _ := client.Do(req) defer resp.Body.Close() scanner := bufio.NewScanner(resp.Body) for scanner.Scan() { ip := strings.TrimSpace(scanner.Text()) if ip != "" { fmt.Printf("Block IP: %s\n", ip) } } } ``` ### Bash Script for Firewall Integration **Using Bearer Token:** ```bash #!/bin/bash # Fetch threat feed THREAT_FEED=$(curl -s -X GET "https://api.gen0sec.com/v1/threat/feeds" \ -H "Authorization: Bearer your-api-key" \ -H "Accept: text/plain") # Process each IP/range echo "$THREAT_FEED" | while read -r ip_range; do if [ -n "$ip_range" ]; then # Add to firewall rules (example for iptables) # iptables -A INPUT -s "$ip_range" -j DROP echo "Blocking: $ip_range" fi done ``` **Using Basic Auth:** ```bash #!/bin/bash API_KEY="your-api-key" # Fetch threat feed using Basic Auth THREAT_FEED=$(curl -s -X GET "https://api.gen0sec.com/v1/threat/feeds" \ -u "api:${API_KEY}" \ -H "Accept: text/plain") # Process each IP/range echo "$THREAT_FEED" | while read -r ip_range; do if [ -n "$ip_range" ]; then # Add to firewall rules (example for iptables) # iptables -A INPUT -s "$ip_range" -j DROP echo "Blocking: $ip_range" fi done ``` ## Use Cases ### Firewall Integration Use the feed to automatically update firewall rules: **Using Bearer Token:** ```bash # Download feed curl -X GET "https://api.gen0sec.com/v1/threat/feeds" \ -H "Authorization: Bearer your-api-key" \ -o /tmp/threat-feeds.txt # Update firewall rules (example) while IFS= read -r ip_range; do iptables -A INPUT -s "$ip_range" -j DROP done < /tmp/threat-feeds.txt ``` **Using Basic Auth:** ```bash # Download feed using Basic Auth curl -X GET "https://api.gen0sec.com/v1/threat/feeds" \ -u "api:your-api-key" \ -o /tmp/threat-feeds.txt # Update firewall rules (example) while IFS= read -r ip_range; do iptables -A INPUT -s "$ip_range" -j DROP done < /tmp/threat-feeds.txt ``` ### Intrusion Detection Systems Import the feed into your IDS/IPS: **Using Bearer Token:** ```bash # Download and format for Snort curl -X GET "https://api.gen0sec.com/v1/threat/feeds" \ -H "Authorization: Bearer your-api-key" | \ sed 's/^/var THREAT_IPS [/' | \ sed 's/$/]/' > /etc/snort/threat-feeds.rules ``` **Using Basic Auth:** ```bash # Download and format for Snort using Basic Auth curl -X GET "https://api.gen0sec.com/v1/threat/feeds" \ -u "api:your-api-key" | \ sed 's/^/var THREAT_IPS [/' | \ sed 's/$/]/' > /etc/snort/threat-feeds.rules ``` ### Scheduled Updates Set up a cron job to regularly update your threat feeds: ```bash # Add to crontab (runs every hour) 0 * * * * /usr/local/bin/update-threat-feeds.sh ``` ## Firewall Platform Integration The Threat Feeds API is designed to integrate seamlessly with major firewall platforms. Below are implementation examples for popular enterprise firewalls. ### Palo Alto Networks Palo Alto Networks firewalls support Dynamic Block Lists (DBL) or External Block Lists (EBL) that can fetch threat feeds from external URLs. **Configuration Steps:** 1. **Navigate to Objects > Dynamic Block List** 2. **Click Add** to create a new block list 3. **Configure the feed:** - **Name**: `Gen0Sec Threat Feed` - **Source URL**: `https://api.gen0sec.com/v1/threat/feeds` - **Authentication**: Configure Basic Auth with username `api` and your API key as password **Using Basic Auth:** ```bash # Test the feed URL first curl -u "api:your-api-key" "https://api.gen0sec.com/v1/threat/feeds" -o /tmp/dbl.txt # Verify the file is accessible cat /tmp/dbl.txt ``` **PAN-OS Configuration:** 1. Go to **Objects > Dynamic Block List** 2. Click **Add** 3. Enter the following: - **Name**: `Gen0Sec-Threat-Feed` - **Source URL**: `https://api.gen0sec.com/v1/threat/feeds` - **Source Type**: `URL` 4. Click **Test Source URL** to verify connectivity 5. Configure authentication: - **Authentication Type**: `Basic` - **Username**: `api` - **Password**: Your Gen0Sec API key **Apply to Security Policy:** 1. Navigate to **Policies > Security** 2. Edit your security policy 3. Add the Dynamic Block List to the **Source** or **Destination** field 4. Set action to **Deny** For more details, see the [Palo Alto Networks documentation](https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClFOCA0). ### Cisco ASA FirePOWER Services Cisco FirePOWER supports custom IP reputation feeds that can be configured through ASDM or the FirePOWER Management Center. **Configuration via ASDM:** 1. **Navigate to Configuration > ASA FirePOWER Configuration > Object Management > Security Intelligence > Network Lists and Feeds** 2. **Click Add Network Lists and Feeds** 3. **Configure the feed:** - **Name**: `Gen0Sec Threat Feed` - **Type**: Select `Feed` from dropdown - **Feed URL**: `https://api.gen0sec.com/v1/threat/feeds` - **Update Frequency**: Set to desired interval (e.g., every 1 minute) - **Authentication**: Configure Basic Auth with username `api` and your API key **Using Basic Auth:** The feed URL should include authentication. You can use Basic Auth: ```bash # Test the feed curl -u "api:your-api-key" "https://api.gen0sec.com/v1/threat/feeds" ``` **Configure Security Intelligence:** 1. Navigate to **Configuration > ASA FirePOWER Configuration > Policies > Access Control Policy** 2. Select the **Security Intelligence** tab 3. Move `Gen0Sec Threat Feed` to the **Blacklist** column 4. Enable logging if desired 5. Click **Store ASA FirePOWER Changes** **Deploy Policy:** 1. Click **Deploy** and select **Deploy FirePOWER Changes** 2. Monitor deployment status in **Monitoring > ASA Firepower Monitoring > Task Status** **Manual Feed Upload (Alternative):** If you prefer to upload a file instead: ```bash # Download the feed curl -u "api:your-api-key" "https://api.gen0sec.com/v1/threat/feeds" -o gen0sec-feeds.txt # Upload via ASDM: # 1. Navigate to Object Management > Security Intelligence > Network Lists and Feeds # 2. Click Add Network Lists and Feeds # 3. Type: Select "List" # 4. Upload List: Browse and select gen0sec-feeds.txt ``` For more details, see the [Cisco ASA FirePOWER documentation](https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-firepower-services/200449-Configure-IP-Blacklisting-Using-Cisco-S.html). ### Fortinet FortiGate FortiGate firewalls support External Block Lists (EBL) that can fetch threat intelligence feeds from external sources. **Configuration via Web UI:** 1. **Navigate to Security Fabric > External Connectors > External Block List** 2. **Click Create New** 3. **Configure the feed:** - **Name**: `Gen0Sec Threat Feed` - **Type**: `IP Address` - **Source**: `URL` - **URL**: `https://api.gen0sec.com/v1/threat/feeds` - **Update Frequency**: Set desired interval (e.g., `1 minute`) **Authentication Configuration:** FortiGate supports HTTP Basic Authentication for external feeds: 1. In the External Block List configuration, enable **Authentication** 2. **Authentication Type**: `Basic` 3. **Username**: `api` 4. **Password**: Your Gen0Sec API key **CLI Configuration:** ```bash config system external-resource edit "Gen0Sec-Threat-Feed" set type address set category 0 set resource "https://api.gen0sec.com/v1/threat/feeds" set username "api" set password "your-api-key" set update-interval 3600 next end ``` **Apply to Firewall Policy:** 1. Navigate to **Policy & Objects > Firewall Policy** 2. Edit your firewall policy 3. In **Source** or **Destination**, add the External Block List 4. Set action to **Deny** or **Block** **Verify Feed Status:** ```bash # Check feed status diagnose external-resource list # Force update execute external-resource update Gen0Sec-Threat-Feed ``` For more details, see the [Fortinet FortiGate documentation](https://docs.fortinet.com/document/fortigate/7.0.2/administration-guide/891236/external-blocklist-policy). ### Sophos Firewall Sophos Firewall supports third-party threat feeds that can be integrated into Active Threat Response policies. **Configuration Steps:** 1. **Navigate to Threat Protection > Active Threat Response > Configure Feeds** 2. **Click Add Feed** 3. **Configure the feed:** - **Feed Name**: `Gen0Sec Threat Feed` - **Feed Type**: `IP Address List` - **Feed URL**: `https://api.gen0sec.com/v1/threat/feeds` - **Update Frequency**: Set to desired interval (e.g., `Every 1 minute`) **Authentication:** Sophos Firewall supports HTTP Basic Authentication: 1. In the feed configuration, enable **Authentication** 2. **Authentication Type**: `Basic` 3. **Username**: `api` 4. **Password**: Your Gen0Sec API key **Apply to Active Threat Response Policy:** 1. Navigate to **Threat Protection > Active Threat Response > Policies** 2. Create or edit a policy 3. Add the `Gen0Sec Threat Feed` to the **Blocked IPs** section 4. Configure action (Block, Drop, or Log) **CLI Configuration (Alternative):** ```bash # Configure via CLI system threat-feed add name "Gen0Sec-Threat-Feed" \ type ip \ url "https://api.gen0sec.com/v1/threat/feeds" \ username "api" \ password "your-api-key" \ update-interval 3600 ``` **Verify Feed:** 1. Navigate to **Threat Protection > Active Threat Response > Configure Feeds** 2. Check the feed status and last update time 3. Click **Update Now** to manually refresh the feed For more details, see the [Sophos Firewall documentation](https://docs.sophos.com/nsg/sophos-firewall/21.0/help/en-us/webhelp/onlinehelp/AdministratorHelp/ActiveThreatResponse/ConfigureFeeds/ThirdPartyThreatFeeds/index.html). ### Cisco Secure Firewall Cisco Secure Firewall (FMC) fetches a network feed over HTTPS and can optionally check an MD5 URL first, downloading the list only when the digest has changed. **Configuration Steps:** 1. **Navigate to Objects > Object Management > Security Intelligence > Network Lists and Feeds** 2. **Click Add Network Lists and Feeds** 3. **Configure the feed:** - **Name**: `Gen0Sec Threat Feed` - **Type**: `Feed` - **Feed URL**: `https://api.gen0sec.com/v1/threat/feeds?format=cisco` - **MD5 URL**: `https://api.gen0sec.com/v1/threat/feeds?format=cisco&checksum=md5` - **Update Frequency**: 30 minutes 4. **Save and deploy**, then add the feed to an Access Control policy's Security Intelligence tab. Authenticate with Basic auth (`api:`). :::tip Keep the query parameters identical between the two URLs. The MD5 describes the feed built with the *same* parameters, so pointing the digest at `?format=cisco&checksum=md5` while the feed URL omits `format` would compare against a different representation and defeat the optimisation. ::: ```bash # Verify both URLs before configuring the appliance curl -u api:$API_KEY "https://api.gen0sec.com/v1/threat/feeds?format=cisco" -o feed.txt curl -u api:$API_KEY "https://api.gen0sec.com/v1/threat/feeds?format=cisco&checksum=md5" md5sum feed.txt # must match ``` ### Check Point R81+ Check Point R81 and later support Custom Intelligence Feeds, which ingest indicators from a CSV URL. **Configuration Steps:** 1. **In SmartConsole, open Security Policies > Threat Prevention > Custom Policy Tools > Indicators** 2. **Add a new Custom Intelligence Feed** 3. **Configure the feed:** - **Feed URL**: `https://api.gen0sec.com/v1/threat/feeds?format=checkpoint` - **Format**: CSV - **Fields**: `value`, `type`, `confidence`, `severity` (in that order, no header row) - **Authentication**: Basic, username `api`, password your API key 4. **Install the Threat Prevention policy** ```bash curl -u api:$API_KEY "https://api.gen0sec.com/v1/threat/feeds?format=checkpoint" # 203.0.113.7,IP,high,high # 10.0.0.0-10.255.255.255,IP Range,low,medium ``` Set per-feed `confidence` and `severity` defaults in the access-rules list configuration, or override them per rule. Verify the values match what your Check Point version expects before deploying. ### General Integration Tips **Feed Format:** The Threat Feeds API returns a plain text list with one IP address or IP range per line: ``` 192.168.1.0/24 10.0.0.0/8 2001:db8::/32 ``` **Authentication:** All platforms support HTTP Basic Authentication: - **Username**: `api` (literal string) - **Password**: Your Gen0Sec API key **Update Frequency:** Recommended update intervals: - **High-security environments**: Every 15-30 minutes - **Standard environments**: Every 1-4 hours - **Low-priority feeds**: Every 12-24 hours **Testing the Feed:** Before configuring in your firewall, test the feed URL: ```bash # Test with Basic Auth curl -u "api:your-api-key" "https://api.gen0sec.com/v1/threat/feeds" | head -20 # Verify authentication works curl -I -u "api:your-api-key" "https://api.gen0sec.com/v1/threat/feeds" ``` **Monitoring:** - Monitor feed update status in your firewall's management interface - Set up alerts for feed update failures - Review firewall logs to verify blocks are being applied - Check feed size and update frequency to ensure timely threat protection ## Rate Limits API rate limits apply to prevent abuse. Contact support if you need higher limits. ## Caching Feeds are rebuilt at most once every 30 seconds server-side, and every response carries an `ETag` and `Last-Modified`. Prefer conditional requests over polling faster: sending `If-None-Match` costs one empty round trip when nothing has changed, whereas polling below the 30-second rebuild interval only re-downloads a body that cannot have changed yet. See [Conditional Requests](#conditional-requests). ## Support For API support, visit: - Discord: https://discord.com/invite/jzsW5Q6s9q - Email: support@gen0sec.com --- # Axiom Source: https://docs.gen0sec.com/docs/observability/axiom import { FAQ } from "@site/src/components/FAQ"; # Axiom Integrate Gen0Sec with Axiom for distributed tracing and observability using OpenTelemetry spans. ## Overview Axiom provides powerful observability capabilities for Gen0Sec deployments, enabling you to collect, analyze, and visualize OpenTelemetry spans for distributed tracing and performance monitoring. ![Axiom Dashboard - OpenTelemetry Traces](@site/static/img/axiom_dashboard.png) *OpenTelemetry Traces dashboard showing performance metrics, span rates, and slowest operations for the ax-proxy service* ## Key Features ### 🔍 Distributed Tracing - OpenTelemetry span collection and storage - Custom security event spans - Performance data tracing - Real-time threat intelligence spans ### 📈 Advanced Analytics - SQL-based query language for span analysis - Custom dashboards and visualizations - Historical data retention and analysis - Anomaly detection and alerting ### 📊 Trace Management - Centralized span collection and storage - Structured span attributes with custom fields - Trace correlation and analysis - Compliance and audit trail support ### ⚡ Real-time Monitoring - Live dashboard updates - Custom alerting rules - Performance monitoring - Security event tracking ## Setup and Configuration ### Prerequisites - **Axiom Account**: Active Axiom account with API access - **API Token**: Axiom API token with write permissions - **Dataset**: Configured dataset for OpenTelemetry spans ### Installation 1. **Get your Axiom credentials**: ```bash # Your Axiom API token AXIOM_TOKEN="your-axiom-api-token" # Your Axiom organization AXIOM_ORG="your-organization" # Your dataset name AXIOM_DATASET="gen0sec-spans" ``` 2. **Configure Gen0Sec integration**: ```jsonc { "observability": { "axiom": { "enabled": true, "api_token": "your-axiom-api-token", "org": "your-organization", "dataset": "gen0sec-spans", "span_interval": 60 } } } ``` 3. **Enable metrics collection**: ```bash # Set environment variable export AXIOM_ENABLED=true export AXIOM_TOKEN="your-axiom-api-token" export AXIOM_ORG="your-organization" export AXIOM_DATASET="gen0sec-spans" ``` ## OpenTelemetry Spans ### Security Spans | Span Name | Description | Attributes | |-----------|-------------|------------| | `security_check` | Security threat detection | `source_ip`, `threat_type`, `severity`, `action` | | `request_analysis` | Request analysis by Gen0Sec | `source_ip`, `status`, `response_time` | | `threat_block` | Request blocked due to threat | `source_ip`, `reason`, `threat_score` | | `api_request` | Gen0Sec API request | `endpoint`, `status`, `response_time` | ### Span Events ```json { "name": "security_check", "trace_id": "1-5f3b3b3b-3b3b3b3b3b3b3b3b", "span_id": "3b3b3b3b3b3b3b3b", "parent_id": "3b3b3b3b3b3b3b3a", "start_time": "2024-01-15T10:30:00Z", "end_time": "2024-01-15T10:30:00.045Z", "duration_ms": 45, "attributes": { "source_ip": "192.168.1.100", "threat_type": "malware", "severity": "high", "action": "blocked", "user_agent": "Mozilla/5.0...", "request_path": "/api/v1/data", "response_code": 403 } } ``` ### Custom Queries #### Threat Analysis ```sql SELECT attributes.threat_type, COUNT(*) as count, AVG(attributes.severity_score) as avg_severity FROM gen0sec-spans WHERE _time > now() - 1h AND name = 'security_check' GROUP BY attributes.threat_type ORDER BY count DESC ``` #### Performance Monitoring ```sql SELECT attributes.endpoint, AVG(duration_ms) as avg_response_time, P95(duration_ms) as p95_response_time, COUNT(*) as request_count FROM gen0sec-spans WHERE _time > now() - 1h AND name = 'api_request' GROUP BY attributes.endpoint ``` ## Dashboard Configuration ### Security Overview Dashboard Create a comprehensive security dashboard with the following panels: 1. **Threat Detection Overview** - Total threats detected (last 24h) - Threats by severity - Top threat sources 2. **Performance Metrics** - API response times - Request volume - Error rates 3. **Geographic Distribution** - Threats by country - Blocked IPs by region - Traffic patterns ### Alert Configuration Set up alerts for critical security events: ```yaml # High severity threat alert name: "High Severity Threat Detected" query: | SELECT COUNT(*) as threat_count FROM gen0sec-spans WHERE _time > now() - 5m AND name = 'security_check' AND attributes.severity = 'critical' condition: threat_count > 0 ``` ## Best Practices ### Data Retention - Configure appropriate retention policies for different data types - Archive historical data for compliance requirements - Optimize storage costs with data sampling ### Performance Optimization - Use efficient queries with proper time ranges - Implement data sampling for high-volume metrics - Monitor query performance and optimize as needed ### Security - Secure API token storage - Implement least-privilege access - Regular token rotation - Monitor access patterns ## Troubleshooting ### Common Issues #### Spans Not Appearing - Verify API token permissions - Check dataset configuration - Ensure OpenTelemetry tracing is enabled - Validate network connectivity #### High Latency - Optimize query time ranges - Use appropriate data sampling - Check network performance - Monitor Axiom service status #### Data Loss - Verify data retention policies - Check API rate limits - Monitor error logs - Validate span format ### Getting Help - **Axiom Documentation**: [docs.axiom.co](https://docs.axiom.co) - **Support**: Contact Axiom support for platform issues - **Community**: Join our [Discord community](https://discord.gg/jzsW5Q6s9q) ## FAQ --- For more information, visit [axiom.co](https://axiom.co) or join our [Discord community](https://discord.gg/jzsW5Q6s9q). --- # Honeycomb Source: https://docs.gen0sec.com/docs/observability/honeycomb import { FAQ } from "@site/src/components/FAQ"; # Honeycomb Integrate Gen0Sec with Honeycomb for distributed tracing and observability using OpenTelemetry spans. ## Overview Honeycomb provides powerful distributed tracing and observability capabilities for Gen0Sec deployments, enabling you to understand request flows, identify performance bottlenecks, and correlate security events across your entire system using OpenTelemetry spans. ## Key Features ### 🔍 Distributed Tracing - End-to-end request tracing across services - Automatic instrumentation of security checks - Custom span creation for threat analysis - Trace correlation with security events ### 📊 Performance Monitoring - Real-time performance spans - Custom instrumentation points - Response time analysis - Error rate tracking ### 🎯 Custom Spans - Security event tracking - Threat intelligence data - Custom business spans - Anomaly detection ### 📈 Advanced Analytics - Query-based analysis with Honeycomb Query Language (HQL) - Custom dashboards and visualizations - SLO/SLI monitoring - Trend analysis and alerting ## Setup and Configuration ### Prerequisites - **Honeycomb Account**: Active Honeycomb account with API access - **API Key**: Honeycomb API key with write permissions - **Team**: Configured team for OpenTelemetry spans ### Installation 1. **Get your Honeycomb credentials**: ```bash # Your Honeycomb API key HONEYCOMB_API_KEY="your-honeycomb-api-key" # Your Honeycomb team HONEYCOMB_TEAM="your-team" ``` 2. **Configure Gen0Sec integration**: ```jsonc { "observability": { "honeycomb": { "enabled": true, "api_key": "your-honeycomb-api-key", "team": "your-team", "sample_rate": 1.0 } } } ``` 3. **Enable tracing**: ```bash # Set environment variables export HONEYCOMB_ENABLED=true export HONEYCOMB_API_KEY="your-honeycomb-api-key" export HONEYCOMB_TEAM="your-team" ``` ## OpenTelemetry Spans ### Security Event Spans ```json { "name": "security_check", "service_name": "gen0sec-proxy", "trace_id": "1-5f3b3b3b-3b3b3b3b3b3b3b3b", "span_id": "3b3b3b3b3b3b3b3b", "parent_id": "3b3b3b3b3b3b3b3a", "timestamp": "2024-01-15T10:30:00Z", "duration_ms": 45, "attributes": { "source_ip": "192.168.1.100", "threat_detected": true, "threat_type": "malware", "severity": "high", "action": "blocked", "user_agent": "Mozilla/5.0...", "request_path": "/api/v1/data", "response_code": 403, "geo_country": "US", "geo_city": "New York" } } ``` ### Custom Spans Create custom spans for detailed security analysis: ```javascript // Threat intelligence check span const threatSpan = honeycomb.startSpan({ name: "threat_intelligence_check", service_name: "gen0sec-proxy", attributes: { "ip_address": sourceIP, "check_type": "reputation", "data_source": "gen0sec_api" } }); // Perform threat check const threatResult = await checkThreatIntelligence(sourceIP); // Add results to span threatSpan.addField("threat_score", threatResult.score); threatSpan.addField("threat_categories", threatResult.categories); threatSpan.addField("confidence_level", threatResult.confidence); threatSpan.end(); ``` ### Query Examples #### Security Event Analysis ```sql SELECT COUNT(*) as event_count, AVG(duration_ms) as avg_duration, P95(duration_ms) as p95_duration FROM honeycomb WHERE timestamp > now() - 1h AND name = "security_check" AND attributes.threat_detected = true GROUP BY attributes.threat_type, attributes.severity ``` #### Performance Monitoring ```sql SELECT name, COUNT(*) as request_count, AVG(duration_ms) as avg_response_time, P99(duration_ms) as p99_response_time, COUNT_WHERE(attributes.response_code >= 400) as error_count FROM honeycomb WHERE timestamp > now() - 1h AND service_name = "gen0sec-proxy" GROUP BY name ORDER BY avg_response_time DESC ``` #### Geographic Threat Analysis ```sql SELECT attributes.geo_country, attributes.geo_city, COUNT(*) as threat_count, COUNT_WHERE(attributes.severity = "critical") as critical_threats FROM honeycomb WHERE timestamp > now() - 24h AND attributes.threat_detected = true GROUP BY attributes.geo_country, attributes.geo_city ORDER BY threat_count DESC LIMIT 10 ``` ## Dashboard Configuration ### Security Overview Dashboard Create comprehensive security dashboards with these panels: 1. **Threat Detection Overview** - Total threats detected (last 24h) - Threats by severity and type - Geographic distribution of threats 2. **Performance Metrics** - Response time percentiles - Error rates by endpoint - Throughput analysis 3. **Trace Analysis** - Slowest security checks - Dependency performance - Error correlation ### SLO/SLI Configuration Set up Service Level Objectives for security operations: ```yaml # Security Check SLO name: "Security Check Response Time" target: 95% of requests under 100ms query: | SELECT COUNT_WHERE(duration_ms < 100) / COUNT(*) * 100 as slo_percentage FROM honeycomb WHERE timestamp > now() - 1h AND name = "security_check" ``` ## Best Practices ### Sampling Strategy - Use adaptive sampling for high-volume endpoints - Sample 100% of security events - Implement custom sampling rules for different event types ### Instrumentation - Add custom attributes for business context - Use consistent naming conventions - Include relevant metadata in spans ### Performance Optimization - Monitor trace volume and adjust sampling - Use efficient queries with proper time ranges - Implement trace batching for high-throughput scenarios ### Security - Secure API key storage - Implement least-privilege access - Regular key rotation - Monitor access patterns ## Troubleshooting ### Common Issues #### Spans Not Appearing - Verify API key permissions - Check team configuration - Ensure OpenTelemetry tracing is enabled - Validate network connectivity #### High Latency - Check sampling configuration - Monitor network performance - Verify Honeycomb service status - Optimize query time ranges #### Data Loss - Verify sampling rates - Check API rate limits - Monitor error logs - Validate span format ### Getting Help - **Honeycomb Documentation**: [docs.honeycomb.io](https://docs.honeycomb.io) - **Support**: Contact Honeycomb support for platform issues - **Community**: Join our [Discord community](https://discord.gg/jzsW5Q6s9q) ## FAQ --- For more information, visit [honeycomb.io](https://honeycomb.io) or join our [Discord community](https://discord.gg/jzsW5Q6s9q). --- # OpenTelemetry Source: https://docs.gen0sec.com/docs/observability/opentelemetry # OpenTelemetry Synapse speaks **OTLP/HTTP** for three signals: security event **logs**, **metrics**, and request **spans**. Point it at any OpenTelemetry collector and they arrive on the standard paths — there is no Gen0Sec-specific protocol in the way. Logs and metrics are on by default; spans are not, and [Tracing](#tracing) turns them on. The backend-specific pages ([Axiom](./axiom.mdx), [Honeycomb](./honeycomb.mdx)) are worked examples of what this page describes generically. ## Turning it on ```yaml # /etc/synapse/config.yaml telemetry: enabled: true otlp: endpoint: "https://collector.example.com:4318" headers: authorization: "Bearer " # Sampling covers HTTP access logs, and only those. Errors bypass the sampler. access_log: sample_rate: 0.1 always_keep_errors: true ``` `endpoint` is a **base URL**. Synapse appends the OTel-standard paths itself: | Signal | Path | |---|---| | Logs | `{endpoint}/v1/logs` | | Metrics | `{endpoint}/v1/metrics` | | Spans | `{endpoint}/v1/traces` — only with [`telemetry.tracing.export_spans`](#tracing) | Every request carries `User-Agent: Synapse/`, so one allowlist entry covers config-poll and telemetry push alike, and the two are distinguishable in a collector's own access log. ## Where telemetry goes Up to three OTLP destinations, and they are **not** a precedence ladder. Synapse pushes to every one that is configured: 1. **The `OTEL_EXPORTER_OTLP_*` environment variables** — the standard OTel knobs. An endpoint set here is a destination of its own, not an override of the file. 2. **The `telemetry.otlp` block** above. 3. **The platform** — `{base_url}/v1/telemetry`, where logs and metrics share one path. This push is gated by `platform.telemetry_sending_enabled`, which defaults to **true**, not by whether the other two set an endpoint. So a stray `OTEL_EXPORTER_OTLP_ENDPOINT` does not quietly replace your config file: it adds a second place the data goes. For a reader who cares where telemetry lands, item 3 is the one to check — an agent with `platform.api_key` set is already pushing to the platform whether or not you configure a collector. Headers follow their own destination. `OTEL_EXPORTER_OTLP_HEADERS` reach the environment's destination only, and `telemetry.otlp.headers` reach that block's only. A URL that two of them name is pushed to once, as the earlier, carrying the headers of both — the earlier's winning a clash. ### The local file sink `{logging.log_directory}/telemetry.log` — `/var/log/synapse/telemetry.log` with the default directory. Three things about it are easy to get wrong: - **It is off by default.** The sink exists only when `logging.file_logging_enabled` is `true`, and that key defaults to **false**. On a default config there is no `telemetry.log` at all. - **It is not a fallback.** When it is on, block events are written to it *and* pushed to every OTLP destination, rather than the file catching what is left when none is configured. - **It carries block events, not the whole log signal.** This is the part worth knowing before you treat `telemetry.log` as a local mirror: access logs, IDS alerts and the fingerprint events take a different emit path that writes to OTLP only. With the sink on and no collector configured, `telemetry.log` fills with block events while those others go nowhere on the telemetry path at all. Their local counterparts are `events.log`, `eventbridge.log` and the access log, not this file. - **`telemetry.enabled: false` takes it with them.** The switch skips telemetry initialisation entirely — no OTLP push, no file sink, no instruments. Block events still reach the blocking log by their own path, so the host is not silent: that is `events.log` when `logging.file_logging_enabled` is on, and stdout when it is not. :::note[Omitting the block does not disable telemetry] `telemetry.enabled` defaults to **true**, so a config with no `telemetry:` section at all still exports — to the platform if `platform.api_key` is set. If you want nothing leaving the host, set `enabled: false` explicitly; leaving the section out does the opposite. ::: ## Metrics | Metric | Type | Unit | | |---|---|---|---| | `synapse.kernel.drops` | Counter | `{packet}` | Packets dropped by the in-kernel BPF firewall, partitioned by layer | | `synapse.block.events.access_rules` | Counter | `{event}` | Static IP / CIDR / country / ASN blocklists | | `synapse.block.events.smart_firewall` | Counter | `{event}` | JA4-family fingerprint matches, kernel and userland | | `synapse.block.events.ids` | Counter | `{event}` | Thalamus Suricata-style signatures | | `synapse.block.events.waf` | Counter | `{event}` | HTTP-layer wirefilter rules | | `synapse.block.events.threat_intel` | Counter | `{event}` | IP-reputation hits | | `synapse.block.events.captcha` | Counter | `{event}` | Challenge issued or failed | | `synapse.block.events.rate_limit` | Counter | `{event}` | Per-route or per-fingerprint rate hits | | `synapse.ban.active` | Gauge | `{ban}` | Bans currently held, observed on the collection interval | | `synapse.ban.records_dropped` | Counter | `{record}` | Ban-ledger records dropped | | `synapse.telemetry.emit_latency` | Histogram | `ms` | Cost of the emit path itself: file write, OTLP enqueue and metrics record | ### Attributes Two attribute keys carry the dimensions, and they are attached to different metrics: | Attribute | On | Values | |---|---|---| | `synapse.action` | every `block.events.*` counter | `block`, `drop`, `allow`, `log`, `notice`, `ratelimit`, `captcha` | | `synapse.layer` | `synapse.kernel.drops`, `synapse.ban.active` | the enforcing layer | So the layer is encoded two different ways depending on the metric, and this is the thing that trips up a first query: **`block.events.*` is one counter per layer**, with the *action* as its attribute. A total across all enforcement is therefore a sum over the metric family, not a single series filtered by a layer label: ```promql sum(rate({__name__=~"synapse_block_events_.*"}[5m])) ``` Whereas `synapse.kernel.drops` and `synapse.ban.active` are single series that you *do* split by `synapse.layer`: ```promql sum by (synapse_layer) (rate(synapse_kernel_drops[5m])) ``` `synapse.telemetry.emit_latency` carries explicit bucket boundaries from 0.01 ms to 100 ms. It measures Synapse's own observability overhead, which is the number to look at before blaming telemetry for a latency regression. Metrics export on a periodic reader. Override the interval with the standard `OTEL_METRIC_EXPORT_INTERVAL` environment variable; it applies to every configured destination. ### Identifying an agent Every record carries `service.instance.id`, set to Synapse's derived agent ID — a stable hash of the agent name that survives restarts, which is what you want for a long-running dashboard. :::caution[Without an agent ID, the instance ID is the PID] The fallback is the process PID, which changes on every restart. That is useful for spotting restarts and useless for a dashboard grouping by instance — a restart looks like one agent disappearing and a different one appearing. ::: ## Sampling One sampler, and it covers HTTP access logs only — one record per proxied request is the highest-volume stream a proxy produces. | Key | Default | What it does | | --- | --- | --- | | `telemetry.access_log.sample_rate` | `1.0` | Probability in `[0.0, 1.0]` of keeping a non-error access-log record. `1.0` keeps every one | | `telemetry.access_log.always_keep_errors` | `true` | 4xx and 5xx responses bypass the sampler | Leave `always_keep_errors` on; sampling away errors is how a sampled stream becomes misleading rather than merely incomplete. `SYNAPSE_ACCESS_LOG_SAMPLE_RATE` overrides the sample rate when it is set. There is **no sampler on the general log stream** — every other event Synapse exports is exported in full. Metrics are not sampled either: they are aggregated at the source, so the export interval, not a sample rate, controls their volume. :::caution[The access-log sampler also suppresses spans] A request's spans are exported only if the access-log sampler kept that request's record, or another hop's spans hang off it (a trusted caller sampled the trace, or Synapse sent its own span upstream as their parent). On a default config neither of those holds — it takes a `trusted_proxies` entry or `trust_lb_tls_headers`, and `propagation` other than `passthrough` — so `access_log.sample_rate: 0.1` really does cost about nine spans in ten, whatever `tracing.sample_rate` says. Sample the access log and you are sampling traces with it. Two corollaries, both of which look like a tracing bug and are not: - **Errors keep their spans**, because `always_keep_errors` keeps their access-log record. The nine-in-ten figure is for non-error traffic. - **An endpoint with access logging switched off exports no spans**, at any sample rate. That includes health-check probes. ::: :::caution[`telemetry.otlp` takes no sampling keys] `sample_rate` and `always_keep_errors` live under `telemetry.access_log`, and nowhere else. Set either one under `telemetry.otlp` and it is silently ignored: the key lands in the `Unused config options in ` warning at startup, and nothing is sampled. An earlier version of this page showed them in both places. ::: ## Tracing Every log record a request produces carries its trace context whatever these settings say. They decide only what reaches the upstream, and whether spans are exported. | Key | Default | What it does | | --- | --- | --- | | `telemetry.tracing.export_spans` | `false` | Export a server span per proxied request, with a child per timed stage, at `/v1/traces` | | `telemetry.tracing.propagation` | `passthrough` | What the proxy does with the `traceparent` / `tracestate` headers it forwards upstream | | `telemetry.tracing.sample_rate` | `1.0` | Ratio of the traces Synapse starts that are sampled (`parentbased_traceidratio`, so a caller's own decision is followed) | `propagation` takes three values: - **`passthrough`** (the default) — forward the headers as the client sent them, so Synapse's hop is not part of the trace the upstream continues. - **`propagate`** — send Synapse's own context: the client's trace, or a new one, with Synapse's span as the upstream's parent. The client's `tracestate` is kept only when its trace is continued. - **`off`** — strip both, so no trace context reaches the upstream. `OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` override `sample_rate`. All three keys are read once, at startup. :::caution[Spans do not go to the platform] Of the three destinations above, spans reach the first two only. The platform destination takes logs and metrics and refuses traces, logging `no spans to : it takes logs and metrics only`. So an agent configured with nothing but `platform.api_key` exports **no spans at all**, however `export_spans` is set — give it an `OTEL_EXPORTER_OTLP_ENDPOINT` or a `telemetry.otlp.endpoint` to send them to. ::: ## Limits worth knowing - **The queue is bounded.** If a collector is unreachable, records buffer and then drop at enqueue once the queue fills, with a warning logged. Telemetry backpressure never becomes request backpressure, which is the right trade — but it does mean a long collector outage is a gap, not a backlog. - **Batching adds delay.** Records flush on a size threshold or a time cap, whichever comes first, so there is a small lag between a decision and its appearance downstream. - **Spans are off by default, not absent.** See [Tracing](#tracing) for the three keys that turn them on and decide what reaches the upstream. ## See also - [Security Event Export](../features/event-export.mdx) — the event model behind the log stream, and the non-OTLP sinks - [Axiom](./axiom.mdx) and [Honeycomb](./honeycomb.mdx) — backend-specific walkthroughs - [Configuration](../client/configuration.mdx) — the full `telemetry` block