Skip to main content

Synapse Operator

The operator is a controller-runtime controller that keeps Synapse in step with the cluster around it. It does the jobs that are awkward to do by hand: rolling pods when their config content changes, turning Ingress objects into Synapse routes, and filling in configuration that can only be known at runtime.

It is a plain Deployment with no CRDs of its own. It works off native Kubernetes objects — ConfigMaps, Secrets, Ingresses, Pods, NetworkPolicies — and off the labels the Helm charts already apply.

Deploying it is covered in Helm Charts. This page is what it does and how to configure it.

One primary mode, plus add-ons

The operator picks exactly one primary mode per process, and then layers any number of optional reconcilers on top.

Config-syncIngress / Gateway API
Flag(default)--ingress-mode
WatchesConfigMaps and Secrets, by labelIngress, plus HTTPRoute with --gateway-api
DoesStamps a config hash on the workload, so Kubernetes rolls itRenders upstreams.yaml, then hot-reloads Synapse
TLS Secret projectionYes, via --certs-out
Publishes LB status on IngressYes, via --publish-status-address

The two are mutually exclusive within a process — ingress mode replaces config-sync rather than adding to it. To run both, run two Deployments. The synapse-stack chart does exactly that when you set operator.ingress.enabled.

The four reconcilers below are different: each is an independent switch that composes with whichever primary mode is running.

Config-sync

The default. Kubernetes does not restart a pod when a mounted ConfigMap's content changes — the file updates in place and the process never notices. The operator closes that gap: it hashes every ConfigMap and Secret matching its label selector, and stamps the hash onto the workload's pod template. A changed hash is a changed pod spec, so Kubernetes rolls it.

synapse.gen0sec.com/config-hash: <sha of the watched config>

Two flags make this less blunt than it sounds:

  • --per-workload-config-hash stamps each workload with a hash of only the config it references, resolved through volumes, envFrom and env. Without it, one combined hash covers every labelled source in the namespace, so changing an agent rule file rolls the proxy too. A workload referencing no labelled source is left alone.
  • --ids-hot-reload-hash-exclude leaves the hot-reloadable IDS fields out of the hash entirely — ids.address_vars, enforce_block, rule_paths, port_vars, flow_timeout_secs, max_flows. The agent reloads those in process, so a change to only those does not need a rollout. Other config.yaml changes still roll.
--ids-hot-reload-hash-exclude assumes every agent can hot-reload

It is only safe once all agents run a Synapse image that reloads those fields in place. On an older image the field changes in the ConfigMap, no rollout happens, and the agent keeps running the old value indefinitely — silently.

upstreams.yaml is excluded from the hash by default (--ignore-configmap-keys). That is deliberate: in ingress mode the operator itself rewrites that key, and hashing it would make the operator roll the dataplane in response to its own writes.

Ingress and Gateway API mode

With --ingress-mode, the operator reconciles class-matched Ingress objects — and, with --gateway-api, Gateway API HTTPRoute objects — into Synapse's upstreams.yaml.

Two output layouts:

FlagHow Synapse picks it up
Sidecar--upstreams-out=/shared/upstreams.yamlShared volume; inotify, then SIGHUP to the co-located process
Central--upstreams-out-configmap=<ns>/<name>Synapse reloads from its own ConfigMap mount; no SIGHUP

In the central layout only the upstreams.yaml key is touched — other keys in that ConfigMap are preserved.

How paths are translated

SourceBecomes
Ingress Prefix, Gateway PathPrefixA Synapse longest-prefix path
Ingress Exact, Gateway ExactApproximated as a prefix, plus a warning event — Synapse matches longest-prefix
Ingress ImplementationSpecific with nginx.ingress.kubernetes.io/use-regex: "true"A regex route: match_expr: http.request.path matches "<regex>"
Gateway RegularExpressionA regex route, same form
/.well-known/acme-challenge/*internal_paths, for the cert-manager HTTP-01 solver

Regex routes are anchored at the start — the leading ^ is supplied by the renderer, since Kubernetes requires the stored path to begin with /.

Header, method and query matches are dropped

Synapse's routing model is host plus path. An Ingress or HTTPRoute rule that also matches on headers, method or query parameters cannot be represented, so those conditions are discarded and a warning event is emitted on the object. The route still exists — it is simply broader than what you wrote.

Route conflicts resolve first-writer-wins, and both the conflicts and the dropped matches are counted in metrics, so neither is invisible.

Certificates and status

  • --certs-out projects referenced Ingress and Gateway TLS Secrets into Synapse's certificates directory as <stem>.crt / <stem>.key. The directory is operator-owned and hot-reloaded, so certificates arrive without a restart. --certs-out-secret does the same into a Secret instead, for a proxy in a separate pod, and takes precedence.

  • --publish-status-address writes the addresses you give it onto matched Ingresses' .status.loadBalancer.ingress, which is what makes kubectl get ingress show an address.

  • --status-leader-election gates those shared status writes behind a Lease, so with more than one replica only the holder writes. Per-pod render and SIGHUP are never gated.

  • --render-once renders upstreams.yaml and exits — for an initContainer that primes the file before Synapse starts, so the proxy never boots with no routes.

The composable reconcilers

Each of these is an independent flag that runs alongside whichever primary mode is active. All four are off by default.

UpstreamsResolver — --upstreams-resolver

Watches ConfigMaps labelled synapse.gen0sec.com/resolve-upstreams=true, substitutes each backend Service's DNS name with its ClusterIP, and writes the result to a sibling ConfigMap.

The point is the request path: with IP literals, Synapse hands upstream_peer an address and never resolves a hostname per request. Point the dataplane at the output with synapse.upstreamsConfigMap.

--resolve-backend-cluster-ips does the same job inside ingress mode's own render, falling back to the FQDN for headless, ExternalName and not-yet-allocated Services.

NetVarsResolver — --netvars-resolver

Watches agent config ConfigMaps labelled synapse.gen0sec.com/resolve-netvars=true and fills ids.address_vars.HOME_NET and EXTERNAL_NET from what the cluster actually is: Node IPs, Pod CIDRs, LoadBalancer VIPs and the RFC1918 supernets.

This matters more than it looks. With inline IDS blocking (enforce_block) and a HOME_NET that does not describe the cluster, the IDS can ban an internal address — a node, a pod, or your own load balancer VIP. Deriving it from the cluster is what stops that.

A HOME_NET you set by hand, or the synapse.gen0sec.com/home-net annotation, overrides the auto-discovery. The synapse-stack chart already applies the label to the agent's ConfigMap, so enabling the reconciler is all that is needed.

IdentityProducer — --identity-producer

Builds a workload-identity MMDB — pod IP to workload, namespace and app — from cluster Pods, and uploads it so agents can pull it for east-west detection. That is what turns a flow between two pod IPs into a flow between two named workloads.

--identity-producer-interval (default 5 minutes) is the baseline re-upload for cold start and resync; per-change freshness comes from event-driven deltas, so it does not need to be frequent. --cluster-id scopes the upload so multiple clusters sharing one workspace do not overwrite each other — without it they share a global path, last writer wins. Set it here and set the matching platform.identity.cluster on every agent in the cluster.

EdgeProducer — --edge-producer

Compiles the cluster's NetworkPolicy objects into a declared-edge allow-list — workload to workload, by port — and uploads it for the agent's edge.* microsegmentation fields. It turns policy you have already written into something the agent can compare observed traffic against.

Both producers need an API key

Each requires an API key with identity:write, supplied as SYNAPSE_API_KEY — the charts wire this from operator.apiKeySecret.

Their RBAC needs synapse-stack 0.6.0 or newer

The identity producer lists Pods; the edge producer also lists Namespaces and NetworkPolicies. Earlier stack charts granted none of these, so both producers failed on a default install. From 0.6.0 the rules are created automatically when you enable the corresponding producer.

The standalone synapse-operator chart does not expose the producers as values, so running them there through operator.extraArgs still needs a ClusterRole of your own.

operator.graphProducer is not implemented

The synapse-stack chart exposes a graphProducer block that passes --graph-producer, --graph-producer-interval and --service-graph-url. The operator defines none of those flags. Go's flag parser rejects unknown flags and exits, so enabling it puts the operator into CrashLoopBackOff with flag provided but not defined. Leave it off.

Flag reference

Common

FlagDefault
--metrics-bind-address:8080Metrics endpoint
--health-probe-bind-address:8081Health probes
--leader-electfalseLeader election for the manager
--namespace(all)Restrict the watch to one namespace

Config-sync

FlagDefault
--label-selectorapp.kubernetes.io/name=synapseSelects config sources and workloads
--config-hash-annotationsynapse.gen0sec.com/config-hashAnnotation key for the hash
--ignore-configmap-keysupstreams.yamlConfigMap keys excluded from the hash
--ignore-secret-keys(none)Secret keys excluded from the hash
--per-workload-config-hashfalseHash only what each workload references
--ids-hot-reload-hash-excludefalseLeave hot-reloadable IDS fields out of the hash

Ingress mode

FlagDefault
--ingress-modefalseRun as an Ingress / Gateway API controller
--ingress-classsynapseThe ingressClassName this controller serves
--upstreams-out/shared/upstreams.yamlRendered file path, sidecar layout
--upstreams-out-configmap(none)<ns>/<name> to render into instead, central layout
--render-oncefalseRender once and exit, for an initContainer
--gateway-apifalseAlso reconcile Gateway API; needs the CRDs
--resolve-backend-cluster-ipsfalseEmit <clusterIP>:port instead of the Service FQDN
--cluster-domaincluster.localCluster DNS domain for backend FQDNs
--certs-out(off)Project TLS Secrets into this directory
--certs-out-secret(off)Project them into this Secret instead; takes precedence
--publish-status-address(none)Addresses to publish on matched Ingresses
--reload-process-namesynapseargv0 of the co-located proxy to SIGHUP
--reload-debounce500msCoalesce SIGHUP bursts; 0 signals every render
--status-leader-electionfalseOnly the Lease holder writes shared status
--status-leader-election-idsynapse-ingress-statusLease name
--leader-election-namespace$POD_NAMESPACENamespace for that Lease

Reconcilers

FlagDefault
--upstreams-resolverfalseSubstitute Service DNS for ClusterIPs
--netvars-resolverfalseFill IDS HOME_NET / EXTERNAL_NET from the cluster
--identity-producerfalseUpload the pod-IP-to-workload MMDB
--identity-producer-interval5mBaseline re-upload interval
--edge-producerfalseUpload NetworkPolicy as a declared-edge allow-list
--edge-producer-interval5mBaseline re-upload interval
--download-api-urlhttps://api.gen0sec.com/v1Where the producers upload
--cluster-id(none)Scopes this cluster's baseline; empty shares a global path

The charts expose the common options as values, including --gateway-api as operator.gatewayAPI (standalone) or operator.ingress.gatewayAPI (stack). Anything without one — --certs-out, --publish-status-address, --render-once and the rest — goes through operator.extraArgs.

Permissions

The operator needs cluster-scoped read on the objects it watches, and write on what it stamps:

API groupResourcesVerbs
coreconfigmapsget, list, watch, create, update, patch
coresecretsget, list, watch, create, update, patch
corenodes, servicesget, list, watch
appsdeployments, daemonsets, statefulsetsget, list, watch, patch, update
networking.k8s.ioingresses, ingressclassesget, list, watch
coordination.k8s.ioleasesfull — leader election
coreeventscreate, patch, update

The charts extend this as you turn features on: ingresses, ingressclasses and ingresses/status with ingress mode, the gateway.networking.k8s.io resources with gatewayAPI, and pods, namespaces and networkpolicies with the producers. Every rule is gated on the same value that enables the feature, so the two cannot drift apart.

Metrics

Exposed on --metrics-bind-address, and picked up by a ServiceMonitor if you enable one in the synapse-operator chart.

Metric
synapse_operator_ready1 once the first successful render completes
synapse_operator_render_totalRender passes attempted
synapse_operator_render_errors_totalRenders that failed on a list or write
synapse_operator_render_changed_totalRenders that actually changed the output
synapse_operator_last_render_timestamp_secondsWhen the last good render happened
synapse_operator_reload_signals_totalSIGHUPs delivered to Synapse
synapse_operator_hosts / synapse_operator_routesSize of the current render
synapse_operator_route_conflicts_totalHost+path conflicts ignored
synapse_operator_unsupported_match_totalMatch features that could not be represented
synapse_operator_backend_unresolved_totalBackends that could not be resolved
synapse_operator_certs / synapse_operator_cert_errors_totalTLS Secrets projected, and those missing or unusable

synapse_operator_unsupported_match_total and synapse_operator_route_conflicts_total are the two worth alerting on — both mean the cluster is asking for routing that Synapse is silently rendering differently.

Troubleshooting

Config changes never roll the workload. Check the selector first — operator.labelSelector has to match both the proxy and the agent. Then confirm the annotation is moving:

kubectl -n synapse-os get pod -l app.kubernetes.io/name=synapse-proxy \
-o jsonpath='{.items[0].metadata.annotations.synapse\.gen0sec\.com/config-hash}'

Ingresses render nothing. Confirm spec.ingressClassName matches --ingress-class, then check for RBAC denials in the operator log — the stack chart does not grant Ingress access.

A route is broader than the Ingress says. Header, method and query matches are dropped; look for the warning event on the object and check synapse_operator_unsupported_match_total.

The operator crash-loops on startup. Read the first log line. Flag provided but not defined means an argument the binary does not know — operator.graphProducer is the usual cause, and operator.extraArgs the other.

See also