Symptom index
Everything here was reproduced on a real cluster.
Images
ImagePullBackOff
Three different causes, and the first thing to establish is which.
kubectl -n <namespace> get pod <pod> -o jsonpath='{.spec.containers[*].image}'
kubectl -n <namespace> describe pod <pod> | tail -20
| The image reference is | Cause | Fix |
|---|---|---|
| Correct, and the error mentions authorization | The gen0sec-registry secret is missing from that namespace | It is needed in both namespaces. See Install step 2 |
| Pointing at the wrong registry | A values file was not passed with -f | Re-run with -f $B/values/<chart>-values-onprem.yaml |
| A path that does not exist in your registry | Mirror path mismatch, on the offline bundle method | See the next entry |
403 with a CAPTCHA challenge from our registry
All images, not one, and only on the registry install method.
The tell: a manual docker pull or skopeo copy from the same node succeeds while the kubelet's
pull returns 403.
# succeeds: the CLI sends its own user agent
skopeo inspect docker://registry.gen0sec.com/gen0sec/auth-api:0.1.0
# 403: what the kubelet gets
kubectl -n gen0sec describe pod <pod> | grep -A3 Failed
On some distributions the component servicing the kubelet's pull sets no User-Agent, so Go sends
Go-http-client/1.1, which our edge challenges as a scanner signature. A kubelet cannot solve a
CAPTCHA, so it never recovers. Not credentials, not TLS, not the pull secret.
Fix: set a user agent in the container runtime on every node, or use the offline bundle method. Both in Known limitations.
ImagePullBackOff on a pod you did not configure
Operator-launched pods (Postgres, the connection pooler, Kafka brokers, the entity operator) get their image reference from the operator, not from the chart. If one of those is stuck, the mirror path is wrong, not the pull secret.
Confirm the path exists in your registry:
curl -su <user>:<password> https://$REGISTRY/v2/_catalog?n=200 | tr ',' '\n'
Two paths look surprising and are not negotiable:
| Image | Must be mirrored to |
|---|---|
| Kafka broker | <registry>/gen0sec/strimzi/kafka:1.0.0-kafka-4.1.1 |
| Entity operator | <registry>/gen0sec/strimzi/operator:1.0.0 |
Everything is under gen0sec/. Kafka additionally keeps the upstream strimzi/ path inside it,
because the operator's defaultImageRegistry replaces only the registry host and keeps the
repository path. A Kafka pod asking for <registry>/strimzi/kafka with no gen0sec/ means that
setting lost the namespace.
401 Unauthorized pulling into a Postgres or pooler pod
The Postgres operator does not copy imagePullSecrets onto the database pods it creates. They inherit
them from their ServiceAccount, which values-onprem.yaml configures through
configKubernetes.pod_service_account_definition.
If you overrode configKubernetes wholesale, you dropped it. Merge instead of replacing.
manifest unknown
A chart is asking for a tag that was never published. Check what it rendered:
helm get manifest <release> -n <namespace> | grep -E '^\s+image:' | sort -u
Then compare against Component versions.
Postgres
Postgres will not start
postgresql/core sits in CreateFailed.
kubectl -n gen0sec-system logs deploy/g0s-infra-postgres-operator | grep -i 'could not create'
| Log line contains | Cause | Fix |
|---|---|---|
endpoints is forbidden | The operator's own ClusterRole omits endpoints, but it creates a master Endpoints object per cluster | The infra chart adds the missing rule. Confirm: kubectl get clusterrole g0s-infra-postgres-operator-endpoints |
Unsupported value: "18" | The CRDs in the cluster are older than the operator chart | Helm never upgrades CRDs. Apply them first. See Upgrade |
| Nothing, and no volume is bound | No default StorageClass, or not enough capacity | See Sizing |
After fixing the cause the operator does not retry promptly. Force it:
kubectl -n gen0sec-system rollout restart deploy/g0s-infra-postgres-operator
Migration failures
The migration job runs as a Helm hook. On failure Helm aborts and deploys nothing, so the cluster is in a safe state.
kubectl -n gen0sec logs job/g0s-db-migrate
| Log line contains | Cause | Fix |
|---|---|---|
relation "..." does not exist | The chain is running against a database that is not at the version it expects. Usually a forced version, or a db-migrate image older than the chart | Read the version table in the next entry before you force anything, then collect diagnostics |
role "..." does not exist | A database role is missing. The migrations GRANT to roles by name and roll back if one is absent | The data chart creates them. Confirm: kubectl -n gen0sec-system get secret | grep credentials |
already exists | A migration was forced to the wrong version and re-ran DDL | See the next entry |
The schema is marked dirty
The migration tool marks the schema dirty when a migration fails halfway, and refuses to continue until you say what happened.
kubectl -n gen0sec-system exec -it core-0 -c postgres -- \
psql -U postgres -d ax_prod -c 'select * from schema_migrations'
force <version> means "this version is applied, continue with the next one". So to re-run a
failed migration, force the version before it. Migrations are not idempotent.
Kafka
Kafka will not start
Broker crash loops with Invalid cluster.id ... Expected X, but read Y.
The volume survived a previous helm uninstall, which is deliberate, and the recreated cluster
generated a new identity. Either restore the original cluster or wipe the volumes:
kubectl -n gen0sec-system delete kafkanodepool kafka
kubectl -n gen0sec-system delete pvc -l strimzi.io/cluster=core
helm upgrade g0s-data ...
This deletes the message data on those volumes.
No KafkaTopic resources exist although the chart rendered them
Helm can skip re-applying objects that were deleted out of band. Re-apply the stored manifest:
helm get manifest g0s-data -n gen0sec-system | kubectl apply -f -
Object store and cache
Services cannot reach the object store
The object store Service is rustfs-svc, with the suffix. S3_ENDPOINT and ARXIGNIS_DATA_URL must
match it.
kubectl -n gen0sec-system get svc | grep rustfs
kubectl -n gen0sec get deploy g0s-download-api \
-o jsonpath='{.spec.template.spec.containers[0].env}' | tr ',' '\n' | grep -E 'S3_ENDPOINT|DATA_URL'
See Install step 7.
500 NoSuchBucket
The bucket does not exist. Install step 5 creates it. Nothing in the charts does.
Everything works but reads are slow
The shared cache is never fatal. Missing or unreachable, each service logs a warning once and runs on its in-process cache only. Confirm from the service side, not the pod list:
kubectl -n gen0sec logs deploy/g0s-auth-api | grep -i "L2 cache"
L2 cache (Redis) connected to server means the cache is in use. falling back to L1-only cache
means REDIS_URL is wrong or unreachable.
kubectl -n gen0sec-system get svc dragonfly
The Service is deliberately not prefixed with the release name, because REDIS_URL names it
directly.
The cache pod crash loops
kubectl -n gen0sec-system logs deploy/dragonfly --previous
| Log shows | Cause | Fix |
|---|---|---|
io_uring errors | Blocked by the host kernel or a seccomp profile | Add --force_epoll to dragonfly.extraArgs in the infra values |
| Read-only filesystem errors | The pod runs with a read-only root, which is why the chart disables snapshotting | Remove any flag you added that re-enables it |
The cache is being evicted or OOM-killed
maxMemory is the eviction threshold and the memory limit is the kill threshold. Keep the first
comfortably below the second, or the kubelet kills the pod before it gets a chance to evict.
Install and workstation
The bundler fails with a yq syntax error
You have the wrong yq. Two unrelated tools share the name.
yq --version
The output must contain mikefarah. If not, install the Go version and point the bundler at it with
MIKEFARAH_YQ_PATH. See Requirements.
zsh: no matches found, before helm runs
zsh expands [0] in a --set argument as a glob. Single-quote the whole argument:
--set 'services.ui.ingress.hosts[0].host=example.internal'
The namespace hangs in Terminating
Strimzi puts a finalizer on all 23 KafkaTopics, and once the operator is gone nothing clears it.
kubectl get kafkatopics.kafka.strimzi.io -n gen0sec-system -o name \
| xargs -r -n1 kubectl patch -n gen0sec-system --type=merge \
-p '{"metadata":{"finalizers":null}}'
uninstall.sh does this for you, in the right order. See Uninstall.
A pod reports Running but nothing works
A pod in CrashLoopBackOff still reports status.phase: Running. Read the READY column, never a
field selector on phase.
kubectl -n gen0sec get pods
Agents get 401 on every relayed artifact
Your Gen0Sec API key is wrong, or the secret holds a trailing newline.
kubectl -n gen0sec logs deploy/g0s-download-api | grep -i 'relayed from upstream' | tail -5
artifact relayed from upstream path=/v1/ids-rules/version status=401
Recreate the secret with printf, not echo. --from-file stores the bytes exactly, so a newline
becomes part of the key. See Data relay.
A missing secret does not produce this: the relay pod fails to start instead. See the next entry.
CreateContainerConfigError on the relay pod
The pod references a secret that does not exist. Almost always
gen0sec-download-proxy: downloadProxy.apiKeySecret.name is set but the secret was never created.
kubectl -n gen0sec describe pod -l app.kubernetes.io/name=download-api | tail -20
kubectl -n gen0sec get secret gen0sec-download-proxy
Create it. See Install step 2.
This failure is the visible one. The dangerous case is the opposite: leaving
apiKeySecret.name unset renders no secret reference at all, the pod starts, and the failure is the
silent 401 above.
The relay reports 502 on every relayed route
The upstream is unreachable. A 502 is deliberate rather than a 404 or 500: a polling agent reads it as "ask again later" and keeps the artifact it already holds.
kubectl -n gen0sec exec deploy/g0s-download-api -- curl -fsS localhost:9999/status/healthz
Check that api.gen0sec.com:443 is allowed. See
Ports and egress.
If you route egress through an explicit proxy, that is the cause: proxies are not honoured in-cluster. See Known limitations.
Routing
An agent gets HTML, or a redirect, from an API
The request reached the dashboard's catch-all instead of the intended service. Routing is by longest
matching prefix and the dashboard owns /, so any path missing from the route table lands there.
kubectl -n gen0sec get ingress -o wide
Compare against Publish the endpoints. The failures this causes are quiet by design of HTTP, not by ours:
| Missing route | What the agent sees |
|---|---|
/v1/authcheck | A redirect and HTML. The agent cannot distinguish a bad key from a routing mistake and continues unverified |
/v1/agents/events | SSE connect returned HTTP 404, then silent fallback to polling, which looks like a slow platform |
/v1/ban-ledger | A redirect the agent reads as success. The contribution is lost silently |
kubectl get ingress shows an empty HOSTS column
The ingress controller ignores host-less rules: the object is created, nothing is logged, and no
traffic is served. Set entrypointHost. See Publish the endpoints.
Air gap and mirroring
Confirming the cluster really is isolated
From a pod, not from a node.
kubectl -n gen0sec run netcheck --rm -it --restart=Never \
--image=$REGISTRY/gen0sec/busybox:1.36.1-glibc -- wget -T5 -qO- https://ghcr.io/ ; echo "exit=$?"
A non-zero exit is the expected result. If it succeeds, the cluster still has egress and an install that appears to work may be pulling from the internet.
An image is missing for one architecture
You built the bundle with --arch. A single-architecture bundle fails when the kubelet tries to pull
on a node of the other architecture, which on an isolated cluster is after the transfer.
Rebuild without --arch, or with the architecture your nodes actually run. See
Prepare the artifacts.
Still stuck
Collect diagnostics, then send us the archive.