Back up and restore
What to back up
Not everything in the deployment is equally valuable. Backing up the wrong things is how backup jobs become expensive and untested.
| Component | Back up | If you lose it |
|---|---|---|
| Postgres | Yes, always | Everything. Configuration, agents, policy, history. This is the system of record |
| Object store | Yes | The datasets and the artifacts your cluster produced about itself. Datasets are re-fetchable, identity and policy-edges are not |
| Kafka | No | In-flight messages only. It is a transport, not a store. Producers re-send |
| Shared cache | No | Nothing. It repopulates |
| Secrets | Yes | Access to your own volumes. See the warning below |
The database passwords are generated by the Postgres operator and stored as Kubernetes secrets. Data is written with those credentials. A volume backup without the matching secret backup restores data you cannot open.
Postgres
The Postgres operator ships a logical backup image, already mirrored as part of the release. Enable
it in the infra values rather than scripting pg_dump yourself.
# gen0sec-infra values
postgres-operator:
configLogicalBackup:
logical_backup_schedule: "0 2 * * *"
logical_backup_provider: s3
Verify a backup ran:
kubectl -n gen0sec-system get cronjob | grep logical-backup
kubectl -n gen0sec-system get job | grep logical-backup
Restore into a scratch namespace once, before you need to. The Quickstart cluster is a reasonable place to rehearse it, and rehearsing a restore is also a rehearsal of an upgrade rollback.
Object store
Two prefixes matter, and they matter differently.
| Prefix | Recoverable without a backup? |
|---|---|
identity/, policy-edges/ | No. Produced by your cluster about your cluster |
| Dataset prefixes | Yes. They are fetched from Gen0Sec |
Back up the whole bucket with any S3-compatible tool, with an in-cluster job or manually, over a port forward as in Install step 5:
kubectl -n gen0sec-system port-forward svc/rustfs-svc 9000:9000
aws --endpoint-url http://localhost:9000 s3 sync s3://platform-data ./platform-data-backup
Restore is the same command with the arguments reversed.
Secrets
kubectl -n gen0sec-system get secret -o yaml > secrets-gen0sec-system.yaml
kubectl -n gen0sec get secret -o yaml > secrets-gen0sec.yaml
These files contain live credentials in base64. Treat them exactly as you treat the database itself: encrypted at rest, access controlled, and out of any shared drive.
If you run an external secret manager, you do not need this: your manager is already the source of truth and the cluster holds copies. See Secrets.
Recovery point and recovery time
| Value | Determined by | |
|---|---|---|
| Recovery point | Your backup interval | The logical backup schedule. Nightly gives up to 24 hours of loss |
| Recovery time | Measure it | Database size and your storage throughput. Rehearse it and record your own figure |
We do not publish an RTO as it depends entirely on your configuration specifics and the data size. Time your own rehearsal and use that number in your own runbook.
Restoring
Order matters, and it is the install order.
- Install infra and data as on a first install, so the operators and empty clusters exist.
- Restore the secrets, before anything tries to use the volumes.
- Restore Postgres from the logical backup.
- Restore the object store bucket.
- Install the platform. The migration hook runs against restored data, so the restored schema must match the release you are installing. Restore into the version it came from, then upgrade.
- Run the smoke test:
$KIT/scripts/verify-deployment.sh.
The migration chain expects to run forward from the schema version in the dump. Restoring a v0.1 backup and installing v0.3 skips the intermediate migrations. Restore v0.1, then upgrade one minor at a time. See Upgrade.