Skip to main content

Upgrade

When this page is done you are running the new release, with the same data.

Read Roll back before you start. The rollback boundary changes what a safe upgrade looks like.

Before you start

  • Read the release notes for every version between yours and the target
  • A current backup. See Back up and restore
  • The new release's install kit, verified
  • A maintenance window. The application is briefly unavailable while pods roll

Rules that are not negotiable

RuleWhy
One minor at a timeSkipping a minor is not supported. Each train's migrations assume the previous train ran
Patches in placeWithin a train, upgrade directly
Same order as install: infra, then data, then platformEach chart depends on the previous one
Apply CRDs by hand firstHelm never upgrades CRDs. See step 1
Back up firstThe migration hook changes your schema, and helm rollback does not undo that

1. Apply the new CRDs

Do this before anything else. Helm installs a chart's CRDs on first install and ignores them from then on. An upgrade that ships a new CRD schema fails with a server-side-apply error, or worse, silently rejects fields it does not know.

cd $B/charts
tar xzf gen0sec-infra-*.tgz
kubectl apply --server-side --force-conflicts \
-f gen0sec-infra/charts/postgres-operator/crds/
kubectl apply --server-side --force-conflicts \
-f gen0sec-infra/charts/strimzi-kafka-operator/crds/

The output is similar to this:

customresourcedefinition.apiextensions.k8s.io/postgresqls.acid.zalan.do serverside-applied
This is not hypothetical

The Postgres operator 1.15 to 2.0 change adds PostgreSQL 18 to the postgresqls schema. Without the CRD apply, the postgresql/core resource is rejected outright and the database never starts.

2. Get the new artifacts

Follow Prepare the artifacts for the new release. Both methods work the same as on a first install, with one addition for the offline bundle.

Offline bundle only. Mirroring is additive: the new tags land beside the old ones. Your running install keeps working until you upgrade, and a rollback still has its images.

Do not reuse the previous bundle's directory. The bundler refuses to write into an existing one, and each bundle's values files are targeted at the version they shipped with.

3. Upgrade infra

Run every command from the new bundle directory, so charts/ and values/ are the new release's.

helm upgrade g0s-infra $B/charts/gen0sec-infra-*.tgz \
-n gen0sec-system \
-f $B/values/gen0sec-infra-values-onprem.yaml

Wait for the operators to finish rolling before continuing.

kubectl -n gen0sec-system rollout status deploy/g0s-infra-postgres-operator
kubectl -n gen0sec-system rollout status deploy/strimzi-cluster-operator
kubectl -n gen0sec-system rollout status deploy/g0s-infra-synapse-operator
kubectl -n gen0sec-system rollout status deploy/dragonfly
kubectl -n gen0sec-system rollout status \
"$(kubectl -n gen0sec-system get sts,deploy -o name | grep -m1 rustfs)"

4. Upgrade data

helm upgrade g0s-data $B/charts/gen0sec-data-*.tgz \
-n gen0sec-system \
-f $B/values/gen0sec-data-values-onprem.yaml
kubectl -n gen0sec-system wait --for=condition=Ready kafka/core --timeout=15m
kubectl -n gen0sec-system wait \
--for=jsonpath='{.status.PostgresClusterStatus}'=Running postgresql/core --timeout=15m

5. Upgrade the platform

The schema migration runs first, as a pre-upgrade hook, before any new pod rolls.

helm upgrade g0s $B/charts/gen0sec-platform-*.tgz \
-n gen0sec \
-f $B/values/gen0sec-platform-values-onprem.yaml \
--set global.imageTag=$VERSION \
--timeout 20m

If the migration fails, Helm aborts and the old application keeps running. That is the designed behavior and it is why the hook runs before the rollout.

kubectl -n gen0sec logs job/g0s-db-migrate

Fix the cause, then re-run the same command. See Migration failures.

6. Verify

kubectl -n gen0sec wait --for=condition=Ready pod --all --timeout=10m
$KIT/scripts/verify-deployment.sh

Run the smoke test after every upgrade, not just after installs. It is the only check that proves the platform still serves an agent rather than merely starting.

Full checklist: Verify the install.

If you changed values along the way

helm upgrade uses the values files you pass, not the ones from last time. Anything you set with --set on the previous install must be set again, or it reverts to the chart default.

Check what the running release actually has:

helm get values g0s -n gen0sec

Keep your overrides in a file rather than in shell history. See Reference configurations.

If it fails

SymptomStart here
CRD schema error on helm upgradeYou skipped step 1
Unsupported value from the Postgres operatorYou skipped step 1
Migration job failsMigration failures
Pods roll but the smoke test failsRoll back
ImagePullBackOff on new tagsThe new images were not mirrored. Offline bundle method, step 2

Next

Roll back.