Collect diagnostics

Collect this before you contact us. Everything here is something we would otherwise ask for, so sending it means the first reply is an answer rather than a request for more output.

There is no single collection command at this release. This page is the list.

Set up

Run the commands from a directory you can archive afterwards.

NS_SYS=gen0sec-system
NS_APP=gen0sec
OUT=gen0sec-diagnostics
mkdir -p $OUT && cd $OUT

Cluster and storage:

kubectl version -o yaml                              > cluster-version.yaml
kubectl get nodes -o wide                            > nodes.txt
kubectl describe nodes                               > nodes-describe.txt
kubectl get storageclass -o yaml                     > storageclasses.yaml
kubectl get pvc -A -o wide                           > pvc.txt

Releases and workloads:

helm list -A                                         > helm-releases.txt
for ns in $NS_SYS $NS_APP; do
  kubectl -n $ns get all -o wide                     > workloads-$ns.txt
  kubectl -n $ns get events --sort-by=.lastTimestamp > events-$ns.txt
  kubectl -n $ns get pods -o yaml                    > pods-$ns.yaml
done

Logs from anything not ready:

for ns in $NS_SYS $NS_APP; do
  for p in $(kubectl -n $ns get pods -o json \
      | jq -r '.items[] | select(any(.status.containerStatuses[]?; .ready == false)) | .metadata.name'); do
    kubectl -n $ns logs $p --all-containers --tail=2000            > logs-$ns-$p.txt 2>&1
    kubectl -n $ns logs $p --all-containers --previous --tail=2000 > logs-$ns-$p-previous.txt 2>&1
  done
done

Operators and custom resources:

kubectl -n $NS_SYS logs deploy/g0s-infra-postgres-operator --tail=2000 > logs-postgres-operator.txt
kubectl -n $NS_SYS logs deploy/strimzi-cluster-operator     --tail=2000 > logs-kafka-operator.txt
kubectl -n $NS_SYS logs deploy/g0s-infra-synapse-operator   --tail=2000 > logs-ingress-operator.txt

kubectl -n $NS_SYS get postgresql,kafka,kafkanodepool,kafkatopic -o yaml > custom-resources.yaml

The migration job, which is the most common single cause:

kubectl -n $NS_APP logs job/g0s-db-migrate --tail=2000 > logs-db-migrate.txt 2>&1

Images actually in use, including init containers:

for ns in $NS_SYS $NS_APP; do
  kubectl -n $ns get pods -o yaml | grep -E '^\s+image:' | sort -u > images-$ns.txt
done

Routing and the relay:

kubectl -n $NS_APP get ingress -o yaml > ingress.yaml
kubectl -n gen0sec get --raw \
  '/api/v1/namespaces/gen0sec/services/g0s-download-api:http/proxy/status/healthz' > relay-healthz.json 2>&1

Then pack it:

cd .. && tar -czf gen0sec-diagnostics-$(date +%Y%m%d-%H%M%S).tar.gz gen0sec-diagnostics

Read the archive before you send it

It is plain text and it is yours. Nothing here redacts anything, so check it, and remove any sensitive data it might have, particularly if you added helm get values, which prints whatever you set.

None of the commands above read secret contents, database rows or object store objects. Logs can contain request paths and key prefixes.

There should be one command for this

A single collect-diagnostics.sh in the install kit would replace this page with two lines, and the list above is the specification for it. A single diagnostics tool can accidentally collect data that's sensitive and may fail to redact that information, so that's why we ask you to assemble your own report. Once we're 100% sure we can provide a diagnostics tool that doesn't leak any information of your sensitive data, we'll do so.

What to tell us alongside it

Four lines, and they save a round trip:

Which install method you used.
The release version.
The last command you ran, verbatim.
What you expected to happen.

Set up​

What to tell us alongside it​

Set up

What to tell us alongside it