Operations

Day-2 reference for running Dreadnode after the initial install — inspecting, backing up, rotating, scaling, and upgrading a deployment that is already configured correctly. To change what the deployment is, see Configure; to change who can use it and what models it serves, see Users and organizations and Model deployments.

Examples use dreadnode as the release name and $NAMESPACE as the install namespace.

For Helm, set the namespace used by your release:

export NAMESPACE=dreadnode

For Embedded Cluster, enter its Kubernetes shell first:

sudo ./dreadnode shell
export NAMESPACE=kotsadm

Run sudo only on ./dreadnode shell, not on kubectl commands inside the shell. See Access Kubernetes if kubectl tries to connect to 127.0.0.1:8080.

Health checks

# All pods
kubectl -n "$NAMESPACE" get pods -l app.kubernetes.io/instance=dreadnode

# API health (use http:// if global.scheme is http)
curl https://dreadnode.example.com/api/v1/health

# Resource usage (requires metrics-server)
kubectl -n "$NAMESPACE" top pods -l app.kubernetes.io/instance=dreadnode

The API’s /api/v1/health endpoint checks Postgres connectivity. A 503 with {"status":"unhealthy","detail":"database unreachable"} means the API is running but can’t reach the database.

Restart components

Rolling restart — no downtime if replicas > 1:

# API
kubectl -n "$NAMESPACE" rollout restart deploy/dreadnode-api

# Frontend
kubectl -n "$NAMESPACE" rollout restart deploy/dreadnode-frontend

# Documentation
kubectl -n "$NAMESPACE" rollout restart deploy/dreadnode-docs

# StatefulSets (use with care — causes brief data-store unavailability)
kubectl -n "$NAMESPACE" rollout restart sts/dreadnode-postgresql
kubectl -n "$NAMESPACE" rollout restart sts/dreadnode-clickhouse
kubectl -n "$NAMESPACE" rollout restart sts/dreadnode-minio

Watch the rollout:

kubectl -n "$NAMESPACE" rollout status deploy/dreadnode-api

View applied configuration

# ConfigMap (non-secret env vars)
kubectl -n "$NAMESPACE" get cm dreadnode-api -o yaml

# Current resource state
kubectl -n "$NAMESPACE" get deploy,sts,ingress -l app.kubernetes.io/instance=dreadnode

Database access

PostgreSQL

# Port-forward
kubectl -n "$NAMESPACE" port-forward sts/dreadnode-postgresql 5432:5432

# Connect (in another terminal)
PGPASSWORD=$(kubectl -n "$NAMESPACE" get secret dreadnode-postgresql \
    -o jsonpath='{.data.password}' | base64 -d) \
    psql -h localhost -U admin -d platform

Or exec directly into the pod:

kubectl -n "$NAMESPACE" exec -it dreadnode-postgresql-0 -- psql -U admin -d platform

ClickHouse

# Port-forward the HTTP interface
kubectl -n "$NAMESPACE" port-forward sts/dreadnode-clickhouse 8123:8123

# Query
curl 'http://localhost:8123/?query=SELECT+1'

Or use the CLI inside the pod:

kubectl -n "$NAMESPACE" exec -it dreadnode-clickhouse-0 -- clickhouse-client

MinIO

Read the root credentials first — you need them either way:

ROOT_USER=$(kubectl -n "$NAMESPACE" get secret dreadnode-minio \
    -o jsonpath='{.data.rootUser}' | base64 -d)
ROOT_PASSWORD=$(kubectl -n "$NAMESPACE" get secret dreadnode-minio \
    -o jsonpath='{.data.rootPassword}' | base64 -d)

MinIO’s web console is disabled by default. It is not exposed through any ingress, and while it is running the MinIO pod contacts subnet.min.io at every start to check a licence — a call the deployment gains nothing from, and one that fails noisily on a disconnected install.

To inspect buckets without enabling it, use mc against the S3 API:

kubectl -n "$NAMESPACE" port-forward sts/dreadnode-minio 9000:9000

mc alias set dn http://localhost:9000 "$ROOT_USER" "$ROOT_PASSWORD"
mc ls dn
mc ls dn/org-data --recursive --summarize

If you would rather have the console, turn it on and let the pod restart:

minio:
  console:
    enabled: true

kubectl -n "$NAMESPACE" port-forward sts/dreadnode-minio 9001:9001

Open http://localhost:9001 and log in with the same root credentials.

Backups

Backup strategy depends on your environment. The chart deploys in-cluster PostgreSQL, ClickHouse, and MinIO by default — back up at the storage layer (PVC snapshots) or export data logically from inside the pods.

PostgreSQL

# Dump to a local file
kubectl -n "$NAMESPACE" exec dreadnode-postgresql-0 -- \
    pg_dump -U admin platform > dreadnode-pg-$(date +%Y%m%d).sql

Restore (destroys existing data):

# Drop and recreate
kubectl -n "$NAMESPACE" exec dreadnode-postgresql-0 -- \
    psql -U admin -d postgres -c "DROP DATABASE platform"
kubectl -n "$NAMESPACE" exec dreadnode-postgresql-0 -- \
    psql -U admin -d postgres -c "CREATE DATABASE platform"

# Restore
cat dreadnode-pg-20260416.sql | \
    kubectl -n "$NAMESPACE" exec -i dreadnode-postgresql-0 -- \
    psql -U admin -d platform

Caution

After restoring Postgres, restart the API so Alembic detects the current schema state: kubectl -n "$NAMESPACE" rollout restart deploy/dreadnode-api

PVC snapshots

If your storage class supports CSI snapshots:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: pg-snapshot
  namespace: dreadnode # use kotsadm for Embedded Cluster
spec:
  volumeSnapshotClassName: <your-snapshot-class>
  source:
    persistentVolumeClaimName: data-dreadnode-postgresql-0

Repeat for data-dreadnode-clickhouse-0 and data-dreadnode-minio-0.

External data stores

If you pointed Dreadnode at external services (RDS, managed ClickHouse, S3), use those services’ native backup tools. The chart doesn’t manage backups for external stores.

Secret rotation

Bundled data-store passwords and API security keys require different rotation procedures. Follow the matching subsection below; editing a generated Secret and restarting pods is not sufficient for bundled data stores.

Data store passwords

Data store Secrets have helm.sh/resource-policy: keep, so Helm preserves them across upgrades and reinstalls. Embedded Cluster also persists generated credentials as hidden KOTS ConfigValues and reuses them on every deployment.

Do not rotate a bundled data-store password by editing its Secret alone. PostgreSQL, ClickHouse, and MinIO also store or enforce the credential inside the running service, and MinIO requires both rootUser and rootPassword keys. A partial Secret replacement can take the platform offline.

For an external data store, rotate the credential at the service first, then update the referenced Kubernetes Secret on Helm or the protected field under Config → Data Stores on Embedded Cluster. Deploy the chart change and confirm the API reconnects before revoking the old credential. Reach out to us before rotating bundled data-store credentials in place.

API security keys

The dreadnode-api-security Secret holds secretKey, jwtSecretKey, and refreshSecretKey. Rotating these invalidates all active sessions and issued tokens — every logged-in user gets logged out.

The dreadnode-api-encryption Secret holds the Fernet key for encrypting user secrets stored in Postgres. Do not rotate this key unless you’re prepared to lose all encrypted user secrets. There is no re-encryption migration.

OpenSandbox API key

The dreadnode-sandbox-server-api-key Secret authenticates the API to the OpenSandbox lifecycle server. The chart generates it on first install and preserves the existing value on every upgrade, so a redeploy never rotates it silently.

Both sides read the same Secret, so rotate it in one step and restart both:

kubectl -n "$NAMESPACE" patch secret dreadnode-sandbox-server-api-key \
    -p "{\"stringData\":{\"api-key\":\"$(openssl rand -hex 32)\"}}"

kubectl -n "$NAMESPACE" rollout restart \
    deploy/dreadnode-sandbox-server deploy/dreadnode-api

Sandboxes already running keep working; lifecycle calls in flight during the restart fail and the caller retries. To manage the key outside the chart, create your own Secret and point both dreadnode-api.config.opensandbox.apiKey.existingSecret and dreadnode-sandbox-server.apiKey.existingSecret at it — the chart rejects a one-sided override at render time.

Scaling

Resource presets

The simplest way to scale is to change the resource preset. That is chart configuration, not an operational command — see Resource sizing for the preset table and Apply changes for how to roll it out on each install path.

Manual replica scaling

The API and frontend Deployments can be scaled horizontally:

kubectl -n "$NAMESPACE" scale deploy/dreadnode-api --replicas=3
kubectl -n "$NAMESPACE" scale deploy/dreadnode-frontend --replicas=2

This doesn’t survive helm upgrade. For persistent scaling, set replica counts in chart configuration:

Helm

dreadnode-api:
  replicaCount: 3
dreadnode-frontend:
  replicaCount: 2

Admin Console

For the API, open Config → Scaling & Resources, enable Set API replicas and worker concurrency, and set API Replicas. Frontend replicas have no dedicated field; set dreadnode-frontend.replicaCount under Advanced Helm Values.

KOTS ConfigValues

apiVersion: kots.io/v1beta1
kind: ConfigValues
spec:
  values:
    scaling_customize:
      value: "1"
    api_replicas:
      value: "3"
    advanced_helm_values:
      value: |
        dreadnode-frontend:
          replicaCount: 2

Note

PostgreSQL, ClickHouse, and MinIO are single-replica StatefulSets. Scaling them horizontally requires configuration changes beyond replica count (replication setup, shared storage, etc.) and is not covered here.

Upgrades

Helm CLI

helm upgrade dreadnode oci://registry.replicated.com/dreadnode/dreadnode \
    --version <new-version> \
    --namespace "$NAMESPACE" \
    -f values.yaml

Admin Console

The Admin Console checks for new versions automatically. When an update appears on the dashboard, confirm the version and click Deploy.

What happens during an upgrade

1. The migrations init container runs alembic upgrade head against Postgres
2. The API pod starts with the new version
3. The frontend pod rolls to the new version

Migrations are forward-only. helm rollback and the Admin Console Rollback button are disabled. If an upgrade fails, see Reinstall from scratch.

Support bundles

Support bundles collect logs, cluster state, and diagnostics into a single archive.

Admin Console: Go to Troubleshoot → Generate a support bundle.

Embedded Cluster CLI: From the directory that contains the installer, outside the Embedded Cluster shell:

sudo ./dreadnode support-bundle

Helm CLI:

kubectl support-bundle --load-cluster-specs -n "$NAMESPACE"

Requires the troubleshoot kubectl plugin. The bundle spec is built into the chart — the plugin discovers it automatically. Air-gapped installs need that binary staged before disconnecting, see Before you begin.

The summary printed at the end names any known problem it recognises, so read that before opening the archive. For what the bundle contains and how to send it to us, see Support bundles.