Backup and restore

What to back up#

PostgreSQL: back it up

The cryo schema is Cryo's source of truth. It holds sources, destinations, presets, table stats and grades, plans, runs and jobs, approvals, pause controls, fences and the audit trail. Connection secrets in it are encrypted with CRYO_SECRET_KEY.

The Secret: back it up separately

A database backup is useless without the CRYO_SECRET_KEY it was written with, because the stored connection secrets can't be decrypted without it. Keep the Secret in your secret manager, not next to the backups.

Redis: no backup needed

Redis holds the job queue, leases and short-lived markers. The governor's reconcile pass rebuilds the queue from PostgreSQL after a restart. The bundled Redis has no volume on purpose. See Losing Redis for the one thing that isn't rebuilt.

If your PostgreSQL is a managed service with snapshots or point-in-time recovery, that's a valid backup. The rest of this page covers Cryo's own tool, which adds encryption and an offsite CronJob.

The backup key#

Cryo's backups are encrypted with AES-256-GCM using CRYO_BACKUP_KEY: 32 random bytes, encoded as URL-safe base64. Generate one, store it in your secret manager, and add it to the cryo-secrets Secret. Without this key, a backup can't be restored.

$ openssl rand 32 | base64 | tr '+/' '-_'

Scheduled offsite backups#

The chart has a backup CronJob, which is off by default. It runs python -m cryo.operations.backup create-offsite with the Cryo image and the same Secret. It takes a pg_dump --format=custom, encrypts it, and uploads the encrypted file first and its checksum manifest last. The manifest marks a complete backup. Retention deletes only Cryo's own timestamped backups under the prefix.

backup:
  enabled: true
  schedule: "0 2 * * *"
  s3Uri: s3://company-recovery/cryo/production/
  retentionDays: 30
  serviceAccount: cryo-backup

serviceAccount must already exist and be able to write, list and delete under s3Uri, for example through IRSA. The chart doesn't create it. Use a bucket in a separate account or region, with versioning and object lock turned on. The CronJob's schedule is in UTC.

$ kubectl -n cryo get cronjob cryo-backup
$ kubectl -n cryo logs -l app.kubernetes.io/component=backup --tail=20

A successful run prints a JSON result with the backup and manifest S3 URIs.

A one-off backup#

Take one before every upgrade. The API pod already has the database URL and the backup key from the Secret.

$ POD=$(kubectl -n cryo get pod -l app.kubernetes.io/component=api -o jsonpath='{.items[0].metadata.name}')
$ kubectl -n cryo exec "$POD" -- python -m cryo.operations.backup create --output /tmp/cryo.dump.enc
$ kubectl -n cryo cp "$POD:/tmp/cryo.dump.enc" ./cryo.dump.enc
$ kubectl -n cryo cp "$POD:/tmp/cryo.dump.enc.manifest.json" ./cryo.dump.enc.manifest.json
$ kubectl -n cryo exec "$POD" -- rm /tmp/cryo.dump.enc /tmp/cryo.dump.enc.manifest.json

The tool passes the database password through PGPASSWORD, never on the command line. The pg_dump in the image comes from the Debian postgresql-client package. If your server's major version is newer than that client, take the dump with your own tooling instead.

Restore#

Restoring is destructive. It runs pg_restore --clean --if-exists against CRYO_DATABASE_URL. Practise it first in an isolated environment with an empty database. Keep everything that writes to the database stopped until you've checked the result.

1. Stop the writers

   Scale the scheduler, governor, workers and scanner workers to zero, and suspend the CronJobs. The API stays up so you can run commands in it.

   $ kubectl -n cryo scale deploy/cryo-scheduler deploy/cryo-governor deploy/cryo-worker deploy/cryo-scanner-worker --replicas=0
   $ kubectl -n cryo patch cronjob cryo-scanner -p '{"spec":{"suspend":true}}'

   Suspend cryo-maintenance and cryo-backup the same way if you turned them on.

2. Copy the backup in and restore it

   The manifest must sit next to the backup. The tool checks the ciphertext against the manifest and then checks the decrypted dump before it restores anything. --force is required.

   $ POD=$(kubectl -n cryo get pod -l app.kubernetes.io/component=api -o jsonpath='{.items[0].metadata.name}')
   $ kubectl -n cryo cp ./cryo.dump.enc "$POD:/tmp/cryo.dump.enc"
   $ kubectl -n cryo cp ./cryo.dump.enc.manifest.json "$POD:/tmp/cryo.dump.enc.manifest.json"
   $ kubectl -n cryo exec "$POD" -- python -m cryo.operations.backup restore --backup /tmp/cryo.dump.enc --force
   $ kubectl -n cryo exec "$POD" -- alembic upgrade head

3. Check before you start anything

   Sign in and check that sources, destinations, presets, tables and recent runs are all there. Open one source and run its connection test, which proves the stored secrets decrypt with this CRYO_SECRET_KEY. Open Control Room (/control-room) and check that the pauses are what you expect before maintenance can start.

4. Start the writers again

   Run your normal helm upgrade command. It puts back the replica counts from your values, and the CronJobs' suspend settings.

   Done when every pod is ready and the next scan and run behave normally.

Write down the start and end times, the backup's manifest hash, and anything that went differently from this procedure. That record is your measured restore time.

Losing Redis#

When Redis restarts empty, the governor rebuilds the queue from PostgreSQL: queued jobs are put back, and stale running jobs and table locks are reaped. One known gap remains. The scheduler and the scan scheduler keep their "last cron fire" markers only in Redis. After a restart, every daily preset and the daily scan can fire again within 24 hours of their last run, at whatever time Redis came back. That repeats maintenance and uses engine time. This is tracked in #177. Until it's fixed, give Redis stable resources and avoid restarting it during the day.

Where to go next#