Maintenance

Canonical order#

Seven operations, skipping the ones a table does not need. Expiry comes after compaction so one run reclaims the space its own compaction freed.

A finished run shows that order table by table. Each bar is one table's job; a failed job stays on its table and the rest of the run carries on.

cryo.local/runs/7f3c9a21

Demo data

Why maintenance matters#

Apache Iceberg gives you hidden partitioning, snapshot isolation, schema evolution and engine-agnostic metadata. It does not promise that tables stay fast. Left alone, four things pile up, and each one compounds independently, so fixing one does not fix the others.

Compaction is not one operation#

Teams pick a strategy from their write pattern and query filters. The same primitive, optimize, does very different work depending on which one runs.

What each operation does#

convert_deletes#

Rewrites data files so delete files are applied and no longer re-merged on every read. On Spark it passes a delete-file threshold. On Trino it runs as an OPTIMIZE.

rewrite_position_deletes#

Consolidates position-delete files without rewriting data. Spark only, because Trino does not support it.

optimize#

Partition-aware compaction of small files, planned from stored scan data. This is the step the chunking rules below exist to protect.

compute_statistics#

Refreshes table statistics straight after compaction, while the new layout is fresh.

optimize_manifests#

Consolidates manifests. Trino calls it optimize_manifests and Spark calls it rewrite_manifests.

expire_snapshots#

Drops snapshots the retention policy would not keep, and the files only they referenced. Guarded, and forcing it is audited.

remove_orphan_files#

Deletes files in storage that no snapshot references. Guarded, and forcing it is audited.

Planner chunking#

A large table is never compacted in one statement. The planner ranks partitions by small-file count, highest first, skips those under the min_small_files threshold, and then picks one of three paths.

• Chunk under a GB budget. Ranked partitions are grouped until the budget is full, and a cap on partitions per chunk keeps Trino's filter planning tractable.
• Oversize partition alone. A partition larger than the budget runs by itself.
• Small-table bypass. A small table skips chunking and gets a single full-table OPTIMIZE.

Admission: load-aware, never eager#

A worker asks the governor for permission before it runs anything. The governor probes live load, from system.runtime.queries on Trino and from slots on Spark on EMR, and either admits the job or defers it. It also applies per-replica caps and round-robins across replicas.

The Runs page shows what ran, what failed and what is held. It reads top to bottom:

• Pulse tiles. Running now, failed runs in the last 24 hours, succeeded in the last 24 hours, the 7-day success rate, and the scanner. The first three tiles filter the history when clicked.
• Run timeline. One bar per recent run, taller for longer runs and coloured by state. A running run is striped. As in Airflow's grid view, a run of failures or a slow night stands out before you read a row.
• History. Filter chips for All, Active and Failed · 24h, each with a count. Each row shows state, run, start time (relative, with the exact time under it), a duration bar scaled to the slowest run on the page, bytes freed, and the trigger.
• Engine load and the last nightly funnel sit at the bottom. Engine load shows the slots the governor is using against each destination's cap.

cryo.local/runs

Demo data

Three guardrails, always on#

Cooldown

Skip tables maintained within the last N hours, read from the audit log and engine history.

Per-table lock

A Redis lock held for the job's lifetime, so a table is never maintained twice at once.

Failure isolation

One job's failure is recorded against that table. The run continues with the next one.

Pausing maintenance#

A hold stops new maintenance without cancelling anything. It is the switch for an engine upgrade, an incident, or a change freeze. Set one from Runs → Holds:

Hold destination

Pauses one engine. Every other engine keeps running. Operators and admins can set it, and the dialog lists each configured engine by name.

Hold all maintenance

Pauses every engine. Admin only.

Both need a reason of at least three characters, and resuming needs a verification note. Every change is recorded with who made it and a new control generation.

cryo.local/runs?tab=holds

Demo data

While a hold is active:

Excluding a schema or table#

An exclusion keeps Cryo's maintenance away from part of the lake, for example a schema another team compacts itself, or a table under migration. Each rule names one lakehouse and one schema, optionally narrowed to one table, and needs a reason.

• An excluded table is skipped by the nightly batch, by scheduled presets, by manual runs and by smart-maintenance layout changes. A manual run where every selected table is excluded returns 409 and nothing is queued.
• The table's maintenance status reports it as blocked, with the reason operator_excluded.
• Scans still read excluded tables, so their health and cost stay visible.
• Removing the rule includes the table again from the next run. Adding and removing rules are both audited.

Exclusions are managed through the API for now, at /api/v1/maintenance/exclusions:

$ curl -X POST "$CRYO/api/v1/maintenance/exclusions" \
    -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
    -d '{"source_id": "<lakehouse id>", "schema_name": "finance", "reason": "Compacted by the finance team"}'

Add "table_name" to exclude one table instead of the whole schema. GET lists the rules, with the number of tables each one covers, and DELETE /api/v1/maintenance/exclusions/{id} removes one. Viewers can list; operators can add and remove.

Where to go next#