v0.1.5

Benchmark

We ran Cryo against the scheduled Spark jobs teams usually write for themselves, on real Iceberg tables, with three trials per test and every row checked afterwards. Here is what we measured.

Updated 6 min readEdit this pageReport an issueView as Markdown
On this page
151 / 151Cryo maintenance jobs succeededAcross every test, with no failed or partial commits.
0rows lost or changedTable fingerprints matched the untouched copy after every run.
4.3×fewer delete files under live CDCTime-weighted average of 4.9 vs 20.9, while the scheduled DAG's compactions all failed.
1.76×faster queries on a 100 GiB tableGeometric mean of the whole query mix. Up to 4.3× on individual queries.
79%fewer pointless rewrites than a scheduled DAG43% fewer unnecessary operations once snapshot expiry is counted.
≤ 4%p95 slowdown for live queries during compactionEvery live query stayed between 0.88× and 1.04× of idle, with zero failures.

What we compared#

The real alternative to Cryo is usually not another product. It is a Spark job on a schedule that someone on the team wrote and now has to maintain. So that is what we measured Cryo against, using the same Spark and Iceberg versions and the same cluster.

No maintenance

The table as ingestion leaves it. Sets the baseline.

Scheduled Spark DAG

Compact, rewrite manifests and expire snapshots on every table, every cycle. The common Airflow pattern.

Hand-run Spark

An engineer running the exact compaction SQL by hand, on the large-table test.

Cryo

Default settings. Grades each table, plans only what it needs, runs it on Spark.

For a feature comparison with Amoro, LakeOps, Crunch, Databricks and managed cloud services, see Reference. This page reports only what we ran and measured.

It touches only the tables that need work#

We gave both approaches the same fleet: healthy tables, quiet ones, fragmenting ones, badly fragmented ones, and tables taking row updates. Over four cycles, the scheduled DAG rewrote everything it saw. Cryo left healthy tables alone and spent its work where the problems were.

Table state Scheduled Spark DAG Cryo Healthy 4 large files 16 not needed 8 expiry 8 expiry Quiet 12 small files 16 not needed 8 expiry 2 not needed 8 expiry Fragmenting 150+ small files 16 useful 8 expiry 8 useful 8 expiry Badly fragmented 600+ small files 16 useful 8 expiry 8 useful 8 expiry Row updates delete files 16 not needed 8 expiry 8 not needed 8 expiry Operations per table class over 4 cycles, trial 7 (trials 8 and 9 identical). Expiry is snapshot expiry, a no-op inside a 2-hour test.
Green chips are rewrites that fixed a real problem. Orange chips rewrote a table that was already healthy. Both approaches expire snapshots on every table by design.

Unnecessary rewrites per trial#

Scheduled DAG48
Cryo10

Identical in all three trials. Counting snapshot expiry, it is 88 vs 50.

Every table ended healthy#

No maintenance66%
Scheduled DAG100%
Cryo100%

Share of query runs that hit a healthy table. Cryo matched the brute-force DAG while doing less of the work nobody needed.

On a 100 GiB table, queries get faster#

One table, 100 GiB, fragmented the way streaming ingestion leaves it. One Cryo run at the default 256 MB target turned thousands of small files into a few hundred right-sized ones.

Before: 7,594 data files After Cryo: 690 data files 7,234 under 32 MiB, mean file 13.6 MiB 30 under 32 MiB, mean file 148 MiB small files (1 square = 20 files) right-sized files
67 GiB rewritten in one run of about 21 minutes. Same rows and same results, a tenth of the files.

Query speed after one Cryo run

no change One day, small-file band 4.34× faster Aggregate, whole table 3.63× faster Customer filter, whole table 3.44× faster One day, mid-size band 1.96× faster Read every row 1.75× faster One day, already-large files unchanged 7-day join 1.1× slower Single-row lookup 1.8× slower Median latency, untouched vs Cryo default (256 MB target), geometric mean of 3 trials.
Scans and aggregates gained the most. Single-row lookups got slower because each one now opens a larger file; see trade-offs.
1.73× – 1.83×

Whatever target size you pick#

128, 256 and 512 MB targets all landed in this range for the whole query mix. The default is a safe choice.

690 = 690

Same output as hand-written Spark#

Cryo's compaction produced exactly the same file count, bytes and results as an engineer running the same SQL by hand.

538 – 802

Queries to pay back the rewrite#

Customer filters repaid the compaction compute after 538 runs, whole-table aggregates after 588 and full reads after 802.

Under live CDC, it keeps up#

This is the case that breaks naive schedules. A MERGE landed on the table every 200 seconds for two hours. The scheduled DAG's compactions collided with the writer and failed every time. Cryo converted delete files in small commits between writes and kept the table lean.

  • No maintenance
  • Scheduled Spark DAG
  • Cryo
0 10 20 30 40 0 min 20 min 40 min 60 min 80 min 100 min 120 min No maintenance and scheduled DAG Cryo Delete files on a table receiving a MERGE every 200 s Scheduled compaction failed 3 of 3 times (commit conflicts) Cryo, checking every 10 min: held at 3 to 6 from its second run
The no-maintenance and scheduled-DAG lines overlap because the DAG's work never committed. Ticks along the bottom mark Cryo's checks.
22 / 22

Cryo jobs succeeded#

Every job finished while the writer kept committing.

0 / 3

DAG compactions committed#

All three failed validation against the concurrent MERGE.

40 → 3

Data files at the end#

Same rows, and aggregate queries ran in 0.58× the time.

Users don't feel it#

We ran a steady stream of dashboard-style queries against the 100 GiB table while Cryo rewrote 67 GiB of it, then compared with the same queries on an idle table.

  • Idle table
  • During Cryo compaction
Point lookup 552 ms idle 485 ms during Cryo compaction One-day scan 813 ms idle 808 ms during Cryo compaction One-week scan 4,169 ms idle 4,195 ms during Cryo compaction p95 latency of live Trino queries, about 500 to 700 runs each, 0 failures.
p95 stayed within 0.88× to 1.04× of idle across every query, and p99 improved for all three. Zero query failures.

Scorecard#

Measured outcomeNo maintenanceScheduled Spark DAGCryo
Fleet restored to healthyNo100%100%
Leaves healthy tables alonen/aNo48 pointless rewritesMostly10 not needed, none on healthy tables
Keeps up with live CDCNo39 delete filesNo3 of 3 failedYes3 to 6 delete files
Data integrityIntactIntactIntact
Live-query p95 during maintenancen/anot measured≤ 4% slower
Decides what to runn/aYou doCryo does
Grade, history and audit trail per tableNoBuild itBuilt in

Trade-offs#

A benchmark you can trust shows the costs too. These are the ones we measured.

Spark compute

Cryo used about twice the Spark compute of the scheduled DAG on small tables, and 1.67× hand-run SQL on the 100 GiB table. It starts one Spark application per operation and expires snapshots on every table. Compute is cheap next to engineer time, but Cryo is not a compute saver.

Point lookups

Bigger files make single-row lookups slower (1.8× at 256 MB). If your workload is mostly point reads, use a smaller target or skip compaction for that table.

Latency to start

Jobs waited a median of about 3 minutes in the queue before Spark started. Fine for maintenance; not built for second-level reaction.

Rough edges

We filed what we hit: run status staying queued (#153), chunked scans skipping tables (#154), and missing file metrics on Spark-on-Kubernetes jobs (#155). None affected data integrity.

How we measured#

  • 3 trialsper test, 95% confidence intervals
  • 100 GiBlargest single table
  • 2 hoursof continuous MERGE for CDC
  • Every rowfingerprinted before and after
  • Same stackSpark 4.1 and Iceberg 1.11 on Kubernetes, for every contender
  • Read-onlyTrino for all query timing

Cryo ran with default settings throughout. The protocol was written and frozen before any scored run, compute was measured from Kubernetes pod requests, and rewritten bytes came from Iceberg's own snapshot summaries. Discovery was also checked at fleet scale: all 100 tables in a 100-table catalog were discovered, with exact file counts, in about two and a half minutes.

Hardware and versions#

The numbers above were measured on one cluster. The protocol, the harness, the raw per-trial data and the analysis scripts are all in the repository, so you can check them — or rerun the tests.
Nodes6
Node typer6a.8xlarge — 32 vCPU, 128 GiB, on-demand
Regionap-south-1 (Mumbai)
Spark4.1.2
Iceberg1.11.0
Trino480, read-only, 6 vCPU / 20 GiB guaranteed
Object storeS3 Standard

The benchmark used about 48 vCPU on average for about 24 hours in total, across every phase.

The harness is in the repository — the frozen protocol, the phase controllers, the raw per-trial data and the analysis scripts. Every analysis table on this page regenerates from the published data:

terminal
cd benchmarks/cryobench-v2
for s in p1_analysis p2_analysis p3_analysis; do python harness/analysis/$s.py; done

The claims register says what each phase does and does not establish. It also records the claims this benchmark does not support — that compaction always improves latency, that Cryo saves compute, that it maintains tables faster than a scheduled job — which is the more useful half of the result.

Esc

↑↓ to navigate↵ to open