Variant Benchmark: Open-Sourced, Two Spark Optimizer Fixes, and a ~4x TPC-DS Speedup

In March 2026, our early benchmark numbers showed that Spark 4’s native Variant type was not yet faster than JSON-as-string — shredded Variant was actually 46% slower overall on the GitHub Archive workload. We treated those numbers as a diagnostic, not a verdict, and noted the implementation was still maturing.

Since then, the benchmark has shipped publicly and surfaced two problems in Spark’s Variant optimizer — one of them silently defeating pushdown on nearly every analytical query. Both are now fixed and slated for Spark 4.3, and the picture inverts: Variant now beats string JSON on TPC-DS by a wide margin.

The benchmark is open source

variant-conformance-benchmark is now public. It is a harness for validating correctness and measuring performance of the Parquet Variant type. The design is engine-neutral — engines plug in through an adapter interface — but today only Spark is implemented (Spark native tables and Spark + Iceberg); adapters for other engines are future work. It has two modes:

• Correctness mode: runs a fixed query suite against both string_json and variant representations of the same data and compares results row-by-row against golden reference files. Any mismatch is a conformance failure.
• Benchmark mode: measures query_median_s (Spark’s own built-in query execution time, median of N timed runs, excluding JVM startup) and writes per-query CSV files to results/ for comparison across runs.

The benchmark covers five data strategies, each designed to stress different aspects of Variant:

The tpcds-flat and tpcds-grouped strategies run 12 TPC-DS queries (q07, q12, q19, q26, q42, q52, q55, q63, q68, q73, q79, q98) covering multi-table JOINs, GROUP BY aggregation, ORDER BY, and filter conditions.

What the benchmark surfaced

Running these strategies against Spark 4 turned up two separate problems in the Variant optimizer. The first was a bug that skipped column pruning and, in a couple of cases, crashed or returned wrong results. The second was a missed optimization that quietly disabled pushdown on almost every real query — the source of the speedup below. Both are now merged and will ship in Spark 4.3.

SPARK-57499 — Column pruning skipped on DSv2 Variant scans

SPARK-57499 · PR apache/spark#56556 · Fixed, fixVersion 4.3.0

The rule that pushes variant_get() down into a scan had a bug that quietly turned off column pruning: whenever variant pushdown kicked in, the scan went back to reading every column in the table, even the ones the query never touched. That is especially costly for an unused VARIANT column, which Spark then rebuilds from its shredded Parquet layout on every row — pure wasted work. (The pruning step was being skipped because the pushdown replaced the plan node it keyed off of before pruning ran.)

The same rule also caused two failures the benchmark hit directly:

• Tables with two or more VARIANT columns crashed. The rule visited the same scan twice and clobbered its own work on the second pass, leaving a dangling column reference. The tpcds-grouped strategy — two VARIANT columns per table — hit this every run, failing with Could not find v1#57 in [a#72,v1#73,v2#74].
• Some queries returned wrong results. When a whole variant column was read as-is, the Parquet reader collapsed it to a placeholder value. ORDER BY variant_get(...) then sorted on the placeholder and returned rows in the wrong order.

If you are running Spark 4.3.0 or later, you already have this fix.

SPARK-58089 — Extraction pushdown doesn’t reach through Aggregate, Sort, or Join

SPARK-58089 · PR apache/spark#57190 · Improvement, Fixed, fixVersion 4.3.0

This is the one with the bigger performance impact. Spark classifies it as an improvement rather than a bug — nothing crashes, and results are correct — but the missed optimization is large. The tpcds-flat and tpcds-grouped strategies exist precisely because they put variant_get() inside the kinds of operators real analytical queries use:

-- q07: variant_get() in AVG arguments and a JOIN-side filter, across a 5-table JOIN
SELECT i.i_item_id,
       avg(variant_get(ss.data, '$.quantity', 'double'))     AS agg1,
       avg(variant_get(ss.data, '$.list_price', 'double'))   AS agg2,
       ...
FROM store_sales_a_variant ss
JOIN date_dim d ON ss.ss_sold_date_sk = d.d_date_sk
JOIN store s    ON ss.ss_store_sk     = s.s_store_sk
...
WHERE variant_get(ss.data, '$.coupon_amt', 'double') BETWEEN 0 AND 100
GROUP BY i.i_item_id
ORDER BY i.i_item_id;

The existing pushVariantExtractions rule could only push a variant_get() call down when it sat directly above the scan. But in real queries the call is usually buried one level up — inside an aggregate argument like AVG(variant_get(...)), a GROUP BY key, a JOIN condition, or an ORDER BY. The Aggregate, Join, and Sort operators sitting in between blocked the rule from seeing it.

Every TPC-DS query in the benchmark is shaped this way, so every one of them silently got no pushdown at all — the machinery existed, but nothing ever reached it.

The fix adds a new rule, PullOutVariantExtractions, that pulls those variant_get() calls out of the Aggregate, Sort, and Join operators and drops them into a Project sitting right above the scan. From there the existing pushdown rule can see them and push them the rest of the way in.

The numbers after the fix

The cleanest comparison is the same machine, same data, and same 12 queries run with the new rule on and off. All numbers are query_median_s — Spark’s own query execution time, median of 3 timed runs in a warm session:

Total query time drops from 31.0 s to 8.0 s — a ~4× improvement. The queries unaffected by the fix (q12, q19) are simple projections that do not use variant_get() in aggregate or join context; they are fast either way.

Compared to the string JSON baseline, Variant with the fix is 41% faster overall (8.0 s vs. 13.6 s; the PR reports a −39.3% geometric-mean delta). The queries most affected by the fix show the largest gains: q07 goes from tied with string to 2.3× faster, q52 from slower-than-string to 1.8× faster. Two queries (q42, q55) land at near-parity or slightly behind string — their extraction paths are already cheap enough at this scale that shredding doesn’t pay off. The win is consistent, not uniform.

These are scale-factor-5 TPC-DS results on a single machine in local mode. The ratios illustrate the optimizer behavior; absolute times will differ at scale. The PR description at apache/spark#57190 has the full run context and methodology.

What this means for Variant adoption

The early March 2026 results were a diagnostic. Variant’s type-safe extraction and shredded column layout mean the engine should skip full Parquet reads for selective filters — but only if the optimizer can see the variant_get() calls and push them into the scan. When the pruning bug and the pushdown gap blocked that path, shredded Variant paid the write-time cost without getting the read-time benefit, which is why it was slower than string.

With both SPARK-57499 and SPARK-58089 fixed and shipping in Spark 4.3:

• Column pruning no longer forces full-schema scans when some VARIANT columns are unreferenced.
• Pushdown is no longer silently defeated for queries with variant_get() inside aggregates, sorts, or joins — which is nearly every real analytical query.

If your workload uses variant_get() in GROUP BY, aggregation, or JOIN conditions against Spark 4, both fixes land in Spark 4.3 — upgrading is what unlocks the speedup above.

Try the benchmark on your data

The benchmark harness, query SQL, prep scripts, and result comparison tooling are all available at cloudera-labs/variant-conformance-benchmark. The quickstart guides in docs/ walk through each strategy. If you have a dataset where Variant shines — or where it should but doesn’t — open an issue there.

Contributions are welcome, too. The two most useful directions are new engine adapters (Trino, Flink, DuckDB, or anything else with a Variant path — the adapter interface is the plug-in point) and new data strategies (different JSON shapes, sizes, and query patterns). See docs/adding-an-engine.md and docs/adding-a-strategy.md to get started.

Once Spark 4.3 is out, re-running the benchmark against it on your own data is a concrete way to confirm the fixes hold for your workload, not just for TPC-DS.