[EPIC] Native Variant Support #5438

Track native Variant support across scans, expressions and transport. Completed milestones stay checked; each linked issue owns its implementation and parity tests.

Current direction

#5978 remains the whole-value performance priority. Buffer reuse landed in #6342; removing the extra reconstruction pass awaits apache/arrow-rs#11260. #5519 is in progress in draft #6443. The next expression steps are #5429, then #5424 and #5425. Arrow/DataFusion upgrades and the associated cleanup in #5477 are deferred.

Progress

• Establish top-level whole-value Parquet scans — feat: support direct Variant projection in native Parquet scans #5868.
  • [Variant] Native top-level Variant projection from ordinary Parquet #5546 — project Variant from ordinary Parquet.
  • [Variant] Export complete Arrow Fields through the native FFI boundary #5547 — export complete Arrow Fields through FFI.
  • [Variant] Carry Spark VariantType identity through Comet schema serialization #5548 — preserve Spark Variant identity in schema serialization.
  • [Variant] Normalize marked Variant arrays at the native Parquet boundary #5549 — normalize marked Variant arrays at the scan boundary.
  • [Variant] Match Spark physical Parquet semantics for projected Variant columns #5550 — match Spark physical Parquet semantics.
  • [Variant] Enable direct top-level Variant projection in ordinary native Parquet scans #5551 — enable direct top-level Variant projection.
• Prune unread Variant columns — feat: Support native scans with unprojected Spark 4 VARIANT columns #5377.
• Match Unicode column names — Support Spark-compatible Unicode case-insensitive Parquet field matching #5495, merged feat: support unicode case sensitive field names for reading parquet #5602.
• Complete whole-value scan integration.
  • [Variant] Support Spark's pushed one-field VariantStruct (whole-value pushVariantIntoScan rewrite) in native Parquet scans #5519 — support Spark's one-field whole-value scan rewrite; draft feat: support whole-value Variant scan pushdown #6443.
  • [Variant] Support native Parquet projection of VariantType nested in struct, array, and map columns #5435 — read Variant nested in structs, arrays and maps.
  • [FEATURE] Native scan support for VariantType columns (Iceberg + Spark 4.0) #4295 — support Iceberg projection, schema evolution and equality deletes on legal primitive keys.
  • Parquet Variant annotation validation is bypassed when reading as an ordinary struct #5741 — validate Parquet Variant annotations; open fix: validate Parquet VARIANT annotation against the requested read type #5932.
• [Variant] Remove the extra Spark byte-reconstruction pass #5978 — remove repeated reconstruction and meet the whole-value performance target.
  • Reuse reconstruction buffers — perf: reuse Variant reconstruction buffers #6342.
  • Expose the upstream row decoder — [Variant] Expose a row decoder for custom unshredding output arrow-rs#11260.
  • Integrate one reconstruction pass and benchmark canonical, partially shredded, fully shredded and empty-key inputs against vanilla Spark.
• Complete compatibility cleanup — [Variant] Track and remove native Variant compatibility workarounds after upstream fixes #5477.
  • Adopt upstream Arrow/DataFusion fixes and remove the corresponding workarounds — deferred.
  • [Variant] Consolidate Spark-compatible missing-value validation and errors #5977 — consolidate missing-state validation and Spark errors.
  • [Variant] Replace missing-key metadata repair with native permissive reconstruction #5979 — reconstruct inputs with missing metadata keys.
  • [Variant] Preserve typed-value precedence without residual pre-rewriting #5980 — preserve typed-value precedence without residual rewriting.
  • [Variant] Remove Variant UTF-16 output rewriting #5474 — remove UTF-16 output rewriting when supported Spark versions permit it.
• Add native extraction.
  • [Variant] Support native variant_get and try_variant_get for literal paths and scalar targets #5424 — literal paths and scalar targets.
  • [Variant] Support Variant-valued native expression output and two-argument variant_get #5425 — Variant results using [Variant] Support native variant_get and try_variant_get for literal paths and scalar targets #5424's extraction, with a shared contract for Variant producers.
  • [Variant] Complete native variant_get and try_variant_get support for dynamic paths and nested targets #5426 — dynamic paths and nested targets.
• Add other native expressions.
  • [Variant] Support native is_variant_null and Spark 4.2 is_valid_variant #5429 — Variant-null and validity predicates.
  • [Variant] Support native casts to and from VariantType #5430 — casts to and from Variant.
  • [Variant] Support native parse_json and try_parse_json #5428 — JSON parsing.
  • [Variant] Support native to_variant_object #5431 — object construction.
  • [Variant] Support native schema_of_variant and schema_of_variant_agg #5427 — schema inference.
  • [Variant] Support native variant_explode and variant_explode_outer #5432 — explode.
• Support Variant transport.
  • [Variant] Support VariantType in native columnar-to-row conversion #5436 — native columnar-to-row conversion.
  • [Variant] Support Variant payloads through Comet shuffle and spill paths #5434 — shuffle and spill; JVM row shuffle uses [Variant] Support VariantType in native columnar-to-row conversion #5436's encoding.
  • [Variant] Support top-level VariantType in Comet MapInArrow and MapInPandas transport #5437 — MapInArrow and MapInPandas.
• Support writes and selective shredded reads.
  • [Variant] Support whole-value VariantType in native Parquet writes #5433 — whole-value Parquet writes.
  • [FEATURE] Shredded Parquet Reader/Writer support for Variant type #3983 — shredded writes, selective subfield reads and predicate pushdown; writes build on [Variant] Support whole-value VariantType in native Parquet writes #5433 and selective reads on [Variant] Support native variant_get and try_variant_get for literal paths and scalar targets #5424/[Variant] Complete native variant_get and try_variant_get support for dynamic paths and nested targets #5426.

The merged scan foundation supports canonical and shredded input with spark.sql.variant.allowReadingShredded=true and spark.sql.variant.pushVariantIntoScan=false. #5519 addresses the latter restriction. #5407 retains the original prototype discussion; it was closed without merging.

Shared contract

Every implementation must preserve:

• Spark VariantType as Struct<value: Binary, metadata: Binary>, in that child order.
• ARROW:extension:name=arrow.parquet.variant on the outer Arrow Field, including expression results and FFI schemas. An unmarked lookalike struct is not Variant.
• SQL NULL as a null parent, distinct from a present Variant-null payload.
• Spark results, byte/layout requirements and errors at the boundary being added, with explicit fallback for unsupported consumers and unchanged Spark 3 behavior.

Spark does not define general Variant ordering or hashing; Variant keys are outside this roadmap. Iceberg Variant payloads may coexist with equality deletes on legal primitive keys.