You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Variant] Support VariantType in native columnar-to-row conversion #5436
Add native columnar-to-row conversion for top-level Variant columns from the foundation merged in #5868.
Spark's UnsafeWriter encodes Variant as 4-byte value length + value bytes + metadata bytes. Generic Struct encoding would produce an incompatible UnsafeRow.
Implementation
Preserve Variant Field identity when initializing native C2R; the physical Struct datatype alone is insufficient. Add a dedicated writer consuming canonical [value: Binary, metadata: Binary] storage and emitting Spark's payload with its row offset/size and alignment conventions.
Preserve SQL NULL versus Variant null and reject malformed child/null combinations without panicking. Admit only the implemented top-level shape.
Completion
Round-trip through UnsafeRow.getVariant with exact value/metadata bytes and assert native C2R. Cover objects, arrays/scalars, both null forms, empty batches and columns before/after Variant. Retain nested fallback and Spark 3 behavior.
Parent: #5438. #5434 reuses this row contract for JVM shuffle; #5425 supplies computed Variant inputs.
changed the title [-]Support VariantType in native columnar-to-row conversion[/-][+][Variant] Support VariantType in native columnar-to-row conversion[/+]on Sep 11, 2026
Activity
peterxcli commentedon Aug 23, 2026
take
[-]Support VariantType in native columnar-to-row conversion[/-][+][Variant] Support VariantType in native columnar-to-row conversion[/+]