Skip to content

[Variant] Support native is_variant_null and Spark 4.2 is_valid_variant #5429

Open

Description

@peterxcli

Implement native is_variant_null (Spark 4.0+) and is_valid_variant (Spark 4.2). The whole-value input foundation landed in #5868.

Required behavior

Input is_variant_null is_valid_variant
Variant null true true
Other valid Variant false true
SQL NULL false SQL NULL
Malformed value/metadata Match Spark's predicate-specific error behavior false

Use Spark's null predicate/helper and validity predicate as the oracle.

Implementation and completion

Add expression-specific serializers/evaluators over [value, metadata]. Validate both components for is_valid_variant and register it only in the Spark 4.2 shim.

Parity and native-plan tests must cover scalar/container kinds, both null forms, malformed value and metadata bytes, and version-specific registration. Keep unrelated Variant operators on fallback and Spark 3 compilation unchanged.

Parent: #5438. This is the first native expression slice before #5424/#5425.

Activity

  1. peterxcli commented on Aug 23, 2026

    @peterxcli
    MemberAuthor

    take

  2. changed the title [-]Support native is_variant_null and Spark 4.2 is_valid_variant[/-] [+][Variant] Support native is_variant_null and Spark 4.2 is_valid_variant[/+] on Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions