[viirya]

Which issue does this PR close?

Closes #7178.

Follows up a review comment on #6130.

Rationale for this change

The native scalar Arrow UDF path fails the task when a UDF's PySpark Python version differs from the embedded interpreter's. The embedded interpreter is the libpython that libcomet was linked against, so its version is fixed when libcomet is built. Spark only requires the workers' Python to match the driver's. After a Python upgrade without a libcomet rebuild, every native UDF query would fail where Spark succeeds.

What changes are included in this PR?

• NativeBase.pythonUdfVersion() returns the embedded interpreter's major.minor from Py_GetVersion. That function may be called before the interpreter is initialized, so planning does not start Python. It returns an empty string without the python-udf feature.
• CometArrowEvalPythonExec.getSupportLevel falls back to Spark when a UDF's pythonVer differs, and the fallback reason names both versions.
• The user guide documents the fallback. Its limitations paragraph now links Interrupt a native scalar Arrow UDF when its task is killed #6811 (interrupting a running UDF) and Surface native scalar Arrow UDF errors as Spark's PythonException #7177 (PythonException parity).

How are these changes tested?

A Rust test checks that the version read before initialization matches sys.version_info afterwards. A planning test next to the profiler check uses a UDF with pythonVer = "2.7" and asserts the fallback and its reason. Planning-only tests that hard-coded a Python version and expected native execution now use the embedded version.

[sunchao]

Reviewed the full six-file diff from 785fdbcb4cbf9057dfcfae3bd9dd45a77bb505bb to 319f2b9fefa32489c1c43bc9fdd723d16303a600. The PR is not a draft. No introduced P1/P2 issues found within this review.

Read AGENTS.md and applied review-comet-pr and review-comet-ffi-pr. The snapshot and live discussion contained no reviews, conversation comments, inline comments, or review threads. Also checked issue #7178 and its originating review comment.

Summary

• Prior state and problem: Native Arrow UDFs detected Python version mismatches when creating the callable, failing tasks even when Spark's external workers could execute them successfully.
• Design approach: Expose the embedded interpreter's major.minor version through JNI and reject native planning when a UDF requires a different version.
• Correctness: Spark 4.1.3 and 4.2.0 populate pythonVer from sys.version_info[:2], and check_python_version compares that same format. The new guard matches this contract, checks each UDF, and preserves the executor-side runtime check. No introduced P1/P2 correctness issue was identified.
• Compatibility analysis: Both supported Spark 4.1 and 4.2 profiles include the changed spark-4.1+ source. Earlier Spark profiles retain their existing behavior. Builds without python-udf remain unsupported. All six affected files on latest release branch branch-1.2, at 889856dc5af821de1f7c6ca2e9538bd663431d04, match the supplied base, so the release-relative behavior change is the intended mismatch fallback.
• Key design decisions: Py_GetVersion avoids starting Python during planning. Returning an empty version when the JNI capability is unavailable conservatively rejects ordinary versioned UDFs. Keeping the runtime check protects executor execution independently of planning.
• Implementation sketch: Rust extracts the version, the JNI entry point returns a Java string, NativeBase wraps capability detection, and getSupportLevel supplies a fallback reason naming both versions. Tests add mismatch coverage and replace hard-coded versions in planning fixtures.
• Performance: The added work is a version probe and small string allocations during planning, with no new per-row or per-batch work. Arrow buffer ownership and execution lifetimes are unchanged. No material performance regression was established, and no benchmark was run.
• Design: The check belongs naturally beside the existing native UDF eligibility guards. It resolves the failure before task execution without adding interpreter initialization or retry machinery to planning.
• Abstraction & complexity: The small native helper and Java accessor fit the existing capability bridge. No additional mutable state, cache, or configuration is introduced. No simplification meeting the P1/P2 reporting bar was identified.
• Behavioral changes worth calling out: A mismatched UDF now stays on Spark's worker path instead of failing in native callable creation. Matching versions remain eligible. The user guide documents this behavior and links existing cancellation and exception-parity limitations, which this PR does not introduce.
• Suggested improvements: No additional code change meeting the P1/P2 reporting bar was identified. Integration validation remains outstanding as described below.

Exact-head CI: Preflight is in progress. label, check-pr-title, and Analyze Actions are queued. There is no completed build/test verdict. No opt-in labels are applied, and the feature-enabled PyArrow suite runs through the merge queue or run-pyarrow-udf-tests.

Validation: git diff --check passed. A disposable probe compiled the exact embedded_python_version function with PyO3 0.28.3 against Python 3.12.13. All 1,001 calls left Python uninitialized, and the returned version matched sys.version_info after initialization. CPython source and documentation also confirm the pre-initialization API contract.

Validation limits: The focused repository Rust test could not build offline because locked dependency aws-config 1.13.0 was unavailable in the cache. Full native/JNI, Scala, and Spark SQL suites were not run. The local environment lacks javac, PyArrow, and PySpark. The isolated probe does not substitute for those integration tests. Project code was unchanged and nothing was published.

[viirya]

Thanks @sunchao