Skip to content

[python] Use Native planning and reads for REST query authorization - #10539

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/native-query-auth-masks
Oct 13, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/native-query-auth-masks

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Dependency

Depends on apache/paimon-rust#1122. Keep this PR in draft until that Rust change is merged. Python Native CI continues to build paimon-rust main; the new Native authorization tests need the corresponding Rust core implementation.

What changes

Use Native planning and reads for existing REST query-authorized batch reads. Support the Java QueryAuthSplit type-5 wire format so the plan carries the captured policy alongside its data/indexed split metadata. Only original Native splits retain a live Rust authorization grant; a serialized plan or Native reader setup failure falls back through Python with the captured rules.

Align Python execution with Java ReadTransform: authorization filter on original values, column masks, the user predicate, LIMIT, then output projection. Remove masked columns from raw partition/stats/PK/bucket pruning, include hidden query fields and their mask dependencies, and include hidden query/partition operands in the REST authorization request. Scan-only partition selectors referencing masked fields are rejected as Java requires.

Fix PAD/OVERLAY integer validation and eager overflow checks despite NULL string inputs, and validate stored field families for the new string transforms. Keep row-kind metadata outside positional predicate operands.

No new public Python or Rust binding API is introduced. The new mask execution stays in Rust core.

Verification

  • Native authorization suite: 64 REST tests across Append, PK and Data Evolution, plus 3 Native explain tests.
  • Combined Native authorization / existing REST / resolved-schema suites: 184 passed before the 3 explain tests were added; 211 Native plans and 187 Native reads exercised.
  • Related Python unit suites: 423 passed, plus 69 subtests. Final parser/split golden subset: 119 passed and 25 subtests.
  • Actual Java-generated QueryAuthSplit golden round-trips exactly, preserving UTF-8/NUL policy text, UTF-16 map ordering, indexed row ranges and scores.
  • Regressions cover Native setup failure, disabled Native reads, wire/pickle round trips, policy changes after planning, hidden masked predicates before LIMIT, masked partitions, authorization request scope, and PK merge with row-kind metadata.
  • Changed-file flake8 and Python 3.6 syntax checks passed.

Scope

This enables REST batch authorization through Native for the supported Rust rules. Transformed authorization row filters and streaming authorization remain separate follow-up work. Explicit authorized partition selectors remain on Python until the binding can represent their scan-only semantics.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete change at b3c503b. The Python regression tests and native plan/read authorization tests passed; no correctness issues found.

@JingsongLi
JingsongLi marked this pull request as ready for review October 13, 2026 12:10
@JingsongLi
JingsongLi merged commit 075ca16 into apache:master Oct 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants