Skip to content

[python] Refine post-mask query filtering for authorized reads - #10523

Open
plusplusjiajia wants to merge 5 commits into
apache:masterfrom
plusplusjiajia:fix/python-filter-after-column-masking
Open

plusplusjiajia wants to merge 5 commits into
apache:masterfrom
plusplusjiajia:fix/python-filter-after-column-masking

Conversation

@plusplusjiajia

@plusplusjiajia plusplusjiajia commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

Purpose

Master (#10539) evaluates the full query predicate after column masking, but it applies every mask before the filter and resolves BLOB payloads before LIMIT. This PR keeps exact post-mask filtering and fixes the remaining performance and semantic gaps:

  • Authorized reads evaluate every query predicate again before LIMIT and keep its columns until the final projection, because split reads may apply it loosely (nested projections, merged primary-key rows). Conjuncts that reference no masked column are still pushed down to scans and split reads.
  • The filter tests a view in which only the masks it reads are applied; rows keep their raw values and all masks are applied after LIMIT. BLOB payloads are deferred until after filtering and LIMIT, so payloads that are output or read only by output masks are fetched for surviving rows only, while payloads read by filter masks stay eager.
  • When a mask moves a column to another type family, such as a string mask on an INT column, comparisons, string predicates and null checks use the query-schema type, and values that do not fit it are treated as NULL, like SQL TRY_CAST. Numbers, times of one kind and nested columns are filtered as masked, so no precision is lost, and NOT IN with a NULL literal skips the conversion. NULL MAP values stay intact and _row_kind columns, including -D rows, pass through the filter.
  • Streaming scans authorize filter columns that are not projected. Nested-field filter aliases authorize only their physical source column, and row-tracking fields are recognized.
  • Scan pruning is rebuilt from the user predicate on every plan, so a later plan without masks restores full pruning. Only predicates on masked columns mark reads as filtered during reading, and chunk shuffling is rejected when authorization filters would resize chunks.

Tests

Adds regression coverage for append, primary-key and data-evolution tables, including partitions, projections, compound and OR predicates, LIMIT, streaming row kinds, chunk shuffling, NULL MAP values, type-changing masks, standalone equality and string predicates, selection-sensitive authorization for partition filters and hidden streaming filter columns, NOT IN with a NULL literal, masked values that do not fit the declared type (OR arms, NULL literals and null checks), numeric, timestamp and nested masks that change the column type, nested-field aliases, pushdown of unmasked conjuncts, OR predicates on unprojected columns under a nested projection, NULL comparisons after primary-key merges, and row-only authorization with a user filter. BLOB tests count payload reads: the post-mask LIMIT cases read 2 payloads instead of 10, including when an output-only mask reads a projected or unprojected BLOB.

  • Python 3.11 query-auth, BLOB-deferral and predicate-parser suites: 464 passed.
  • Python 3.11 read, filter, streaming, LIMIT, data-evolution and query-auth test files: 3051 passed; 15 failed only on missing optional modules (lance, vortex, duckdb), and 6 files need ray.
  • Python-reader cases of the native query-auth, transform-filter and calendar-mask tests, run locally with the Rust-runtime skip removed: 198 passed. Native paths are covered by the Rust Plan CI job.
  • Python 3.6 / PyArrow 6 query-auth filter suite: 194 passed; the MAP case is deselected because Polars 0.12.5 lacks nested-type support.
  • Flake8 and git diff --check passed.

@plusplusjiajia
plusplusjiajia force-pushed the fix/python-filter-after-column-masking branch 4 times, most recently from 2e6e8e8 to 5578a9a Compare October 13, 2026 08:41
@plusplusjiajia
plusplusjiajia marked this pull request as ready for review October 13, 2026 08:41
@JingsongLi

Copy link
Copy Markdown
Contributor

Please rebase master.

@plusplusjiajia
plusplusjiajia force-pushed the fix/python-filter-after-column-masking branch from 5578a9a to 7d9503c Compare October 14, 2026 02:24
@plusplusjiajia plusplusjiajia changed the title [python] Apply query filters after column masking [python] Refine post-mask query filtering for authorized reads Oct 14, 2026
@plusplusjiajia

Copy link
Copy Markdown
Member Author

Rebased onto master. Since #10539 already filters after masking, this PR now keeps that exact post-mask filter and adds: only the masks the filter reads run before it, all masks run after LIMIT with BLOB payloads deferred, type-aware comparisons on masked values, and authorization of filter columns in streaming scans. Explicit partition filters on masked columns now use master's ValueError.

@plusplusjiajia
plusplusjiajia marked this pull request as draft October 14, 2026 06:12
@plusplusjiajia
plusplusjiajia marked this pull request as ready for review October 14, 2026 07:59
Predicates on masked columns could match raw values or drop rows before
masking. Evaluate them on masked values after the row filter and before
LIMIT, authorize filter columns that are not projected, including in
streaming scans, and reject chunk shuffling that authorization filters
would resize.

Conjuncts that reference no masked column are still pushed down for pruning.
Only the masks the filter reads run before it; all masks are applied after
LIMIT, so BLOB payloads are fetched only for the rows that survive.
Comparisons and string predicates use the query-schema types, and NOT IN
with a NULL literal skips that conversion.
Split reads can apply a query predicate loosely: a nested projection drops
OR arms on unprojected columns, and merged primary-key rows compare NULL
literals as Python values. Evaluate every query predicate again before
LIMIT, keeping its columns until the final projection, and accept
authorization results without column masks.
Casting a masked comparison column to its declared type fails for values
such as 'REDACTED' on an INT column, which broke OR predicates whose other
arm already matched. Evaluate such batches row by row instead: values that
fit the declared type compare as it, the others keep their masked value.
…ters

Falling back to the Python row predicate for such batches treated
`x != NULL` as true and raised on ordering comparisons. Cast them like
SQL TRY_CAST instead and keep the evaluation path; null checks see the
same values. Only columns whose type a mask changed are cast.
The TRY_CAST cast turned valid but lossy values into NULL, such as 1.5
on an INT column, a BIGINT outside the INT range or sub-millisecond
timestamps, which master compares exactly. It also failed on ARRAY, ROW
and MAP masks, even for null checks, because Arrow cannot hash nested
values. Let Arrow compare numbers and times of one kind directly, leave
nested columns to null checks as masked, and filter on the masked values
when Arrow has no cast between the two types.
@plusplusjiajia
plusplusjiajia force-pushed the fix/python-filter-after-column-masking branch from 3d7ee31 to 9cd4a8d Compare October 14, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants