Repository navigation
[python] Refine post-mask query filtering for authorized reads - #10523
Open
plusplusjiajia wants to merge 5 commits into
Open
plusplusjiajia wants to merge 5 commits into
plusplusjiajia wants to merge 5 commits into
Conversation
plusplusjiajia
force-pushed
the
fix/python-filter-after-column-masking
branch
4 times, most recently
from
October 13, 2026 08:41
2e6e8e8 to
5578a9a
Compare
plusplusjiajia
marked this pull request as ready for review
October 13, 2026 08:41
Contributor
|
Please rebase master. |
plusplusjiajia
force-pushed
the
fix/python-filter-after-column-masking
branch
from
October 14, 2026 02:24
5578a9a to
7d9503c
Compare
Member
Author
|
Rebased onto master. Since #10539 already filters after masking, this PR now keeps that exact post-mask filter and adds: only the masks the filter reads run before it, all masks run after LIMIT with BLOB payloads deferred, type-aware comparisons on masked values, and authorization of filter columns in streaming scans. Explicit partition filters on masked columns now use master's |
plusplusjiajia
marked this pull request as draft
October 14, 2026 06:12
plusplusjiajia
marked this pull request as ready for review
October 14, 2026 07:59
Predicates on masked columns could match raw values or drop rows before masking. Evaluate them on masked values after the row filter and before LIMIT, authorize filter columns that are not projected, including in streaming scans, and reject chunk shuffling that authorization filters would resize. Conjuncts that reference no masked column are still pushed down for pruning. Only the masks the filter reads run before it; all masks are applied after LIMIT, so BLOB payloads are fetched only for the rows that survive. Comparisons and string predicates use the query-schema types, and NOT IN with a NULL literal skips that conversion.
Split reads can apply a query predicate loosely: a nested projection drops OR arms on unprojected columns, and merged primary-key rows compare NULL literals as Python values. Evaluate every query predicate again before LIMIT, keeping its columns until the final projection, and accept authorization results without column masks.
Casting a masked comparison column to its declared type fails for values such as 'REDACTED' on an INT column, which broke OR predicates whose other arm already matched. Evaluate such batches row by row instead: values that fit the declared type compare as it, the others keep their masked value.
…ters Falling back to the Python row predicate for such batches treated `x != NULL` as true and raised on ordering comparisons. Cast them like SQL TRY_CAST instead and keep the evaluation path; null checks see the same values. Only columns whose type a mask changed are cast.
The TRY_CAST cast turned valid but lossy values into NULL, such as 1.5 on an INT column, a BIGINT outside the INT range or sub-millisecond timestamps, which master compares exactly. It also failed on ARRAY, ROW and MAP masks, even for null checks, because Arrow cannot hash nested values. Let Arrow compare numbers and times of one kind directly, leave nested columns to null checks as masked, and filter on the masked values when Arrow has no cast between the two types.
plusplusjiajia
force-pushed
the
fix/python-filter-after-column-masking
branch
from
October 14, 2026 10:02
3d7ee31 to
9cd4a8d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Master (#10539) evaluates the full query predicate after column masking, but it applies every mask before the filter and resolves BLOB payloads before LIMIT. This PR keeps exact post-mask filtering and fixes the remaining performance and semantic gaps:
TRY_CAST. Numbers, times of one kind and nested columns are filtered as masked, so no precision is lost, and NOT IN with a NULL literal skips the conversion. NULL MAP values stay intact and_row_kindcolumns, including-Drows, pass through the filter.Tests
Adds regression coverage for append, primary-key and data-evolution tables, including partitions, projections, compound and OR predicates, LIMIT, streaming row kinds, chunk shuffling, NULL MAP values, type-changing masks, standalone equality and string predicates, selection-sensitive authorization for partition filters and hidden streaming filter columns, NOT IN with a NULL literal, masked values that do not fit the declared type (OR arms, NULL literals and null checks), numeric, timestamp and nested masks that change the column type, nested-field aliases, pushdown of unmasked conjuncts, OR predicates on unprojected columns under a nested projection, NULL comparisons after primary-key merges, and row-only authorization with a user filter. BLOB tests count payload reads: the post-mask LIMIT cases read 2 payloads instead of 10, including when an output-only mask reads a projected or unprojected BLOB.
lance,vortex,duckdb), and 6 files needray.git diff --checkpassed.