Skip to content

Drop column bounds when a row group with values has no min/max - #4313

Open
Remenby31 wants to merge 1 commit into
apache:mainfrom
Remenby31:fix-bounds-row-group-without-min-max
Open

Remenby31 wants to merge 1 commit into
apache:mainfrom
Remenby31:fix-bounds-row-group-without-min-max

Conversation

@Remenby31

Copy link
Copy Markdown

Closes #4312

Rationale for this change

data_file_statistics_from_parquet_metadata built a column's bounds from the row groups that report a min/max and silently skipped a row group whose statistics have no min/max. PyArrow writes such statistics when a value is larger than 4 KB. The stored bounds then exclude that row group's values, and the metrics evaluator prunes the file from scans that should match it.

With this change, a row group that holds non-null values but reports no min/max drops the column's bounds for the file. The column keeps its null and value counts, because the row group still reports them. A row group whose values are all null still has no effect on the bounds.

Are these changes tested?

Yes. test_bounds_absent_when_a_row_group_with_values_has_no_min_max writes two row groups per column. One column has a 5,000-character value in its second row group, and the other column has an all-null first row group. The test fails without the change: the first column keeps the bound b'a'. tests/io/test_pyarrow_stats.py and the unit tests of tests/io/test_pyarrow.py pass, and prek passes on the changed files.

Are there any user-facing changes?

Data files written from now on have no bounds for such a column, so scans no longer skip their matching rows. Files already written keep their bounds.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Data file bounds too narrow when a row group has no min/max for a column (string values over 4 KB), so scans skip matching files

1 participant