Skip to content

Add inspect regression test for bounds after type promotion - #4314

Open
avison9 wants to merge 1 commit into
apache:mainfrom
avison9:fix/inspect-bounds-after-type-promotion
Open

avison9 wants to merge 1 commit into
apache:mainfrom
avison9:fix/inspect-bounds-after-type-promotion

Conversation

@avison9

@avison9 avison9 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Closes #3744

Rationale for this change

#3744 reports that after a spec-allowed promotion (int → long, float → double), every metadata
inspection API raises struct.error on tables that still have files written before the promotion. Bounds
are stored without a type tag, and older files keep the 4-byte encoding while the current schema type is
8 bytes.

This is already fixed on main by #3293 (released in 0.12.0): from_bytes(LongType/DoubleType) now falls
back to 4-byte decoding, and every bounds decode in inspect.py goes through it. The issue was reported
against 0.11.1, which predates that fix.

#3293 only added coverage for the metrics evaluators, though; the inspect path was fixed as a side effect
with no test pinning it. This PR adds that coverage so the crash can't silently come back, e.g. if
inspect moves to a different decode path (#4297).

tbl.append(pa.table({"qty": pa.array([1, 2], pa.int32())}))
with tbl.update_schema() as u:
    u.update_column("qty", field_type=LongType())

tbl.inspect.files()
# 0.11.1:  struct.error: unpack requires a buffer of 8 bytes
# main:    readable_metrics.qty -> lower_bound=1, upper_bound=2

Are these changes tested?

Test-only change. test_inspect_bounds_after_type_promotion writes with int/float, promotes to
long/double, and asserts correct decoded bounds from entries, files, data_files, all_files,
all_data_files, and the manifests() partition summaries.

Verified that it fails with the original struct.error when #3293's change to conversions.py is reverted,
and passes on main. Full unit suite: 4524 → 4525 passed, no new failures; make lint clean.

Are there any user-facing changes?

No.

Bounds written before an int->long or float->double promotion keep their
4-byte encoding. apache#3293 fixed decoding them, but only covered the metrics
evaluators. Cover the inspect metadata tables too, so the crash reported
in apache#3744 can't regress.

Closes apache#3744

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Metadata inspection APIs fail with struct.error after int→long / float→double type promotion

1 participant