Skip to content

fix(glue): Exclude historical schemas from Glue columns - #3390

Open
VABVAT wants to merge 3 commits into
apache:mainfrom
VABVAT:fx/glue_duplicate_columns
Open

VABVAT wants to merge 3 commits into
apache:mainfrom
VABVAT:fx/glue_duplicate_columns

Conversation

@VABVAT

@VABVAT VABVAT commented Oct 10, 2026

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

This PR updates Glue schema conversion so AWS Glue StorageDescriptor.columns is populated only from the current Iceberg schema.

Previously, GlueSchemaBuilder::from_iceberg converted the current schema and then also converted historical schemas from table metadata. Those historical fields were written to Glue with iceberg.field.current=false, but Glue does not use that parameter to filter visible columns, which could cause duplicate columns to appear in the Glue Console after schema evolution.

Historical schemas are still preserved in Iceberg table metadata. This change only prevents them from being projected into Glue's visible column list.

Are these changes tested?

Yes.

Added a regression test covering schema evolution with historical schemas, and verified the Glue schema output contains only current schema fields.

Ran locally:

cargo test -p iceberg-catalog-glue

Copilot AI balanced review requested due to automatic review settings October 10, 2026 06:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

iceberg-catalog-glue pushes all historical schema fields to Glue, causing duplicate columns in Glue UI

2 participants