You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Failure-injection tests for native Iceberg writes #5646
The native Iceberg write path has documented failure semantics (mid-task failure retries with attempt-unique file names, partial results are never committed, commit failures reuse SparkWrite.abort), and two known gaps (#5618 task-attempt cleanup, #5277 orphans on commit failure). What it does not have is tests that inject failures and assert those semantics hold. CometIcebergWriteActionSuite has one commit-time conflict test and one aborted-job test; neither fails inside the native writer, and neither checks what files are left behind.
Describe the potential solution
Add a failure-injection section to the Iceberg write tests covering, for the native path:
A task that fails inside iceberg-rust after at least one data file has been finalized (for example an object store that starts rejecting writes after N puts, using the in-memory or a wrapped file:// store): the retry succeeds, the table commits exactly once, and no file from the failed attempt is referenced by any manifest.
Speculative execution: two attempts of the same task both finish; exactly one set of files is committed.
Executor loss mid-write on a multi-task write: the stage retries and the final row count is exact.
Each test should assert both the visible outcome (rows, snapshot count) and the storage state (files under the data location versus files referenced by manifests), since the orphan gaps are invisible to readers.
Additional context
Part of the native Iceberg writes epic, #5649. Related: #5618, #5277, #5361 (introduced the failure-handling section in iceberg-writes.md).
Activity
sam-1112 commentedon Sep 16, 2026
take
sam-1112 commentedon Sep 22, 2026
I’d like to split #5646 into four PRs, each focused on a distinct failure boundary:
TaskCommit(test: cover native Iceberg write handoff cleanup #6117 ).