Part of #6240. Related: #5122, #5649.
What is the problem?
On Spark 3.5+, an Iceberg merge-on-read DELETE, UPDATE or MERGE writes through Spark's WriteDelta command. Comet's split Iceberg write path (IcebergWrite + IcebergCommit) does not recognise it today, so those writes either stay on Spark's stock plan or are caught by checks written for copy-on-write and append writes. There is no explicit, tested statement of what happens to a position-delta write.
The native position-delta writer in #6240 needs this first. Before native code can route data rows and position deletes, Comet must recognise WriteDelta, extract what a delta write needs (the operation column and the data, row-id and metadata projections), and keep running Iceberg's JVM DeltaWriter for it. That makes later native work a swap of the writer behind a known dispatch point rather than a planner change.
Proposed solution
The first item of the incremental split in #6240, with behaviour unchanged for users:
- Recognise Iceberg
WriteDelta on Spark 3.5+; Spark 3.4 keeps the stock plan.
- Carry only what the JVM delta path needs: the operation and Spark's projection state, using Spark's
RowDeltaUtils operation constants instead of copies of the numeric ids.
- Run position-delta rows through Iceberg's JVM
DeltaWriter. The native Iceberg writer stays out of the delta path (CometIcebergWriteExec is never used for position-delta rows).
- Keep row-level write summaries, writer reporting, AQE behaviour and failure cleanup the same as the stock Iceberg path.
- Fall back cleanly when the version-specific logical-plan extraction cannot be done, with a stated reason.
Scope
In scope: planning, extraction and JVM dispatch for position-delta writes, with tests.
Out of scope (later items in #6240): a native delta operator or protobuf, a native position-delete writer or file layout contract, loading previous deletes, and a native feature flag. No Rust changes or dependencies.
Acceptance criteria
DELETE, UPDATE and MERGE results match the stock Iceberg JVM path.
- A position-delta write plans without
CometIcebergWriteExec.
- Copy-on-write and merge-on-read
MERGE plans match the stock path with AQE on and off.
- Row-level snapshot summaries match where Spark exposes them.
- Writer reporting classifies the intercepted JVM
WriteDelta write.
- Task commit location extraction covers position-delta data and delete files.
- Unsupported shapes fall back with a reason.
Part of #6240. Related: #5122, #5649.
What is the problem?
On Spark 3.5+, an Iceberg merge-on-read
DELETE,UPDATEorMERGEwrites through Spark'sWriteDeltacommand. Comet's split Iceberg write path (IcebergWrite+IcebergCommit) does not recognise it today, so those writes either stay on Spark's stock plan or are caught by checks written for copy-on-write and append writes. There is no explicit, tested statement of what happens to a position-delta write.The native position-delta writer in #6240 needs this first. Before native code can route data rows and position deletes, Comet must recognise
WriteDelta, extract what a delta write needs (the operation column and the data, row-id and metadata projections), and keep running Iceberg's JVMDeltaWriterfor it. That makes later native work a swap of the writer behind a known dispatch point rather than a planner change.Proposed solution
The first item of the incremental split in #6240, with behaviour unchanged for users:
WriteDeltaon Spark 3.5+; Spark 3.4 keeps the stock plan.RowDeltaUtilsoperation constants instead of copies of the numeric ids.DeltaWriter. The native Iceberg writer stays out of the delta path (CometIcebergWriteExecis never used for position-delta rows).Scope
In scope: planning, extraction and JVM dispatch for position-delta writes, with tests.
Out of scope (later items in #6240): a native delta operator or protobuf, a native position-delete writer or file layout contract, loading previous deletes, and a native feature flag. No Rust changes or dependencies.
Acceptance criteria
DELETE,UPDATEandMERGEresults match the stock Iceberg JVM path.CometIcebergWriteExec.MERGEplans match the stock path with AQE on and off.WriteDeltawrite.