You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The only performance numbers for native Iceberg writes are the ad hoc measurements in the #5361 description (2.9x unpartitioned, 5.5x partitioned fanout, from a harness in a personal fork). There is no benchmark in the repository, so a regression in the writer, in the shuffle feeding it, or in the JVM metrics rebuild would not be noticed, and the partitioned number was measured with write.distribution-mode=none rather than the default plan shape.
Describe the potential solution
Add a Spark benchmark (the org.apache.spark.sql.benchmark framework used by CometReadBenchmark and the shuffle benchmark) that measures INSERT INTO ... SELECT into an Iceberg table for:
a copy-on-write DELETE (rewrite of affected files)
Each case runs Spark, Comet scan only, and Comet scan plus native write, so the writer's contribution is isolated the way the #5361 table did. Wire it into the benchmark targets in the Makefile and document how to run it. Follow-on: run it on the EC2 benchmark runner and keep results with the other benchmark files.
Keep the Spark benchmark framework's warmup and minimum-time floors in mind when sizing the data (see the shuffle benchmark work in #5388 for how that dominated wall clock).
Additional context
Part of the native Iceberg writes epic, #5649. Related: #5361, #3595 (fused sort plus write, which this benchmark would measure).
Activity
0lai0 commentedon Sep 17, 2026
take