fair_unified: a sort fails on its first request when consumers that cannot spill hold the task's share #7197

Under fair_unified, the pool limits each consumer that can spill to (task_share - unspillable) / num_spill (#7096). When consumers that cannot spill, such as the build side of a shuffled hash join or a sort-merge join's buffered rows, hold the task's whole share, that limit is zero. A sort in the same task is then refused on its first request.

DataFusion 55.2's ExternalSorter does not spill in that case. It returns "Not enough memory to continue external sort" when it cannot reserve a batch while in_mem_batches is empty, when the retry after a spill fails, and when the try_resize in sort_batch_stream fails. So the query fails, although Spark would still grant the task memory: Spark lets a task hold more than off_heap_size / running_tasks while other tasks have not asked for memory yet.

The same paths fail when one batch is larger than the sort's share, for example with wide rows, many cores, little off-heap memory and several consumers that can spill in one task.

a_consumer_that_can_spill_is_refused_once_the_others_hold_the_tasks_share in fair_pool.rs pins the shape. The contributor guide, the tuning guide and the 1.2.0 migration note list the workarounds: a larger spark.memory.offHeap.size, a smaller spark.comet.batchSize, or spark.comet.exec.memoryPool=greedy_unified.

Options include letting a consumer that holds nothing take its first batch from Spark, or falling back to Spark's grant when the limit is below one batch. Either changes behavior that the #7096 benchmarks did not measure, so it needs its own measurement.

Found in #7096 (#7096 (comment)).