create: parallel FREEZE on fresh server fails with 'shadow/increment.txt is empty' (CounterInFile first-create race) #1651

Symptom

On a fresh clickhouse-server (no shadow/increment.txt yet), the first create / create_remote covering several tables can fail:

ALTER TABLE `repl_db_acm_new`.`rmt_9` FREEZE WITH NAME 'c69ce70d38e04758a1039d88d0d09e9c';
ALTER TABLE `repl_db_acm_new`.`rmt_8` FREEZE WITH NAME '5efe4fa45bd349fda4caa95990181f56';
ERR pkg/backup/create.go:558 > b.AddTableToLocalBackup error: b.ch.FreezeTable: can't freeze table:
  code: 32, message: File /var/lib/clickhouse/shadow/increment.txt is empty. You must fill it manually with appropriate value.

Seen in CI: https://github.com/Altinity/clickhouse-backup/actions/runs/40683044506/job/122799508345 (Testflows, CH 23.8, replicated database / acm restore dropped database). This was the first FREEZE on a freshly started testcontainers cluster.

Root cause (clickhouse-server)

FREEZE increments a counter in <path>/shadow/increment.txt via CounterInFile::add (src/Common/CounterInFile.h). The source code comment says the race exists: "when the first file is created, the race condition is possible, so it's better to create the file in advance":

1. Thread A sees !fs::exists(path) and creates the file with open(O_CREAT). The file is empty at this point.
2. Thread B sees that the file exists, opens it and gets flock(LOCK_EX) before A.
3. B reads the empty file and throws File ... is empty.

The in-process mutex does not help, because each FREEZE constructs its own Increment instance.

clickhouse-backup runs FREEZE concurrently (createBackupWorkingGroup.SetLimit(max(clickhouse.max_connections, 1)) in pkg/backup/create.go). So on a fresh server, any first backup with more than one table can hit this. clean already keeps increment.txt (shadowIncrementFile in pkg/backup/delete.go), so this can happen only once per server lifetime. It is still a failed backup for real users, and in CI it causes flaky failures.

Proposal

In create, before the parallel table loop: if shadow/increment.txt does not exist under the default disk path, run the FREEZE for the first table serially, then start the parallel workers for the rest.

• We should not write the file ourselves. clickhouse-backup can run on a different host/container than clickhouse-server, as in testflows, and the file ownership belongs to clickhouse-server.
• If the file existence can't be checked (no local access to the CH data path), we could fall back to always doing the first FREEZE serially. The cost is one table's FREEZE latency.
• use_embedded_backup_restore doesn't use FREEZE, so it is not affected.

Open questions

• Other parallel-FREEZE races in clickhouse-server are still unfixed. Should this be generalized, e.g. a retry on known transient FREEZE errors, or a separate concurrency limit for FREEZE only?
• Should testflows/integration environments also pre-warm increment.txt at cluster start, to remove the CI flake independently of the product fix?