[RyoKusnadi]

Problem

On a fresh clickhouse-server, the first create / create_remote over several tables can fail with code: 32, File .../shadow/increment.txt is empty. You must fill it manually. It happens once per server lifetime, but it is a failed backup (and a flaky CI run).

How it happens: a new server has no shadow/increment.txt. create freezes tables in parallel (clickhouse.max_connections workers). Two FREEZE queries race to create the file: one creates it empty, the other locks it first and finds it empty (the CounterInFile race described in #1651).

Cause

createBackupLocal in pkg/backup/create.go starts all table workers at once, so the first FREEZEs overlap.

Fix

Before starting the workers, serialFreezeTableIndex checks <default disk path>/shadow/increment.txt. If it is missing (or can't be checked, for example clickhouse-backup runs on another host), the first table that will really run FREEZE is backed up serially, then the rest run in parallel as before. I don't write the file myself, since it belongs to clickhouse-server. The per-table body moved into a local closure so the serial and parallel paths share it.

Scope

• Not changed: retry on other transient FREEZE errors, a FREEZE-only concurrency limit, pre-warming increment.txt in testflows (open questions in the issue).
• use_embedded_backup_restore doesn't use FREEZE and is unaffected.
• clean --all already keeps increment.txt.

Testing

TestSerialFreezeTableIndex (temp dirs, no ClickHouse): fresh dir, unreachable path, existing file, doBackupData=false, skipped/schema-only tables, empty list.

• Before: undefined: serialFreezeTableIndex (test written first).
• After: go test ./pkg/backup/ -run TestSerialFreezeTableIndex passes; go test ./pkg/backup/ passes (376 tests); go vet ./pkg/backup/ and gofmt -l are clean.
• Not run: integration/testflows tests (need a live ClickHouse), so the race itself is not reproduced here.

Risk

Only the first backup on a server without increment.txt is a bit slower (one serial FREEZE). When the file can't be checked remotely, every create has one serial FREEZE first.

Fixes #1651

[RyoKusnadi]

@Slach friendly ping on this one when you have a moment. Happy to change anything.

[RyoKusnadi]

Closing this for now, happy to reopen if it's useful.

[Slach]

please reopen, need time for full review