fix(catalog-sql): tolerate concurrent schema migrations

[manuzhang]

Which issue does this PR close?

Fixes SQL catalog initialization failures when multiple catalog instances concurrently upgrade an existing iceberg_tables schema from V0 (without the iceberg_type column) to V1 (with the column used to distinguish tables from views). One instance can complete the upgrade while another encounters a duplicate-column or schema-lock error and fails to initialize. This PR accepts a concurrent migration conflict only after verifying that V1 is installed, while preserving genuine database failures.

What changes are included in this PR?

Migration recovery uses the workspace's existing backon retry helper with exponential backoff configured for four retries, a 100 ms initial delay, and a 60-second maximum delay. After a failed V0-to-V1 migration, the catalog immediately checks whether V1 is already installed. If V1 is not visible yet, detection retries after waits of 100, 200, 400, and 800 ms to allow a competing migration to commit. Recovery uses the schema itself without classifying database-specific error codes or messages. When V1 cannot be verified, the original DDL error is returned, and failed recovery probes are logged. Failed migrations can incur 1.5 seconds of additional retry waiting, plus the time spent checking the schema.

Schema detection sends both reads as one unprepared SQL batch through AnyPool, with a single fetch call. The empty read executes first to refresh the connection's schema before the second statement prepares its metadata. The second statement returns one NULL row, even for an empty catalog, exposing current column metadata without relying on cached prepared-statement descriptions. Both reads run in one transaction so they observe the same table definition. This lets connections that previously detected V0 observe the migrated V1 schema. Database errors are propagated directly.

This change was extracted from #2773 so the Unknown-type work can be reviewed independently.

Are these changes tested?

• SQLite regressions cover concurrent and delayed migrations, immediate duplicate-column recovery, bounded recovery after readonly failures, preservation of the original DDL error when recovery probes also fail, populated and empty V0/V1 catalogs, missing tables, and closed connections.
• A backend-neutral migration regression verifies that a reused connection detects V1 and recovers from a duplicate-column migration after detecting V0. It uses a temporary table and runs against SQLite by default or a database supplied through ICEBERG_SQL_TEST_URI.
• A SQLite regression verifies schema detection on a reused V0 connection after another connection commits the migration. It was verified to fail with a SQLx column-metadata panic when the initial empty read is removed, and to pass with the combined batch.
• cargo test -p iceberg-catalog-sql --all-targets --all-features (62 passed, none ignored)
• ICEBERG_SQL_TEST_URI=... cargo test -p iceberg-catalog-sql --lib test_schema_detection_after_migration (the same regression passed on PostgreSQL 18.6; its PostgreSQL case was verified to fail before the metadata fix)
• cargo test -p iceberg-catalog-sql --doc --all-features (1 passed)
• cargo fmt --all -- --check
• cargo clippy -p iceberg-catalog-sql --all-targets --all-features -- -D warnings

AI Disclosure

This PR was prepared with assistance from Codex.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[laskoviymishka]

The direct V1-column probe plus a re-detect-on-conflict loop is a clean way to handle the concurrent-init crash, and the barrier test is a nice way to force the interleaving.

I don't think it should merge as-is, because both halves of the core mechanism have a correctness gap the SQLite test can't catch:

• detect collapses every V1-probe failure to V0 (the is_ok() on the probe). A transient connection / permissions / proxy error on the V1 probe, followed by a successful SELECT *, returns V0 for a V1 catalog — which drops the view-row filter and refuses writes. Match on the error kind and only fall through on a genuine "no such column".
• the retry loop has no backoff, so on a networked DB all three probes can finish before the winning instance's DDL is visible, and init fails on exactly the concurrent case this targets. A short backoff (50/100/200ms) gives the commit time to land.

Two smaller things while we're here: the "resolved concurrently" log is at debug (below prod level, so a rolling deploy logs nothing) — I'd make it info and include the original error; and the test only exercises SQLite's file-lock serialization, so the genuine-error Err arm and the actual retry window are both uncovered — a unit test injecting a non-concurrent DDL failure would close that.

Once the detect error-kind check and the backoff are in, I'll take another pass.

[laskoviymishka]

Took another pass now that the detect check and the backoff are in. All four from last round landed: detect matches on error kind via is_missing_record_type_column instead of collapsing every V1-probe failure to V0, the retry loop has the 50/100/200ms backoff, the "resolved concurrently" log is info with the original error attached, and test_missing_record_type_column_errors + test_v0_schema_migration_preserves_ddl_error cover the genuine-error arm. The core mechanism reads correct now.

One thing I noticed while re-reading, and it's new this pass rather than a leftover: the backoff loop fires on every DDL failure, not just the migration race. A readonly connection or a missing ALTER privilege now sleeps the full 350ms across three probes that can never see V1 before the original error surfaces — test_v0_schema_migration_preserves_ddl_error pays exactly that. I'd classify the error before sleeping so genuine non-transient failures return immediately and only the ambiguous lock/duplicate-column cases go through the loop; specifics inline.

Rest is nits — the swallowed detect error in the loop, the exact-message == on SQLite, and the dropped tokio dev-dep. None of those block merge; tighten the retry classification and I'm happy.