Skip to content

embedded read paths: take embedded mode from backup tags, fix #1671 - #1672

Merged
Slach merged 1 commit into
masterfrom
fix_embedded_cas_disk_check_1671
Nov 7, 2026
Merged

Slach merged 1 commit into
masterfrom
fix_embedded_cas_disk_check_1671

Conversation

@Slach

@Slach Slach commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

Fixes #1671

Problem

download / restore / restore_remote / upload of an embedded backup required clickhouse.use_embedded_backup_restore: true (or --env CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE=1 on every call), even though the backup metadata already carries the embedded tag. On Antalya CAS disks this failed in ~50ms with disk "cas" ... has no local files, set clickhouse.use_embedded_backup_restore: true.

Relaxing only the disk check (the fix suggested in the issue) is not enough: the local embedded backup path (EmbeddedBackupDataPath) was also resolved from the config flag only, so with the check relaxed restore failed with read backup metadata file: ... no such file or directory and download wrote into a relative path.

Fix

create keeps deciding by the config; read paths decide by the backup embedded tag:

  • checkDisksConsistency(disks, isEmbeddedBackup) runs after the backup metadata is known in download / restore / upload (streaming restore_remote rejects embedded backups before the check, so it gets the specific error instead of the misleading CAS one)
  • EmbeddedBackupDataPath is resolved by the tag (setEmbeddedBackupDataPathByTags), restore also looks for metadata.json on the embedded backup disk
  • restore opens the remote destination for an embedded backup without embedded_backup_disk by the tag
  • embedded ON CLUSTER: with use_embedded_backup_restore_cluster set, restore, restore_remote and delete local of an embedded backup drive the other nodes too (embeddedClusterInitiator(isEmbeddedBackup)), remote tags come from the listing remoteCloudBackupPrefix already does, worker commands get --env="CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE=1" because a worker validates the flag before it reads the tags

Tests

  • unit: TestCheckDisksConsistency (CAS + embedded tag without config flag), TestEmbeddedOnClusterWorkerValidation (fan-out by tag), TestEmbeddedClusterRestoreWorkerCommand (--env in worker command, shlex round trip); make test passes
  • integration TestAntalyaCASDisk (altinity/clickhouse-server:26.6.8.20002.altinityantalya): download, restore, upload of the embedded backup with CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE=false — passes locally (Antalya is not in the CI matrix)
  • testflows embedded on cluster/restore remote on cluster by backup tags: two shards, restore_remote --env=CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE=false via API, worker gets the --env, row counts match — /clickhouse backup/embedded* 7/7 ok locally

🤖 Generated with Claude Code

download/restore/restore_remote/upload checked CAS disks and resolved
the local embedded backup path from clickhouse.use_embedded_backup_restore,
so an embedded backup created with
--env CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE=1 failed on Antalya CAS disks
unless the override was repeated for every command. Relaxing only the
disk check was not enough: EmbeddedBackupDataPath stayed empty, restore
could not find metadata.json and download wrote to a relative path.

- checkDisksConsistency gets isEmbeddedBackup, evaluated after the
  backup metadata is read; create keeps the config-only behaviour
- EmbeddedBackupDataPath is resolved by the `embedded` tag on read paths,
  restore also looks for metadata.json on the embedded backup disk
- restore opens the remote destination for an embedded backup without
  embedded_backup_disk by the tag
- embedded ON CLUSTER fan-out of restore/restore_remote/delete local is
  enabled by the tag when use_embedded_backup_restore_cluster is set,
  remote tags come from the existing remoteCloudBackupPrefix listing,
  workers get --env CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE=1 because they
  validate the flag before reading the tags

Fixes #1671

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Slach Slach added this to the 2.8.4 milestone Nov 7, 2026
@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 42384833894

Coverage increased (+0.08%) to 70.745%

Details

  • Coverage increased (+0.08%) from the base build.
  • Patch coverage: 14 uncovered changes across 4 files (59 of 73 lines covered, 80.82%).
  • 16 coverage regressions across 6 files.

Uncovered Changes

File Changed Covered %
pkg/backup/restore_cloud.go 13 5 38.46%
pkg/backup/download.go 5 3 60.0%
pkg/backup/restore.go 10 8 80.0%
pkg/backup/restore_remote.go 9 7 77.78%
Total (10 files) 73 59 80.82%

Coverage Regressions

16 previously-covered lines in 6 files lost coverage.

File Lines Losing Coverage Coverage
pkg/backup/upload.go 5 74.35%
pkg/storage/general.go 3 70.27%
pkg/clickhouse/clickhouse.go 3 81.36%
pkg/backup/delete.go 2 74.44%
pkg/backup/create_remote.go 2 66.8%
pkg/status/status.go 1 90.91%

Coverage Stats

Coverage Status
Relevant Lines: 30060
Covered Lines: 21266
Line Coverage: 70.75%
Coverage Strength: 35926.32 hits per line

💛 - Coveralls

@Slach
Slach merged commit bdc35ec into master Nov 7, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

download/restore of an embedded backup fails on CAS disks unless --env=CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE=1 is repeated

2 participants