load all archives' metadata with one Store.gather, reuse it for Archive

[ThomasWaldmann]

Listing the archives read every archive's metadata object with its own repository.get, i.e. one store roundtrip per archive. Then repo-list (default format) and the per-archive loops of info, tag, etc. opened an Archive per archive, which looked the archive up again (1 list + 1 load) and loaded its metadata again (1 load). So borg repo-list for 1000 archives made about 4000 store calls.

What changes

• Repository.gather_many(ids, raise_missing=True): yields the objects in the requested order, reading them in batches with one Store.gather call per batch (at most 1000 objects, or until 16 MiB). Archive metadata objects are small and usually each in a tiny pack of its own, which is what Store.gather (borgstore 0.7.0) is made for: REST (ssh://) reads all ranges with one roundtrip.
  • Objects of a pack in the in-memory pack cache are sliced from it.
  • Unknown or still buffered ids, and all ids of a batch that hit a missing or truncated pack, are read with get(), so error handling stays as it was.
  • get_many() (whole-pack reads) is unchanged.
• Archives._infos() reads the metadata objects with gather_many.
• ArchiveInfo.metadata: the parsed ArchiveItem comes along with each listed ArchiveInfo. It is an attribute of a namedtuple subclass, not a tuple field, so equality and hashing are unchanged (prune uses sets of ArchiveInfos). Infos made without it (_make(), _replace(), legacy repos, placeholders for missing/corrupt metadata objects) have metadata=None.
• Archive(manifest, archive_info): Archive accepts an ArchiveInfo. If it carries the metadata, the archive is neither looked up nor its metadata loaded again; otherwise it is looked up by id as before.
• Callers: ArchiveFormatter (repo-list, prune --list, ...), the per-archive loops of info, list, tag, find, compact, recreate and analyze, and transfer (for the source archives) pass the ArchiveInfo. mount still opens archives by id, as it opens them long after listing them.

Benchmark

borg repo-list for 1000 archives (one tiny metadata pack each), local posixfs repo, BORGSTORE_LATENCY emulating latency per store backend call (a gather is one call); median of 5 runs (0 ms) / 3 runs (50 ms):

	latency	master	this PR	store calls master → PR
repo-list	0 ms	6.59 s	0.46 s	3016 load, 1009 list → 16 load, 9 list, 1 gather
repo-list	50 ms	238.5 s	2.07 s
repo-list --short	0 ms	0.43 s	0.42 s	1016 load, 9 list → 16 load, 9 list, 1 gather
repo-list --short	50 ms	56.0 s	2.0 s

The output of repo-list, repo-list --json and info is identical to master's.

Commits

1. load all archives' metadata with Store.gather
2. reuse the listed archive metadata when opening an Archive

Testing

• Full test suite locally on macOS passes, except the known local macFUSE test_migrate_lock_alive issue.
• New tests:
  • gather_many: objects from several packs with one gather and no load (local, ssh:// / REST, store cache), batch limits, missing ids, missing packs, cached packs, packs still being stored.
  • _infos() gathers all metadata and keeps the order, placeholders for missing/corrupt metadata objects.
  • ArchiveInfo.metadata takes no part in equality/hashing; list(), get(), get_by_id() attach it.
  • repo-list: one gather for all archives' metadata, and the default format needs no further store access; Archive without metadata still loads it.

Not changed here: an archive whose metadata object is missing still makes repo-list and compact abort, see #10435.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 96.77419% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.63%. Comparing base (dca2327) to head (925c55f).
⚠️ Report is 1 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/borg/archive.py	88.23%	1 Missing and 1 partial ⚠️
src/borg/repository.py	97.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10436      +/-   ##
==========================================
+ Coverage   88.54%   88.63%   +0.09%     
==========================================
  Files         103      103              
  Lines       19148    19206      +58     
  Branches     2978     2990      +12     
==========================================
+ Hits        16955    17024      +69     
+ Misses       1529     1516      -13     
- Partials      664      666       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.