Repository navigation
lockfile: key params by def_path to match how they are loaded - #11097
KR-Ravindra wants to merge 1 commit into
Conversation
|
Closing: opened against the wrong repository; this change is for iterative/dvc. |
|
Reopening: iterative/dvc now redirects to this repository, so this is the correct target. Issue #9518 lives here as well. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #11097 +/- ##
==========================================
+ Coverage 90.68% 90.98% +0.30%
==========================================
Files 504 505 +1
Lines 39795 41158 +1363
Branches 3141 3263 +122
==========================================
+ Hits 36087 37448 +1361
- Misses 3042 3071 +29
+ Partials 666 639 -27 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
The red
Happy to send the one-line fix as a separate PR if you want it from me; otherwise I will rebase once main is green. The change itself is unaffected: |
The params section of dvc.lock was keyed through Output.dumpd(), which rewrites in-repo paths relative to the stage wdir, while deps and outs in the same lockfile are keyed by def_path verbatim and StageLoader looks everything up by def_path. A params file written as `./my_params.yaml` in dvc.yaml was therefore stored as `my_params.yaml`, never found on load, and reported as `new` by `dvc status` (and re-run by `dvc repro`) forever. Use def_path for the params key too, so the serializer and the loader agree. Closes treeverse#9518
5f73986 to
2fb1d49
Compare
|
Re-pushed on the same content ( For what it is worth on the red checks that remain: the three So nothing here is waiting on me. Happy to send a separate PR for the Drafted with AI assistance and checked against the code before posting. |
|
The red and this PR touches The cause looks like shtab's annotation rather than a real defect: script = shtab.complete(parser, shell=shell, preamble=get_preamble()[shell])Happy to send that as its own one-line PR if it would help — it would take every open PR's No changes needed here as far as I can tell. |
Problem
When a params file is written in
dvc.yamlwith a./prefix (params: [./my_params.yaml]),dvc statusreports the whole file asnewforever anddvc reprore-runs the stage every time, even though nothing changed:Root cause
dvc.lockentries fordepsandoutsare keyed verbatim bydef_path, but theparamssection is keyed throughOutput.dumpd(), which rewrites in-repo paths asrelpath(fs_path, stage.wdir). So the lock holdsparams: {my_params.yaml: ...}while theParamsDependencykeepsdef_path == "./my_params.yaml".On load,
StageLoader.fill_from_locklooks items up bydef_path, the lookup returnsNone,fill_valuesis a no-op,hash_info.valuestaysNone, andworkspace_statusreports the file asnew.Fix
Key the lockfile
paramssection byparam_dep.def_path, the same waydepsandoutsalready are in the same file, so the serializer and the loader agree. One line in_serialize_params_values; the now-unusedPARAM_PATHconstant is removed.Lock files already written with the normalized key are rewritten on the next
dvc repro/dvc commit— which affected repos run on every invocation today anyway — and the stage is clean after that.Three tests cover it: the lock keeps
def_path, a serialize-then-load round trip fills the values, and a functionalstage add+reproduceleavesdvc statusempty.Links
I have followed the Contributing to DVC checklist.
Documentation: no changes to the docs are needed for this fix.
Yes, AI-assisted; reviewed by me.