Repository navigation
lockfile: key params by def_path to match how they are loaded - #11098
Closed
KR-Ravindra wants to merge 1 commit into
Closed
KR-Ravindra wants to merge 1 commit into
KR-Ravindra wants to merge 1 commit into
Conversation
The params section of dvc.lock was keyed through Output.dumpd(), which rewrites in-repo paths relative to the stage wdir, while deps and outs in the same lockfile are keyed by def_path verbatim and StageLoader looks everything up by def_path. A params file written as `./my_params.yaml` in dvc.yaml was therefore stored as `my_params.yaml`, never found on load, and reported as `new` by `dvc status` (and re-run by `dvc repro`) forever. Use def_path for the params key too, so the serializer and the loader agree. Closes treeverse#9518
Author
|
Closing: opened against the wrong repository; this change is for iterative/dvc. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a params file is written in
dvc.yamlwith a./prefix (e.g.params: [./my_params.yaml]),dvc statusreports the whole file asnewforever, anddvc reprore-runs the stage on every invocation, even though nothing changed:Reproduced on current
main(3.67.2.dev) with the script from the issue.Root cause
dvc.lockentries fordepsandoutsare keyed verbatim bydef_path(dvc/stage/serialize.py:159,_dumpd), but theparamssection is keyed throughOutput.dumpd()(dvc/stage/serialize.py:104-105,_serialize_params_values), which rewrites in-repo paths asrelpath(fs_path, stage.wdir)(dvc/output.py:843-846). The lock therefore ends up withparams: {my_params.yaml: ...}while theParamsDependencykeepsdef_path == "./my_params.yaml".On load,
StageLoader.fill_from_locklooks every item up bydef_path(dvc/stage/loader.py:63-65):get_in(lock_data, ["params", "./my_params.yaml"])returnsNone,fill_valuesis a no-op,hash_info.valuestaysNone, andworkspace_statusreports the whole file asnew.Fix
Key the lockfile
paramssection byparam_dep.def_path, the same waydeps/outsare already keyed in the same lockfile, so the serializer and the loader agree. One line in_serialize_params_values; the now-unused module constantPARAM_PATHis removed.Lock files previously written with the normalized key are rewritten on the next
dvc repro/dvc commit(which affected repos already trigger on every run today); after that the stage is reported clean.How tested
Added three tests that fail on
mainand pass with the fix:tests/unit/stage/test_serialize_pipeline_lock.py::test_lock_params_keeps_def_pathtests/unit/stage/test_loader_pipeline_file.py::test_load_stage_with_params_def_path_roundtrip(serialize to lock, load back, values filled)tests/func/test_status.py::test_params_file_with_dot_slash_path(stage add+reproduce, thenstatus == {})Before:
After:
pytest tests/unit/stage tests/unit/dependency tests/func/test_status.py tests/func/params tests/func/test_lockfile.py tests/func/test_stage_load.py tests/func/test_run.py tests/func/test_stage.pypasses.ruff check,ruff format --checkandmypyare clean on the changed files.End-to-end with the script from the issue:
dvc.locknow containsparams: {./my_params.yaml: {foo: 1}},dvc statusprints "Data and pipelines are up to date.", a seconddvc reproskips the stage, and editingfoois still reported asmodified.Links
I have followed the Contributing to DVC checklist.
Documentation: no changes to the docs are needed for this fix.
This change was prepared with an AI agent operated by KR-Ravindra, who reviewed and tested it.