Skip to content

lockfile: key params by def_path to match how they are loaded - #11098

Closed
KR-Ravindra wants to merge 1 commit into
treeverse:mainfrom
KR-Ravindra:fix/lockfile-params-def-path
Closed

KR-Ravindra wants to merge 1 commit into
treeverse:mainfrom
KR-Ravindra:fix/lockfile-params-def-path

Conversation

@KR-Ravindra

Copy link
Copy Markdown

Problem

When a params file is written in dvc.yaml with a ./ prefix (e.g. params: [./my_params.yaml]), dvc status reports the whole file as new forever, and dvc repro re-runs the stage on every invocation, even though nothing changed:

$ dvc status
test:
	changed deps:
		new:                my_params.yaml

Reproduced on current main (3.67.2.dev) with the script from the issue.

Root cause

dvc.lock entries for deps and outs are keyed verbatim by def_path (dvc/stage/serialize.py:159, _dumpd), but the params section is keyed through Output.dumpd() (dvc/stage/serialize.py:104-105, _serialize_params_values), which rewrites in-repo paths as relpath(fs_path, stage.wdir) (dvc/output.py:843-846). The lock therefore ends up with params: {my_params.yaml: ...} while the ParamsDependency keeps def_path == "./my_params.yaml".

On load, StageLoader.fill_from_lock looks every item up by def_path (dvc/stage/loader.py:63-65): get_in(lock_data, ["params", "./my_params.yaml"]) returns None, fill_values is a no-op, hash_info.value stays None, and workspace_status reports the whole file as new.

Fix

Key the lockfile params section by param_dep.def_path, the same way deps/outs are already keyed in the same lockfile, so the serializer and the loader agree. One line in _serialize_params_values; the now-unused module constant PARAM_PATH is removed.

Lock files previously written with the normalized key are rewritten on the next dvc repro/dvc commit (which affected repos already trigger on every run today); after that the stage is reported clean.

How tested

Added three tests that fail on main and pass with the fix:

  • tests/unit/stage/test_serialize_pipeline_lock.py::test_lock_params_keeps_def_path
  • tests/unit/stage/test_loader_pipeline_file.py::test_load_stage_with_params_def_path_roundtrip (serialize to lock, load back, values filled)
  • tests/func/test_status.py::test_params_file_with_dot_slash_path (stage add + reproduce, then status == {})

Before:

FAILED tests/unit/stage/test_serialize_pipeline_lock.py::test_lock_params_keeps_def_path
E     {'params': OrderedDict({'my_params.yaml': OrderedDict({'foo': 'foo'})})} != {'params': {'./my_params.yaml': OrderedDict({'foo': 'foo'})}}
FAILED tests/unit/stage/test_loader_pipeline_file.py::test_load_stage_with_params_def_path_roundtrip
E       name: None != 'params'
E       value: None != {'foo': 1}
FAILED tests/func/test_status.py::test_params_file_with_dot_slash_path
E   AssertionError: assert {'my_params.yaml': {'foo': 1}} == {'./my_params...': {'foo': 1}}
3 failed

After:

3 passed in 0.36s

pytest tests/unit/stage tests/unit/dependency tests/func/test_status.py tests/func/params tests/func/test_lockfile.py tests/func/test_stage_load.py tests/func/test_run.py tests/func/test_stage.py passes. ruff check, ruff format --check and mypy are clean on the changed files.

End-to-end with the script from the issue: dvc.lock now contains params: {./my_params.yaml: {foo: 1}}, dvc status prints "Data and pipelines are up to date.", a second dvc repro skips the stage, and editing foo is still reported as modified.

Links


  • I have followed the Contributing to DVC checklist.

  • Documentation: no changes to the docs are needed for this fix.

This change was prepared with an AI agent operated by KR-Ravindra, who reviewed and tested it.

The params section of dvc.lock was keyed through Output.dumpd(), which
rewrites in-repo paths relative to the stage wdir, while deps and outs
in the same lockfile are keyed by def_path verbatim and StageLoader
looks everything up by def_path. A params file written as
`./my_params.yaml` in dvc.yaml was therefore stored as
`my_params.yaml`, never found on load, and reported as `new` by
`dvc status` (and re-run by `dvc repro`) forever.

Use def_path for the params key too, so the serializer and the loader
agree.

Closes treeverse#9518
@github-project-automation github-project-automation Bot moved this to Backlog in DVC Sep 9, 2026
@CLAassistant

CLAassistant commented Sep 9, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@KR-Ravindra

Copy link
Copy Markdown
Author

Closing: opened against the wrong repository; this change is for iterative/dvc.

@KR-Ravindra KR-Ravindra closed this Sep 9, 2026
@github-project-automation github-project-automation Bot moved this from Backlog to Done in DVC Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dvc status: shows that params have changed when they haven't

2 participants