Skip to content

lockfile: key params by def_path to match how they are loaded - #11097

Open
KR-Ravindra wants to merge 1 commit into
treeverse:mainfrom
KR-Ravindra:fix/lockfile-params-def-path
Open

KR-Ravindra wants to merge 1 commit into
treeverse:mainfrom
KR-Ravindra:fix/lockfile-params-def-path

Conversation

@KR-Ravindra

@KR-Ravindra KR-Ravindra commented Sep 9, 2026 •

Copy link
Copy Markdown

Problem

When a params file is written in dvc.yaml with a ./ prefix (params: [./my_params.yaml]), dvc status reports the whole file as new forever and dvc repro re-runs the stage every time, even though nothing changed:

$ dvc status
test:
	changed deps:
		new:                my_params.yaml

Root cause

dvc.lock entries for deps and outs are keyed verbatim by def_path, but the params section is keyed through Output.dumpd(), which rewrites in-repo paths as relpath(fs_path, stage.wdir). So the lock holds params: {my_params.yaml: ...} while the ParamsDependency keeps def_path == "./my_params.yaml".

On load, StageLoader.fill_from_lock looks items up by def_path, the lookup returns None, fill_values is a no-op, hash_info.value stays None, and workspace_status reports the file as new.

Fix

Key the lockfile params section by param_dep.def_path, the same way deps and outs already are in the same file, so the serializer and the loader agree. One line in _serialize_params_values; the now-unused PARAM_PATH constant is removed.

Lock files already written with the normalized key are rewritten on the next dvc repro/dvc commit — which affected repos run on every invocation today anyway — and the stage is clean after that.

Three tests cover it: the lock keeps def_path, a serialize-then-load round trip fills the values, and a functional stage add + reproduce leaves dvc status empty.

Links


  • I have followed the Contributing to DVC checklist.

  • Documentation: no changes to the docs are needed for this fix.

Yes, AI-assisted; reviewed by me.

@github-project-automation github-project-automation Bot moved this to Backlog in DVC Sep 9, 2026
@CLAassistant

CLAassistant commented Sep 9, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@KR-Ravindra

Copy link
Copy Markdown
Author

Closing: opened against the wrong repository; this change is for iterative/dvc.

@KR-Ravindra KR-Ravindra closed this Sep 9, 2026
@github-project-automation github-project-automation Bot moved this from Backlog to Done in DVC Sep 9, 2026
@KR-Ravindra KR-Ravindra reopened this Sep 9, 2026
@KR-Ravindra

Copy link
Copy Markdown
Author

Reopening: iterative/dvc now redirects to this repository, so this is the correct target. Issue #9518 lives here as well.

@github-project-automation github-project-automation Bot moved this from Done to Backlog in DVC Sep 9, 2026
@codecov

codecov Bot commented Sep 9, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.98%. Comparing base (2431ec6) to head (2fb1d49).
⚠️ Report is 213 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #11097      +/-   ##
==========================================
+ Coverage   90.68%   90.98%   +0.30%     
==========================================
  Files         504      505       +1     
  Lines       39795    41158    +1363     
  Branches     3141     3263     +122     
==========================================
+ Hits        36087    37448    +1361     
- Misses       3042     3071      +29     
+ Partials      666      639      -27     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@KR-Ravindra
KR-Ravindra marked this pull request as ready for review September 9, 2026 10:33
@KR-Ravindra

Copy link
Copy Markdown
Author

The red lint jobs here are not from this change. They fail in dvc/commands/completion.py, which this PR does not touch:

dvc/commands/completion.py:20:63: error: Argument "preamble" to "complete" has
incompatible type "dict[str, str]"; expected "str"  [arg-type]

main fails the same way on its own scheduled runs (every nightly since at least 2026-09-15 is red at 56e5982), so this is a shtab signature drift rather than anything in the diff. The tests matrix failures on this PR are the same aggregate job reporting that lint failed.

Happy to send the one-line fix as a separate PR if you want it from me; otherwise I will rebase once main is green. The change itself is unaffected: dvc/stage/serialize.py plus the three test files.

The params section of dvc.lock was keyed through Output.dumpd(), which
rewrites in-repo paths relative to the stage wdir, while deps and outs
in the same lockfile are keyed by def_path verbatim and StageLoader
looks everything up by def_path. A params file written as
`./my_params.yaml` in dvc.yaml was therefore stored as
`my_params.yaml`, never found on load, and reported as `new` by
`dvc status` (and re-run by `dvc repro`) forever.

Use def_path for the params key too, so the serializer and the loader
agree.

Closes treeverse#9518
@KR-Ravindra
KR-Ravindra force-pushed the fix/lockfile-params-def-path branch from 5f73986 to 2fb1d49 Compare September 23, 2026 22:24
@KR-Ravindra

Copy link
Copy Markdown
Author

Re-pushed on the same content (2fb1d49) to get a fresh CI run: the previous one was from 2026-09-09 and its Windows failure was tests/func/test_repo_index.py::test_ignored_dir_unignored_pattern, which this change does not touch — it only alters how params are keyed in the lockfile.

For what it is worth on the red checks that remain: the three lint jobs fail on main as well, on today's run (35806491443), with a single mypy error that predates this PR:

dvc/commands/completion.py:20:63: error: Argument "preamble" to "complete" has incompatible type "dict[str, str]"; expected "str"  [arg-type]

So nothing here is waiting on me. Happy to send a separate PR for the shtab signature if that is useful.

Drafted with AI assistance and checked against the code before posting.

@KR-Ravindra

Copy link
Copy Markdown
Author

The red lint here is not from this branch. The only mypy finding is

dvc/commands/completion.py:20:63: error: Argument "preamble" to "complete" has
incompatible type "dict[str, str]"; expected "str"  [arg-type]

and this PR touches dvc/stage/serialize.py plus three test files, none of which are near it. lint (ubuntu-latest) is failing the same way on #11100, #11105 and #11106, so it is main-wide rather than per-PR.

The cause looks like shtab's annotation rather than a real defect: get_preamble() returns dict[str, str] keyed by shell, shtab.complete accepts that at runtime and selects by shell, but its signature types preamble as str, and shtab<2,>=1.3.4 is unpinned so a new release changed what mypy sees. Passing the already-known shell through keeps the behaviour identical and is type-correct:

script = shtab.complete(parser, shell=shell, preamble=get_preamble()[shell])

Happy to send that as its own one-line PR if it would help — it would take every open PR's lint green. Say the word and I will open it; I did not want to push an unrelated fix into this branch.

No changes needed here as far as I can tell.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dvc status: shows that params have changed when they haven't

2 participants