[NoopDog]

Closes #771

What changed

• A bcftools view of one VCF is now a step. Its options pick the term: VariantSubsetActivity (new, under VariantProcessingActivity) where it keeps samples or regions (-S/-s/-R/-r/-T/-t), and VariantFilterActivity where it keeps rows by a test on the calls (-f, --min-ac, -q, -i/-e, …). The VCF is processed. A view whose options give both kinds, neither, or an option no group declares gives no step (no_activity).
• Where the header's data flow names no step, its last ## line is read (producer_steps.producing_step, so every reader of the producing step shares it). This applies for want of one end, which is the case when the views write to stdout, or when a line's tool is undeclared. If that last line is a bcftools view writing to stdout or to this file, the view is the producing step.
• Every view step is checked against its parent's own header (from the last line or the one end of the flow) (fits_parent, the parent read from the VCF cache or fetched into it). The parent's header, with exactly one view's own lines added (that view being the step itself), must be the file's header, and the file must hold no sample the parent lacks. Otherwise the outcome is parent_header_differs, or parent_unreadable where the parent's header can't be read.
• Steps read from the last line name their own rules, variant_subset_by_last_line and variant_filter_by_last_line. A view that is the one end of the flow keeps variant_subset_by_header or variant_filter_by_header. This lets the edges that rest on the last-line assumptions be counted and found.
• The variant-kind caller lookup now gets the last line's view as the producing step. It still refuses a header with an undeclared tool's line.
• Supporting changes:
  • VCFHeader.last_meta_line.
  • Step.options and VcfCommand.line, both outside equality.
  • steps_of tells steps apart by their option set.
  • GZIP_SUFFIXES moved to fetchers.
• Docs:
  • CLAUDE.md.
  • An ADR-0002 amendment (decision 6), plus its activity table.
  • docs/anvil-fss-departures.md (thirteen terms of our own).
  • A new AnVIL data issue, entry 12: 2,908 subsets with no stated parent, with the ask to record the parents.

Why

ANVIL_T2T's per-population, singleton, --min-ac and region subsets of its joint callsets have no stated parent. No submitter table or anvil_activity row names them. So they, their .csi indexes and their bcftools stats reports inherited nothing, although the CHM13 PASS callset they come from is WGS, Illumina, NovaSeq 6000. Each subset's own header records the bcftools view that made it.

Assumptions I made

• The rule's five assumptions are stated in variant_subset_by_last_line's rationale:
  • bcftools appends its line. Read in htslib's bcf_hdr_add_hrec and bcftools' vcfview.c.
  • GATK and Picard re-sort a header they rewrite. Read in htsjdk's VCFWriter.writeHeader, which uses getMetaDataInSortedOrder. This is what keeps the liftovers out.
  • A tool that rewrites records but not the header is unseen, as it is by the end-of-flow reader.
  • view keeps each call as made, so VariantProcessingActivity's passes hold. The variant kind does not pass.
• What this doesn't rest on: the VCF specification gives header lines no order ("may appear in any order"). The rule leans on how the tools behave, and data issue 12 asks AnVIL/T2T to state the links instead.
• Known gap: a later view --no-version writes no line, so its file would be linked to its grandparent, which is in the same chain. The parent check doesn't catch this, and it's documented.
• Measured on the cached headers before the run:
  • 2,572 last-line steps, all passing the parent check: 2,522 exactly the parent's header plus the view line, and 50 differing only by bcftools' first-pass lines.
  • 50 one-end views, for 2,622 view steps in ANVIL_T2T.
  • 0 of 204,219 headers have two steps that differ only in their option set.
• Decisions on the issue: two terms by options (a line with both gives no step); the move into producing_step; the registry stays split until a second tool needs options; separate rule ids; the parent check.

How to verify

• Unit tests, make test, in tests/test_producer_steps.py, under the "ANVIL_T2T: give the bcftools view subsets of the joint callsets a step from their header's last line #771" section:
  • -S, -R, --min-ac and -q last-line views give their rule, by last line.
  • The PASS filter agrees with the chromosome table's step.
  • A lone view is …_by_header.
  • A lifted header, with ##reference/##source after the view, gives no step.
  • Each decline has its outcome: another output, both kinds, neither, an undeclared option, stdin, two inputs, no parent, not a VCF, ambiguous, cut short, .pvar.
  • An undeclared tool's line still lets the last line be read.
  • fits_parent accepts a view of the parent and bcftools' first pass. It refuses an added contig, INFO or other line, a changed reference, a dropped line, and a sample the parent lacks.
  • The option-drift test ties the option groups to the parser's TOOL_ARGUMENTS.
• The real run, at the PR head (done: run 20261027_210224 at the final code, 5d8e7e4, against main's 20261027_095223, same inputs. It is identical to the earlier branch run 20261027_195130, so the review fixes changed no output. The reports are regenerated from it; sample-conflicts.tsv is reordered only, Make the sample-conflicts TSV and a sample source's cited activity independent of line order #767): make classify, make reconcile, make reconcile-report, make rules-report, then a diff against main's run 20261027_095223. Check that:
  • The ANVIL_T2T CHM13 subsets, their .csi indexes and their stats reports have WGS / ILLUMINA / Illumina NovaSeq 6000. The ANVIL_T2T open-slot count falls by about 21K: assay, platform and instrument about 5,952 each, variant_origin about 3,456. Run 20261027_210224: exactly 5,952 / 5,952 / 5,952 / 3,456 (21,312), all ANVIL_T2T, all not_classified → inherited; nothing opened, no answered value changed.
  • No liftover VCF (*.liftover.vcf.gz) has a generated_by. 0.
  • The reconcile report's lineage block shows no new conflict or misfit. The PASS callsets' header step agrees with the chromosome table's. No conflicts; the misfits are unchanged; the 48 PASS callsets are now content_read+repository_metadata.
  • The rules report lists the four view rules with their fire counts: about 1,732 variant_subset_by_last_line, 840 variant_filter_by_last_line, 48 variant_filter_by_header and 2 variant_subset_by_header. Exactly those counts.
  • Any variant_kind/variant_origin move on a file whose producing step changed is explained (expected: reason text only). variant_kind is unchanged everywhere; variant_origin moves only by the inheritance above.
• Definition of done from ANVIL_T2T: give the bcftools view subsets of the joint callsets a step from their header's last line #771:
  • The unit tests: last-line view steps, the liftover giving none, and each decline with its outcome. Covered above.
  • make classify and make reconcile: the subsets, indexes and stats have WGS, Illumina and NovaSeq 6000, and the ANVIL_T2T gap falls by about 21K. These are the run steps above.
  • No liftover VCF has a generated_by. This is a run step above.
  • No new lineage conflict or misfit. This is a run step above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS

🐦 Built with cc-claude-tools

[Copilot]

🟡 Changes recommended

Unresolved view-step defects can inherit pre-liftover references, create self-parent cycles that halt reconciliation, and reject valid explicit-stdout views.

3 open issues

• Validate parent headers for all view steps · New
• Prevent self-parent resolution in view parent lookup · New
• Treat explicit stdout output as a valid view destination · New

What changed in this PR

This PR adds header-derived lineage for ANVIL_T2T’s bcftools subsets and filters, enabling reconciliation to inherit metadata from their parent callsets.

Changes:

• Recognizes bcftools view activities by options, with a last-header-line fallback and parent-header validation.
• Adds VariantSubsetActivity, distinct provenance rules, and shared parser/cache support.
• Expands regression tests and documents the inference assumptions and missing AnVIL parent links.

File	Description
tests/​test_variant_origins.py	Checks origin inheritance through subsets.
tests/​test_variant_kinds.py	Tests caller lookup with last-line views.
tests/​test_reconcile_report.py	Updates expected outcome columns.
tests/​test_producer_steps.py	Tests view rules, parent checks, and declines.
tests/​test_activities.py	Verifies subset inheritance behavior.
src/​meta_disco/​validators/​header_extractors.py	Records the last metadata line.
src/​meta_disco/​validators/​command_lines.py	Preserves option names and source lines.
src/​meta_disco/​schema/​classification.yaml	Defines the subset activity.
src/​meta_disco/​schema/​classification_model.py	Updates the generated activity enum.
src/​meta_disco/​rules/​activities.yaml	Declares subset inheritance semantics.
src/​meta_disco/​producer_steps.py	Implements view lineage and parent validation.
src/​meta_disco/​pipeline.py	Imports shared gzip suffixes.
src/​meta_disco/​file_types.py	Supplies the VCF parent-cache configuration.
src/​meta_disco/​fetchers.py	Hosts shared gzip suffixes.
src/​meta_disco/​code_rules.py	Registers four view provenance rules.
src/​meta_disco/​callers.py	Preserves unread-tool rejection during caller lookup.
src/​meta_disco/​activities.py	Adds filter and subset constants.
docs/​anvil-fss-departures.md	Lists the additional activity type.
docs/​anvil-data-issues.md	Documents missing subset-parent links.
docs/​adr/​0002-derivation-graph.md	Records the lineage decision and assumptions.
CLAUDE.md	Updates the documented inference behavior.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Unresolved validation and parsing defects can admit invalid lineage edges from truncated headers or bundled flags, while inconsistent compression handling makes lineage results depend on parent-header cache state.

No open issues 🎉

3 previously missed issues

In code that hasn't changed since last review

Reject truncated lone-view headers with unknown sample columns

src/​meta_disco/​producer_steps.py:356

A header truncated after a lone view -S ... parent.vcf.gz can still be selected by end_of_flow, bypassing the completeness guard in last_line_view. Here, sample_columns=None becomes an empty set, allowing a lineage edge without checking the child's samples or remaining header. Reject unknown child sample columns and add a regression test for a truncated lone-view header.

Honor file format when checking parent compression

src/​meta_disco/​producer_steps.py:396

Unlike the main VCF pipeline, this compression check ignores file_format, so a gzip-compressed parent.vcf declared as vcf.gz cannot be read on a parent-cache miss. ParentHeaders retains the failure for the run, making its children's lineage depend on whether the parent header was already cached. Enable the fetcher's magic-byte detection for both compressed and plain parents, or retain file_format in the parent index and share the pipeline's compression check.

Handle bundled short options before applying the activity allowlist

src/​meta_disco/​validators/​command_lines.py:368

Bundled short flags are not all recorded here. For example, view -pG parent.vcf.gz is parsed with options == ('-p',) because G is treated as an attached value; view_rule consequently accepts it as a filter. The equivalent view -p -G parent.vcf.gz correctly gives no step because -G is in VIEW_NO_STEP_OPTIONS. Expand no-argument short-option clusters, or conservatively reject them, before applying the activity allowlist, and cover the bundled spelling in a regression test.

3 resolved since last review

• Prevent self-parent resolution in view parent lookup
• Validate parent headers for all view steps
• Treat explicit stdout output as a valid view destination

_{Review effort: Balanced}

[NoopDog]

Copilot round 2's three previously-missed findings are fixed in 822fa7f:

• A cut-short child: fits_parent now requires the child's header to have been read to a VCF #CHROM line.
• A parent's compression: read from its name or its file_format, as the pipeline reads it.
• Option clusters: a short-option cluster led by a flag that takes no value (-pG) is recorded whole, so view_rule declines it.

Each has a regression test. None changes a cached header's outcome.

[Copilot]

🟡 Changes recommended

Active issues allow non-string file formats to crash processing and cause valid compressed-output filters to be rejected, losing classification rows or inherited metadata.

1 open issue

• Guard non-string file_format before calling lower() · New

1 previously missed issue

In code that hasn't changed since last review

Preserve compression suffix in self-parent comparison

src/​meta_disco/​producer_steps.py:486

This rejects a valid filter such as view -f PASS -Oz -o sample.vcf.gz sample.vcf even when both files exist as distinct records. data_name() removes the compression suffix, but resolve_names() matches the full basename and would resolve sample.vcf to the separate parent, so the guard unnecessarily drops the step and its inherited metadata. Preserve compression suffixes in this self-parent comparison.

_{Review effort: Balanced}

[NoopDog]

The previously missed finding from Copilot round 3 is fixed in 96fbf9e: the self-parent guard now compares the input's full base name, as parent lookup resolves it, so a recompression of an uncompressed twin (view -Oz -o x.vcf.gz x.vcf) keeps its step. A regression test covers it.

[Copilot]

🔵 Needs a closer look

The heuristic changes inherited biological metadata across thousands of records, but required full-corpus validation remains unverified for liftover exclusions, inheritance changes, and new lineage conflicts.

No open issues 🎉

1 resolved since last review

• Guard non-string file_format before calling lower()

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues are recorded for the header-derived lineage changes.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Human assessment is needed because validation can select an inherited view command over a newly added one, accepting ambiguous provenance and incorrectly recording a VariantFilterActivity.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Human assessment is needed because an unpublished finding identifies a contradiction between the ADR’s parent-header invariant and the code and tests.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The ADR reverses the header-validation invariant and must state that the child’s header contains the parent’s lines plus one new view command.

1 open issue

• Correct ADR to describe added child header lines · New

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

There are no active review issues blocking approval.

No open issues 🎉

1 resolved since last review

• Correct ADR to describe added child header lines

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues remain for the header-derived subset and filter provenance changes.

No open issues 🎉

_{Review effort: Balanced}