Skip to content

Add Hi-C, Strand-seq and Iso-Seq assay terms, declared from HPRC's submitter tables (#399) - #764

Merged
NoopDog merged 7 commits into
mainfrom
noopdog/399-assay-terms
Oct 27, 2026
Merged

NoopDog merged 7 commits into
mainfrom
noopdog/399-assay-terms

Conversation

@NoopDog

@NoopDog NoopDog commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #399 (its assay terms only; Bionano, seqspec, chains and GRCm39 stay open there)

What changed

  • Three assay terms in assay_type_enum (src/meta_disco/schema/classification.yaml, generated model regenerated):
    • Hi-C: EFO:0007693. No parent among our values; its EFO parents, the chromosome conformation capture terms, aren't values here.
    • Strand-seq: EFO:0008952. No parent among our values.
    • Iso-Seq, titled "Isoform sequencing": OBI:0002575, under RNA-seq. EFO has no long-read RNA term, so this uses OBI's; the term's description records why (see Assumptions).
  • Translation rows (rules/value_map.yaml):
    • assay_type.hi_c now declares assay_type: Hi-C beside its existing modality.
    • assay_type.isoseq is authored for the first time: Iso-Seq, transcriptomic.
    • New assay_type.strandseq: Strand-seq, genomic.
  • Slot map: ANVIL_HPRC's sample.strandseq column now gives assay_type from its name, as sample.hic already does.
    • ANVIL_HPRC's evidence is re-imported as generation 20261027T015110Z. The only difference from 20261012T192136Z is 1,728 new strandseq assay lines.
  • Consistency rule assay_for_genomic admits Strand-seq.
  • Lookup tables: the manifest survey's name tokens give hic and strandseq their terms, and HPRC validation maps isoseq to Iso-Seq.
  • CLAUDE.md: the assay_type_enum note.
  • Tests: pin each new term's id and parent, and check the new assays against the consistency rules. Tests that used Hi-C as their example of a value with no term now use CenSat.
  • Reports: regenerated from a full run at this branch's head.

Why

AnVIL's HPRC submitter tables already state these assays: HPRC_R2's hic and kinnex tables, and the hic and strandseq columns of ANVIL_HPRC's sample table. Their files stayed unclassified for assay type only because our vocabulary had no term to translate them into. #399 recorded these as vocabulary decisions owed; this change makes the three assay decisions.

Assumptions I made

  • Iso-Seq's id comes from OBI, not EFO. I compared how others name this data:

    • EFO has no long-read RNA term. A request filed 2026-10-06 ([NTR] Request for new long-read RNA, single-cell long-read and DNA methylation assay terms EBISPOT/efo#2842) asks for one, citing OBI:0002575, and is still open. EFO's nearest term is FL-cDNA (EFO:0003755), any sequencing of full-length cDNA. It is broader (SRA uses it for Sanger-era cDNA libraries too) and sits under EFO's RNA assay rather than RNA-seq. The no-context review found it; Dave kept OBI's term (2026-10-27), and the term's description records why FL-cDNA was not taken.
    • OBI's isoform sequencing is defined as sequencing a cDNA library enriched for full-length reads. That fits PacBio's Iso-Seq and Kinnex and names the method rather than a vendor's kit. Direct RNA sequencing has its own OBI term, so it stays separate.
    • ENCODE names the same data long read RNA-seq, under its own placeholder id NTR:0000445.

    Dave chose OBI's id over a term of our own with no id. The value is Iso-Seq and the title "Isoform sequencing", the same pattern as WGS / "Whole genome sequencing". If EFO imports the term, its meaning can move to EFO's id.

  • Hi-C and Strand-seq sit at the top level, as Methylation array and Genotyping array do: none of their EFO parents is one of our values.

  • Strand-seq's modality is genomic: it measures the DNA sequence and its structure, from template strands phased into haplotypes and inversions. MODAL's genomic has no child after Drop data_modality terms that name a method, not a biological signal (genomic.assembly and its siblings) #756. The reason argues from what is measured, not from the molecule sequenced, which the table already refuses as modality evidence for SRA's GENOMIC.

  • Iso-Seq's row also declares transcriptomic. This agrees with inference's existing answer on all 230 files.

  • Translation rows only. No inference rule claims these assays. The consistency linter reads inference output, so a Hi-C ↔ modality rule would never fire; that belongs with Run the consistency linter over the reconciled run, not only inference output #597.

  • docs/manifest-survey.md is not regenerated. It still says Hi-C has no term, but regenerating it would also pull in unrelated changes from a newer manifest download (a duos_dataset_registration table, a file_path column).

  • The evidence re-import writes into the shared data/ folder. Until this merges, a run from main in another clone would list strandseq in the review queue, because main has no row for it. The PRIMED change (Read a PLINK .psam's sample IDs as its fileset's samples: PRIMED's 11,414 genotype files #751) did the same.

How to verify

  1. Checks: run make test-all lint-all format-check validate-metadata. Everything should pass.
  2. The terms:
    • In src/meta_disco/schema/classification.yaml, find Hi-C (EFO:0007693), Strand-seq (EFO:0008952) and Iso-Seq (OBI:0002575, is_a: RNA-seq). Iso-Seq's description should say why the id is OBI's.
    • tests/test_assay_type_vocabulary.py pins all three.
  3. The rows: make review-queue. In docs/review-queue-report.md, isoseq should no longer be queued, and the rows assay_type.hi_c, assay_type.strandseq and assay_type.isoseq should show what they declare.
  4. The evidence: ls data/source_evidence/anvil/anvil15/ANVIL_HPRC/ should list 20261027T015110Z. Compared with 20261012T192136Z, its sample.ndjson should add exactly 1,728 lines, all assay_type strandseq, and the other three files should be identical.
  5. Output (run 20261026_194027, classified at 6edf390, reconciled at e6bf23f; since then only reasons, descriptions and tests changed, so the inference output stands): run make classify, then compare its reconciled/reconcile_report.json values with main's run 20261026_120218 (main bdf43ce). You should see:
    • ANVIL_HPRC: assay_type Hi-C +768 and Strand-seq +1,728; data_modality genomic +1,728.
    • AnVIL_HPRC_R2: assay_type Hi-C +3,002 and Iso-Seq +230.
    • No other value moves, the conflict rate is unchanged, and the inference files at the run root are unchanged. The committed reports come from that run.

#399's definition of done, for the three tokens this covers:

  • Each token is in exactly one recorded state, mapped to a term this work adds: steps 2 and 3. hic and Hi-C → Hi-C, isoseq → Iso-Seq, strandseq → Strand-seq.
  • Each term this work adds is a schema change reviewed on its own: step 2. The terms are in the schema diff, apart from the rows.
  • The file counts are re-measured against the current catalog: step 5. Hi-C 3,770 as Vocabulary decisions: the closed set of declared terms our schema has no word for #399 measured, Iso-Seq 230, and Strand-seq 1,728 (not 1,344: the column also reaches the HPRC_PLUS Strand-seq files).
  • A declined token makes no claim and stays in the review queue: not exercised. This change declines no token.

Review findings declined (Dave, 2026-10-27), recorded in a PR comment:

  • The schema change is not a separate commit. Dave reviewed the three terms before any rule was written, and they are in their own two files in the diff.
  • The HPRC evidence fixture is pinned to one measured generation, so it gains no strandseq line.
  • Tests do not restate what bundled rows declare, or the slot map's spans, per the project's rule on tests.
  • docs/manifest-survey.md is not regenerated (see Assumptions).

🤖 Generated with Claude Code

https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
🐦 Built with cc-claude-tools

NoopDog and others added 4 commits October 26, 2026 18:57
…PRC's tables (#399)

- assay_type_enum: Hi-C (EFO:0007693) and Strand-seq (EFO:0008952), with no parent among the
  values; Iso-Seq, OBI's isoform sequencing (OBI:0002575) under RNA-seq, since EFO has no
  long-read RNA term (EBISPOT/efo#2842 open). The term's description records why.
- value_map: assay_type.hi_c declares Hi-C beside its modality; assay_type.isoseq authored
  (Iso-Seq, transcriptomic); assay_type.strandseq new (Strand-seq, genomic).
- slot map: ANVIL_HPRC's sample.strandseq gives assay_type from its column name; evidence
  re-imported as generation 20261027T015110Z (+1,728 strandseq lines, nothing else changed).
- consistency rule assay_for_genomic admits Strand-seq; manifest survey's name tokens give
  hic and strandseq their terms; CLAUDE.md's assay_type_enum note.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
Against main's run 20261026_120218 (bdf43ce): assay_type +768 Hi-C and +1,728 Strand-seq in
ANVIL_HPRC, +3,002 Hi-C and +230 Iso-Seq in AnVIL_HPRC_R2; data_modality +1,728 genomic in
ANVIL_HPRC (Strand-seq), and the 230 Iso-Seq files' transcriptomic now credited to the
submitter table beside inference. Inference output is unchanged. The review queue loses
`isoseq`; the unprocessable report's ANVIL_T2T examples are another five of the same 23 (#732).
docs/sample-conflicts.tsv is left as main's (same lines, worker order, #750).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
…ree docstring

- The Iso-Seq row's reason cites the term's description for why OBI, instead of restating it;
  the Strand-seq row's and the consistency rule's descriptions drop the method restated from
  the term's; the Iso-Seq term's description ends without restating the EFO request.
- test_consistency: the WGS case joins the parametrized assay/modality test.
- test_value_map's docstring names SEEDED_IN_BUNDLED_TABLE rather than counting it.
- Rules report and review queue regenerated for the reworded reasons.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
Co-authored-by: cc-claude-tools <cc-claude-tools@clevercanary.com>
…eason is the information measured, no stale "Hi-C has no term"

- validation_maps: HPRC's isoseq / Iso-Seq map to the enum term `Iso-Seq`, not `ISO-seq`;
  the comment no longer says Hi-C and Iso-Seq have no term.
- value_map: the Strand-seq reason argues from what is measured (the DNA sequence and its
  structure), not the molecule sequenced, which the table refuses as modality evidence.
- Tests whose comments or example raw values called Hi-C a vocabulary gap use CenSat.
- assay_type_enum's description scopes "nearest EFO ancestor" to name Iso-Seq's OBI-based
  parent; the Iso-Seq description scopes "the one term" to EFO and OBI.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
Co-authored-by: cc-claude-tools <cc-claude-tools@clevercanary.com>
Copilot AI balanced review requested due to automatic review settings October 27, 2026 02:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Final human review should confirm the ontology choices and externally stored evidence generation’s provenance.

No open issues 🎉

What changed in this PR

Adds Hi-C, Strand-seq, and Iso-Seq to Meta-disco’s assay vocabulary, addressing the assay-term portion of #399 through submitter-evidence translations rather than new inference rules.

Changes:

  • Defines the three assay terms and ontology identifiers, with Iso-Seq under RNA-seq.
  • Updates evidence routing, translations, consistency checks, and lookup tables.
  • Adjusts vocabulary tests and refreshes reports showing 5,728 newly classified assay values.
File Description
tests/​test_value_map.py Updates vocabulary-gap fixtures and remaining queue expectations.
tests/​test_source_evidence.py Uses CenSat as the raw vocabulary-gap example.
tests/​test_rule_engine.py Replaces the Hi-C state-claim example with CenSat.
tests/​test_hprc_validation.py Expects canonical Iso-Seq spelling.
tests/​test_consistency.py Checks Strand-seq and Iso-Seq modality compatibility.
tests/​test_claim_record.py Removes Hi-C from a vocabulary-gap comment.
tests/​test_assay_type_vocabulary.py Pins new terms’ ontology identifiers and parents.
src/​meta_disco/​validation_maps.py Maps HPRC Iso-Seq spellings to the canonical term.
src/​meta_disco/​sources/​anvil_slot_map.yaml Routes the Strand-seq column to assay evidence.
src/​meta_disco/​schema/​classification.yaml Defines the new assay terms and their rationale.
src/​meta_disco/​schema/​classification_model.py Updates the generated assay enum and documentation.
src/​meta_disco/​rules/​value_map.yaml Declares assay and modality translations.
src/​meta_disco/​rules/​consistency_rules.yaml Allows Strand-seq for genomic data.
src/​meta_disco/​manifest_survey.py Recognizes Hi-C and Strand-seq name tokens.
schema/​tests/​test_output_validation.py Updates the unrestricted raw-evidence example.
docs/​unprocessable-report.md Refreshes the run reference and failure examples.
docs/​rules-report.md Reports new declarations and application counts.
docs/​review-queue.html Refreshes the queue and authored translation display.
docs/​review-queue-report.md Removes translated Iso-Seq values from the queue.
docs/​reconcile-report.md Records classification gains and evidence counts.
docs/​consistency-report.md Refreshes the inference consistency run reference.
CLAUDE.md Documents the expanded assay vocabulary.

Review effort: Balanced

Same counts as run 20261026_185725 against main's 20261026_120218: assay_type Hi-C +3,770,
Strand-seq +1,728, Iso-Seq +230; data_modality genomic +1,728. What moves is the run named,
the Strand-seq reason's wording, and tie order (#732, #750). docs/sample-conflicts.tsv is left
as main's.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
Co-authored-by: cc-claude-tools <cc-claude-tools@clevercanary.com>
Copilot AI balanced review requested due to automatic review settings October 27, 2026 02:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Static review supports the consistency of these additive schema, mapping and report changes; no blocking issues remain, though project checks were not executed.

No open issues 🎉

Review effort: Balanced

NoopDog and others added 2 commits October 27, 2026 00:43
…t EFO parents, accurate reasons

- Iso-Seq keeps OBI's isoform sequencing (Dave, 2026-10-27); its description, the enum's and
  CLAUDE.md now name EFO's FL-cDNA (EFO:0003755) as the nearest EFO term and why it was not
  taken (broader, SRA's Sanger-era cDNA strategy too, under RNA assay not RNA-seq), instead of
  saying EFO has none.
- Hi-C's and Strand-seq's descriptions list their direct EFO parents.
- Reasons: hi_c's `hic` is a table- or column-name span; isoseq quotes the kinnex table's
  design_description; strandseq says what is read (template strands) and so what is measured.
- validation_maps' docstring drops "its own spelling where our vocabulary has no term";
  assay_for_genomic says Strand-seq cannot reach it yet (the linter reads inference output).
- Synthetic test rows declaring Hi-C genomic use WGS instead.
- Rules report and review queue regenerated for the reworded reasons.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
Co-authored-by: cc-claude-tools <cc-claude-tools@clevercanary.com>
…and translation-table hash are the PR's (#399)

Classified at 6edf390 (inference unchanged since: only reasons, descriptions and tests moved),
reconciled at e6bf23f. Every count is unchanged; only the provenance lines move.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
Co-authored-by: cc-claude-tools <cc-claude-tools@clevercanary.com>
Copilot AI balanced review requested due to automatic review settings October 27, 2026 07:50
@NoopDog

NoopDog commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator Author

Findings from the no-context review with no review thread, and what became of each:

Fixed (e6bf23f):

  • Iso-Seq's description, the enum's and CLAUDE.md named no EFO term. They now name FL-cDNA (EFO:0003755) as EFO's nearest and say why it was not taken. Dave kept OBI:0002575.
  • Hi-C's and Strand-seq's descriptions now list their direct EFO parents.
  • The hi_c, isoseq and strandseq reasons were made accurate.
  • validation_maps' module docstring was fixed.
  • assay_for_genomic now notes that Strand-seq can't reach it yet.
  • Two synthetic test rows no longer declare Hi-C as genomic.

Declined (Dave, 2026-10-27):

  • The schema terms aren't a separate commit. They were reviewed with Dave before any rule was written, and they sit in their own two files.
  • The HPRC fixture gets no strandseq line. The fixture is pinned to the measured 20260920 generation, which has 41 keys and 43 strings.
  • Mutations to row declarations and to the slot-map span survive. Catching them would mean restating bundled rows in tests, which the project's rule on tests forbids.
  • docs/manifest-survey.md is left stale. Regenerating it would pull in unrelated manifest drift.

🐦 Built with cc-claude-tools

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Static review supports the consistency of the schema, mappings, tests, and reported deltas, with no blocking issues identified; automated checks were not run.

No open issues 🎉

Review effort: Balanced

@NoopDog
NoopDog merged commit 51e7e40 into main Oct 27, 2026
2 checks passed
@NoopDog
NoopDog deleted the noopdog/399-assay-terms branch October 27, 2026 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants