Skip to content
This repository was archived by the owner on Aug 24, 2026. It is now read-only.

Commit c5cf78e

Browse files
committed
docs: combined #994 + title-passthrough LoCoMo QA result (0.439 -> 0.611)
Adds the combined-fix measurement to the v1.2 results record. With both the FTS-revival (#994, product) and the title-passthrough (#31, harness) active, BM LoCoMo q300 QA accuracy goes 0.439 -> 0.611 (+17.2 pts), driven by multi_hop 4/63 -> 40/63 (relative-date questions that were unanswerable without the session date). Notes the single_hop 23->20 wobble honestly (judge noise, still +2 vs main) and the product follow-up (fold heading/date into matched_chunk). Signed-off-by: Drew Cain <groksrc@gmail.com>
1 parent ded4678 commit c5cf78e

1 file changed

Lines changed: 34 additions & 5 deletions

File tree

benchmarks/results/matrix-v1.2-summary.md

Lines changed: 34 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -72,11 +72,40 @@ Every category improves; largest on adversarial (+0.12 r5) and open_domain
7272
By category (correct/total): open_domain 105→112, single_hop 18→23,
7373
multi_hop 4→5, temporal 5→3 (n=19, noise).
7474

75-
The QA gain is smaller than the retrieval gain because the largest retrieval
76-
improvements land in the adversarial category (excluded from QA-meaningful
77-
scoring) and **multi_hop stays ~0.08 — bottlenecked by a separate gap, not
78-
FTS**: BM returns bullet-level matched chunks that strip document-level
79-
context (the session date lives in the title). That is the next product fix.
75+
The #994-only QA gain is smaller than the retrieval gain because the largest
76+
retrieval improvements land in the adversarial category (excluded from
77+
QA-meaningful scoring) and **multi_hop stayed ~0.08 — bottlenecked by a
78+
separate gap**: BM returns bullet-level matched chunks that strip
79+
document-level context (the session date lives in the title). That gap is
80+
addressed next.
81+
82+
## #994 + title-passthrough (combined) — corrected LoCoMo q300
83+
84+
The harness was discarding the dated `title` that `search_notes` returns
85+
(PR #31, `bm_local._row_to_hit`); surfacing it lets the answerer anchor
86+
relative-date references ("two days ago"). Combined with #994, on the same
87+
q300 subset and QA stage:
88+
89+
| QA accuracy | BM main | +FTS (#994) | +FTS +title |
90+
|---|---|---|---|
91+
| overall | 0.439 | 0.475 | **0.611** |
92+
93+
By category (correct/total), main → +FTS → +FTS+title:
94+
95+
| category | main | +FTS | +FTS+title |
96+
|---|---|---|---|
97+
| single_hop | 18/55 | 23/55 | 20/55 |
98+
| multi_hop | 4/63 | 5/63 | **40/63** |
99+
| temporal | 5/19 | 3/19 | 5/19 |
100+
| open_domain | 105/164 | 112/164 | 119/164 |
101+
102+
**+17.2 points overall (0.439 → 0.611)**, driven by multi_hop (4 → 40 — these
103+
are mostly relative-date questions that were unanswerable without the session
104+
date). single_hop dips 23 → 20 vs the FTS-only run (the title header
105+
occasionally distracts simple lookups; n=55, within LLM-judge noise, still +2
106+
vs main). The title fix is harness-side and provider-faithful — it uses data
107+
BM already returns. A product follow-up could fold the parent heading/date
108+
into `matched_chunk` so naive agents get the anchor without reading the title.
80109

81110
## Pending
82111
- supermemory-local provider validated against the real server but not yet in

0 commit comments

Comments
 (0)