Codex parser ignores token_usage_record; usage silently zeroes if token_count is ever dropped

[xiufengsun]

What

Codex rollout files carry a second, newer usage event that the parser does not recognise at all:

$ grep -rn "token_usage_record" src/ test/
(no matches)

extractTokenCount() (src/lib/codex-rollout-parser.js:383-394) only accepts event_msg with payload.type === "token_count" (or the nested payload.msg.type form). Everything else returns null.

The new event is top-level, not an event_msg:

{"type":"token_usage_record","payload":{"thread_id":"…","turn_id":"…","response_id":"…",
 "usage":{"input_tokens":265683,"cached_input_tokens":264576,"cache_write_input_tokens":0,
 "output_tokens":436,"reasoning_output_tokens":0,"total_tokens":266119},
 "turn_token_usage":{…},"thread_token_usage":{…}}}

Why it matters

It is converging on token_count one-for-one. Counted per file on my own machine, tc = token_count, tur = token_usage_record:

09/01  cli 0.151.0-alpha.7.2   tc=852  tur=0
09/03  cli 0.151.0             tc=100  tur=0
09/04  cli 0.151.0-alpha.7.2   tc=181  tur=103   <- first appearance
09/05  cli 0.153.3             tc=101  tur=95
09/07  cli 0.153.3             tc=451  tur=449
09/14  cli 0.154.0             tc=410  tur=404
09/17  cli 0.154.0             tc=91   tur=85

Nothing is broken today: both events are still written, and a reporter on ChatGPT desktop 26.915.31029 confirmed token_count=1 / token_usage_record=1 in a fresh session (#645). But the day a Codex build stops emitting token_count, every affected session silently reports zero tokens — no parse error, no warning, just missing usage.

The trap to avoid when fixing this

Do not simply start counting both. While both events are present they describe the same turn, so consuming each one independently double-counts. token_usage_record.usage is the per-response delta (the last_token_usage equivalent); turn_token_usage and thread_token_usage are cumulative and must not be summed.

The safe shape is a fallback: keep token_count authoritative, and read token_usage_record only for turns where no token_count was seen. Whatever the approach, it needs a totals reconciliation against real rollout data before and after — a parser change that looks right and silently doubles a total is exactly the failure this repo has shipped before.

Where this came from

Found while investigating #645. The hypothesis there (that 26.915 had dropped token_count) was disproved by the reporter's data, so this is not an active bug — it is a fuse.

[linear]

001-325