fix(ai-proxy): include cache tokens in Anthropic prompt usage and read them from message_delta

[AlinsRan]

Description

Anthropic Messages requests with prompt caching report wrong token usage. Users see a cache hit ratio above 100% (cache reads / prompt tokens), and token-based rate limits and metrics count fewer tokens than the request actually used.

How the two protocols define input tokens

	OpenAI Chat Completions	Anthropic Messages
Input field	prompt_tokens	input_tokens
Includes cache tokens?	Yes. Cache reads/writes are parts of the prompt (prompt_tokens_details.*)	No. Only tokens after the last cache breakpoint
Cache read	prompt_tokens_details.cached_tokens	cache_read_input_tokens
Cache write	prompt_tokens_details.cache_write_tokens	cache_creation_input_tokens
Total input	prompt_tokens	input_tokens + cache_read_input_tokens + cache_creation_input_tokens
Streaming usage	Last chunk	message_start, then cumulative values in message_delta

APISIX uses the OpenAI meaning for $llm_prompt_tokens on every protocol:

• prompt_tokens = total input, including cache reads and writes
• total_tokens = prompt_tokens + completion_tokens

Bugs fixed

#	Path	Bug	Fix
1	Native Anthropic	prompt_tokens was input_tokens only, so it did not include cache tokens	prompt_tokens = input + cache_read + cache_creation
2	Native Anthropic, streaming	Cache tokens were read only from message_start. Some providers report them only in message_delta, so they were logged as 0	Accumulate raw usage across message_start and message_delta, then compute
3	Anthropic client → OpenAI-compatible upstream	The converter subtracted only cached_tokens from prompt_tokens, so cache writes were counted twice	input_tokens = max(0, prompt - cached - cache_write)
4	Anthropic client → OpenAI-compatible upstream, streaming	The client got the right cache-write value, but $llm_cache_creation_input_tokens was 0 unless the upstream used prompt_tokens_details.cache_creation_input_tokens	Shared cache-token helpers in openai-chat (below)

Example

Request with 100,000 input tokens: 90,000 read from cache, 9,000 written to cache, 1,000 uncached.

	Before	After
Native Anthropic $llm_prompt_tokens	1,000	100,000
Native Anthropic cache hit ratio	9000%	90%
Converted path, input_tokens sent to the client	10,000 (cache writes counted twice, sum = 109,000)	1,000 (sum = 100,000)

Implementation

• anthropic-messages.lua: compute prompt_tokens / total_tokens as above; in streaming, merge raw usage from message_start and message_delta before computing. This matches LiteLLM (AnthropicConfig.calculate_usage, streaming _handle_usage).
• openai-chat.lua: new shared helpers, used by parse_sse_event, extract_usage and the converter:
  • cache_read_tokens(): prompt_tokens_details.cached_tokens, then DeepSeek prompt_cache_hit_tokens.
  • cache_creation_tokens(): top-level cache_creation_input_tokens, then prompt_tokens_details.cache_creation_tokens / cache_write_tokens / cache_creation_input_tokens (LiteLLM order).
• anthropic-messages-to-openai-chat.lua: use the helpers on the non-streaming path and both streaming paths.

Corrected values

No configuration, schema or API changes. Only requests that use prompt caching on the paths above report different values.

Consumer	Change
$llm_prompt_tokens, $llm_total_tokens, llm_summary	Include cache reads and writes (Anthropic)
$llm_cache_read_input_tokens, $llm_cache_creation_input_tokens	No longer 0 in the cases of bugs 2 and 4
Prometheus apisix_llm_prompt_tokens(_dist)	Include cache tokens (Anthropic)
ai-rate-limiting with total_tokens / prompt_tokens	Cache tokens count toward the quota, same as OpenAI
ai-rate-limiting with expression	On the converted non-streaming path, input_tokens no longer includes cache writes (fixture: 20 → 15)
Native OpenAI-compatible upstreams	prompt_tokens_details.cache_write_tokens is now recognized

ctx.llm_raw_usage stays the raw upstream usage, except on the converted non-streaming path, where it already held the converted Anthropic usage before this PR (moderation plugins use it to build deny responses in the client protocol). This PR keeps that behavior.

Tests

File	Tests
t/plugin/ai-proxy-anthropic.t	TEST 8, 50, 51: updated expectations. TEST 65-69: cache tokens only in message_delta; converted path $llm_prompt_tokens equals upstream prompt_tokens. TEST 70: converter cache-write field shapes. TEST 71-74: converted streaming logs match client values for each field shape and DeepSeek. TEST 75: unit test of extract_usage / parse_sse_event
t/plugin/ai-rate-limiting.t	TEST 43/44: Anthropic cache tokens count toward total_tokens
t/plugin/ai-rate-limiting-expression.t	TEST 29/30: input_tokens + cache_read_input_tokens + cache_creation_input_tokens equals upstream prompt_tokens

All new and changed tests fail without the fix.

Docs for ai-proxy, ai-proxy-multi, ai-rate-limiting and prometheus (en/zh) are updated.

Which issue(s) this PR fixes:

Fixes #

Checklist

• I have explained the need for this PR and the problem it solves
• I have explained the changes or the new features added to this PR
• I have added tests corresponding to this change
• I have updated the documentation to reflect this change
• I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

This is a bug fix. It does not change configuration, schemas or APIs.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open findings

• The function name/comment says “accumulate”, but the implementation is actually a field-wise… · New
• cache_creation_tokens will raise an error if it’s ever called with a non-table usage (e.g.,… · New
• The Chinese docs line mixes singular/plural (“prompt token”) and English “token” wording in a way… · New

What changed in this PR

This PR aligns Anthropic Messages token accounting with OpenAI semantics by including cache read/write tokens in prompt/total usage and ensuring streaming usage captures cache tokens even when reported only in message_delta.

Changes:

• Adjust Anthropic usage calculation so prompt_tokens = input_tokens + cache_read_input_tokens + cache_creation_input_tokens and total_tokens = prompt_tokens + completion_tokens.
• Update Anthropic streaming parsing to merge usage fields across message_start and message_delta.
• Extend the Anthropic↔OpenAI converter and test fixtures/docs to prevent double-counting cache-creation tokens and reflect the new semantics.

File	Description
apisix/​plugins/​ai-protocols/​anthropic-messages.lua	Computes prompt/total tokens including cache tokens and merges streaming usage fields across events.
apisix/​plugins/​ai-protocols/​converters/​anthropic-messages-to-openai-chat.lua	Deduces cache-creation tokens from multiple upstream shapes and subtracts them when building Anthropic input_tokens.
t/​plugin/​ai-proxy-anthropic.t	Updates/adds tests for cache token handling in logging and conversion, incl. cache tokens reported in message_delta.
t/​plugin/​ai-rate-limiting.t	Adds coverage for rate limiting with total_tokens including cache tokens for native Anthropic.
t/​fixtures/​anthropic/​messages-streaming-with-cache-in-delta.sse	Adds an SSE fixture where cache tokens appear only in message_delta.
docs/​en/​latest/​plugins/​*.md	Documents that prompt tokens include cache read/write tokens and provides Anthropic formula.
docs/​zh/​latest/​plugins/​*.md	Same documentation updates in Chinese.

🧠 Review effort: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Non-streaming converted responses alter llm_raw_usage despite the PR’s compatibility claim.

4 open findings

• Preserve raw usage before converting non-streaming responses · New
• cache_creation_tokens will raise an error if it’s ever called with a non-table usage (e.g.,…
• The function name/comment says “accumulate”, but the implementation is actually a field-wise…
• The Chinese docs line mixes singular/plural (“prompt token”) and English “token” wording in a way…

🧠 Review effort: Balanced

[membphis]

A non-blocking accounting observation for the author to decide whether to address in this PR or in a follow-up:

The converter now recognizes usage.cache_creation_input_tokens, prompt_tokens_details.cache_creation_tokens, and prompt_tokens_details.cache_write_tokens, but the converted streaming path still collects normalized usage through openai-chat.parse_sse_event, which only reads prompt_tokens_details.cache_creation_input_tokens.

For example, an Anthropic client with an OpenAI-compatible upstream can receive this final streaming usage:

{
  "prompt_tokens": 100,
  "completion_tokens": 10,
  "cache_creation_input_tokens": 30,
  "prompt_tokens_details": { "cached_tokens": 20 }
}

A focused probe of the reviewed Lua modules shows that the converted client response reports input_tokens=50, cache_read_input_tokens=20, and cache_creation_input_tokens=30, while the streaming normalized usage reports cache_creation_input_tokens=0. That zero flows into $llm_cache_creation_input_tokens, the access log, and llm_summary. The non-streaming conversion reports 30 correctly. The same mismatch occurs with the other two newly supported cache-write field aliases. The corrected prompt and total token counts on this path remain correct.

Relevant code: converter and streaming usage extraction.

A possible fix is to share the cache-write field extraction rules between conversion and usage normalization, preserving raw usage, and add streaming E2E assertions for both the response and access-log cache counts for all three field shapes.

[membphis]

LGTM

[AlinsRan]

The converter now recognizes usage.cache_creation_input_tokens, prompt_tokens_details.cache_creation_tokens, and prompt_tokens_details.cache_write_tokens, but the converted streaming path still collects normalized usage through openai-chat.parse_sse_event, which only reads prompt_tokens_details.cache_creation_input_tokens.

Thanks @membphis, good catch. Fixed in 03ccd33.