[maralbahari]

Part of the RFC #500 (one typed path for rehydrating and persisting history).

Summary

Stored history now holds only the items the client sees. For gateway tools (web search, MCP, code interpreter) the gateway talks to the model with function_call / function_call_output pairs; those pairs used to be stored and leaked into GET /v1/responses/{id}/input_items, which OpenAI never returns. They now live only in the inference input, and one function lowers every public tool item to a function call when a request goes upstream. Every gateway call has a public item: each handler must return its own, so a custom web search executor produces a web_search_call too.

The turn's storage path is also reduced to one entry point each way. persist.rs has a single persist that builds the turn's metadata and items and owns session checkpoints, and the storage handlers only write. The rehydrate loaders share one step for turning history into the model input.

Pending client calls have one API: CallLedger scans an item sequence once and answers every question about its calls, on the shared ClientCallId and ClientCallKind types. Call IDs follow OpenAI, as the new call-ids cassettes record it: a repeated or reused call_id is accepted and an empty one is refused, and the llm-d split path uses the same rules as the gateway instead of its own. Every item a client sends (Responses input, Conversations items) passes one admission check, InOutItem::admit_all. Overlapping multi-agent turns in a conversation are refused like OpenAI refuses them: a multi-agent tree is the latest turn's snapshot, so a multi-agent turn stored after another turn would replace that turn's tree.

Before	After
Gateway tool pairs (function_call + function_call_output) were stored and returned by input_items	History stores only public items (web_search_call, mcp_call, code_interpreter_call, ...); the pairs exist only in the inference input
Lowering to function calls was spread across storage, rehydration and the upstream projection	ResponsesInput::normalized_model_input is the one lowering point, using From conversions in types/io/web_search.rs and types/io/code_interpreter.rs
History was written in two places: per round (gateway results, session rounds) and at persist, reconciled by is_recorded_as_history, is_already_in_history, record_round_history and mark_outputs_recorded	History is written once, at persist: the request's own items, then the response's public output
Five persist entry points (persist_if_needed, persist_response, persist_prepared_response, persist_turn, persist_prepared_turn)	One persist(ctx, TurnOutput, tool_search_metadata, exec_ctx); persist_turn and commit are thin public wrappers
ResponseHandler::persist held session checkpoints, transient-parent promotion and output selection	The handlers only write; that logic is in persist.rs
Three rehydrate loaders each repeated "history + request items, check unanswered calls, set model input, apply settings"	The loaders only load; one set_model_input(ctx, history, window) does the rest (a session passes canonical_session_history as its window)
Pending calls were read through pending_calls, first_unanswered_sync_call, resolved_prefix_len and rehydration's reject_unanswered_sync_call, each scanning the items again	One CallLedger::scan(items) with pending(), awaits_output(), require_answered() and resolved_prefix_len()
pending_calls.rs kept its own CallKind and String call IDs beside ClientCallKind and ClientCallId; the multi-agent checkpoint mapped one enum onto the other	The ledger uses ClientCallKind and ClientCallId; the mapping and the empty-ID handling are gone
Two multi-agent turns that rehydrated the same tree both committed; the later one replaced the earlier one's tree, losing its agents and pending client calls	The later one is refused with 400 conversation_locked, as on OpenAI (overlapping-multi-agent cassettes)
commit (llm-d) checked a relayed response's call IDs against the continued history with its own loop (validate_output_call_ids), a rule the gateway path does not have	llm-d goes through the same rehydration and ledger as the gateway; no llm-d-only check
CallLedger refused a repeated call_id and a second output for a call; OpenAI accepts both (call-ids cassettes)	A repeated call adds nothing, one output answers a call_id, further outputs are accepted; empty and unknown IDs are still refused
Client items were checked in three places: Responses input (message ID prefix only, in resolve_input_items), Conversations items (every ID prefix, inside ConversationStore), and nowhere for an empty call_id until a later turn	One InOutItem::admit_all, called once per endpoint in the handlers: every kind's ID prefix and a non-empty call_id, with OpenAI's invalid_value / empty_string errors; storage no longer validates
Conversations items listed a client function_call and output without a status	into_public gives an omitted status OpenAI's value: a call in_progress, an output completed
A gateway executor without its own public_output (a custom web search executor registered with with_gateway_executor) left its call as a bare function_call; once history held only public items, the next turn failed with MissingOutput	started_output and public_output are required: every handler returns its own items, and a custom web search executor builds them with the public web_search::started_output_item and web_search::output_item
A gateway call no executor could run (a declared file_search) stayed in the public output as a bare function_call	It leaves the public output and history like a hidden max_tool_calls refusal; the model still sees its error output
upstream_request wrapped a private upstream_request_with_guidance	One upstream_request(ctx, stream, guidance, withheld)
Standalone compaction re-read stored history to recover the public form of open calls	Rehydrated history is already public; the extra read and carry_over::public_calls are removed

Redundancy removed

• Tool lowering in three places: InOutItem::into_input_items and OutputItem::to_input_item turned custom and shell calls into function calls, and the upstream projection lowered the rest. Now only ResponsesInput::normalized_model_input, as one loop (its separate pre-scan, which repeated the same match, is gone).
• Each gateway tool call stored twice: as the model-facing function_call + function_call_output pair and as its public item. Now only the public item.
• History written in two places, per round and at persist, with four helpers reconciling them (record_round_history, mark_outputs_recorded, is_recorded_as_history, is_already_in_history). Now written once, at persist.
• Five persist entry points, each re-checking the response status and should_persist. Now one persist.
• Session and transient-parent logic in ResponseHandler::persist beside the turn logic in persist.rs. Now only in persist.rs; the handlers write.
• Three rehydrate loaders repeating the same steps (history plus request items, unanswered-call check, model input, settings), and the session loader's own copy of apply_effective_settings. Now one set_model_input and apply_effective_settings.
• Four pending-call entry points (pending_calls, first_unanswered_sync_call, resolved_prefix_len, reject_unanswered_sync_call) over one scan. Now CallLedger.
• A second copy of the client-call types: pending_calls.rs's CallKind (the same four kinds as ClientCallKind) and String call IDs, the multi-agent checkpoint's variant-by-variant mapping between the two enums, and into_steering_required_input's empty-ID branch. Now the shared ClientCallKind and ClientCallId.
• A second call-ID rule for llm-d: commit read the continued history again and checked the relayed output's call IDs with its own loop, which the gateway path never ran. Now both paths rely on CallLedger.
• Client item validation in three places with different rules: resolve_input_items (message IDs only), ConversationStore::create and create_items (validate_client_id), and StorageError::InvalidItemId beside the executor errors. Now InOutItem::admit_all with one ClientItemError; the call_id rule lives once, in ClientCallId::validate, which ClientCallId::try_from and the ledger use.
• Two upstream request builders: upstream_request only called upstream_request_with_guidance(ctx, stream, None, None). Now one function.
• Wrappers around a push loop: gateway/history.rs (append_output_items_to_input, append_gateway_results). File removed; callers use ResponsesInput: Extend<InputItem>.
• A shell/custom match in append_round_input repeating to_input_item. Removed.
• Compaction re-reading stored history (carry_over::public_calls) to recover the public form of calls rehydration had already loaded. Removed.
• Standalone model-facing helpers for web search and code interpreter. Now From conversions next to their types.

Changes

• History and lowering
  • OutputItem::to_input_item keeps every public type, including hosted tool-search calls and outputs.
  • ResponsesInput::normalized_model_input is one loop with one match that builds the model copy: custom and shell calls become function calls; web_search_call, mcp_call and code_interpreter_call become a function call and its output; message files become text and images get a detail. It returns the copy (ResponsesInput) instead of a Cow, and drops the pre-scan that decided whether a copy was needed.
  • types/io/web_search.rs (new) and types/io/code_interpreter.rs hold the From conversions from a stored public item to its model-facing call and output. The web search output types and code interpreter execution types stay in tool/.
  • Gateway results feed only the next round's model input. executor/gateway/history.rs is removed; ResponsesInput implements Extend<InputItem>.
• Persist
  • persist.rs::persist checks the storage policy and the response status once, times the write, maps errors, builds the turn's metadata and items, and routes to the conversation or response handler.
  • TurnOutput::Response (a generated response, with its snapshot when stored), TurnOutput::Empty (an empty session turn) and TurnOutput::Compaction (a standalone compaction, stored whatever store says so previous_response_id can continue it).
  • Session turns: an in-turn compaction marks how many outputs it absorbed (mark_outputs_compacted); persistence drops those outputs, except MCP discovery and async calls.
  • commit refuses a missing response ID and an unfinished response, then persists; it no longer checks call IDs.
  • ResponseHandler::persist(&ctx, items, snapshot, &metadata) and ConversationHandler::persist(&ctx, items, snapshot, &metadata) make one store call each.
• Rehydrate
  • from_response, from_session_response and from_conversation load; set_model_input appends the request's items, applies the window (identity, or canonical_session_history for a session) and checks for unanswered calls; the session loader uses apply_effective_settings instead of its own copy.
  • from_conversation sets ResponseCheckpoint::requires_latest for a multi-agent turn, one that continues a restored tree or builds one, so storage refuses it if another turn was stored since (the check PR3 added for a second answer to a pending call). Storage is unchanged.
• Pending calls
  • CallLedger is the module's one public type: scan, pending, awaits_output, require_answered, resolved_prefix_len. Its calls are keyed by ClientCallId, and PendingCall carries a ClientCallId and a ClientCallKind.
  • ClientCallKind gains call_item_name and output_item_name; CallKind is removed.
  • Multi-agent compaction builds its snapshot only for a non-empty resolved prefix.
  • Repeated call_ids follow OpenAI: a repeated call adds nothing, a call is answered once any output for it follows, and further outputs are accepted, for synchronous calls as well as async ones.
• Client item admission
  • InOutItem::admit_all(items, field) (types/io/item/item_id.rs) checks each item a client sends: its ID begins with its kind's prefix (every kind, as OpenAI does), and a call or an output has a non-empty call_id (ClientCallId::validate). A refusal is ClientItemError (InvalidId, EmptyString) with OpenAI's code, param (input[N].id, items[N].call_id) and message.
  • It is called once per endpoint: ResponseHandler::resolve_input_items (Responses input) and ConversationHandler::create and create_items (Conversations items). resolve_input_items keeps the two Responses-only rules: a duplicate item id is refused, and an id naming a stored item is replaced by it. OpenAI accepts duplicate item IDs in a Conversations batch.
  • Storage no longer validates items: validate_client_id and StorageError::InvalidItemId are removed.
  • into_public completes an omitted status, a function_call in_progress and an output completed, through the same complete_omitted_status that into_canonical_items uses.
  • A custom tool call sent without an id decodes with an empty one; id() now reports it as having none.
• Gateway public items
  • GatewayExecutor::started_output and public_output are required, with no None default; plan_gateway_events defaults to the handler's started_output. Every handler implements both: web search, MCP and the code interpreter build their items; hosted tool search builds its call (its planned result still supplies the completed pair); web_fetch returns None, since Messages answers it with its own result blocks.
  • web_search::started_output_item and web_search::output_item are public, so a custom web search executor implements each method in one line. The test executors do.
  • GatewayScheduler::plan omits a call with no bound executor from the public output, as it omits a hidden refusal.
• Upstream request: upstream_request(ctx, stream, guidance, withheld) is the only builder; llm-d passes None, None.
• Removed: validate_output_call_ids, upstream_request_with_guidance, InOutItem::call_id, record_round_history, append_gateway_calls_to_new_input, append_gateway_results' history branch, OutputItem::is_recorded_as_history and is_already_in_history, ResponseContinuation::mark_outputs_recorded, persist_response, persist_if_needed, persist_prepared_response, persist_prepared_turn, log_custom_tool_calls, carry_over::public_calls.
• Tests:
  • Added overlapping_multi_agent_turns_keep_the_first_turns_pending_call (three turns rehydrate one tree; after the first commits with a pending call, a stale-tree turn and a plain turn are refused, and the client's output continues the first turn's call).
  • session_continuation_test's LocalSearch builds its items with the public web search builders, so session_tool_choice_inherits_request_policy_after_internal_relaxation (two session turns over previous_response_id) and session_explicit_conversation_tool_turns_match_no_session_history run a custom executor end to end; both fail with MissingOutput when its public_output returns None.
  • session_commit_rejects_reused_call_id_against_pinned_history becomes session_continuation_accepts_a_call_id_the_pinned_history_already_used: as on OpenAI, the repeated call is stored and the session turn answering it continues.
  • Ledger: empty_call_ids_are_rejected and repeated_call_ids_are_accepted replace the duplicate-rejection test. Admission: the ID-prefix tests move from storage to item_id.rs, plus tests for an empty call_id on calls and outputs and for the omitted-status defaults. function_call_history_status_gets_openai_default replaces the test that kept a missing call status. conversation_storage_regressions checks the refused batch through ConversationHandler.
  • scheduler_retains_event_slot_for_gateway_tool_without_handler becomes scheduler_omits_a_gateway_tool_without_handler_from_public_output.
  • Removed the five llm-d tests that expected commit to refuse a repeated or empty call ID (relayed_{json,sse}_tool_call_ids_are_validated_before_persistence, relayed_{json,sse}_call_id_cannot_reuse_continued_history, relayed_call_id_cannot_reuse_a_call_replayed_in_the_current_input). The strict stream checks that a call ID cannot change between events keep their tests.
• Cassettes: an overlapping-multi-agent scenario in record_cassette.py and record_conversations_api_cassettes.sh (multi-agent turns B, streamed, and C overlap on one tree; D answers C's call), recorded against OpenAI (gpt-5.6-sol, responses_multi_agent=v1 beta, sent on Responses requests only, since the Conversations API rejects it) and the gateway; conversations_api_cassette_test.rs checks both. The gateway's concurrent-writes and competing-tool-outputs cassettes are re-recorded with this PR's code. relayed_stream_validation_test lists both refused B streams as known non-terminal streams.
  • A call-ids scenario (RESPONSES_SCENARIO=call-ids in record_cassette.py and record_responses_api_cassettes.sh), recorded against OpenAI (gpt-4.1) and the gateway: repeated, reused and empty call_ids in Responses input, previous_response_id continuations and Conversations items, 16 turns each with every request named in x-run-id. responses_api_test replays both through the gateway (replay_call_ids_openai, replay_call_ids_gateway), and its recording comparisons pair turns by x-run-id.
• Docs: ARCHITECTURE.md describes public-only history, the one lowering point, the persist path, the public item defaults, client item admission and the ledger's repeated-ID rules; the cassette README lists the call-ids suite; docs/design/tool-framework.md lists the required started_output and public_output. The engineering skill asks to search types/ for an existing type or rule before adding a parallel one.

Behaviour changes

• GET /v1/responses/{id}/input_items no longer returns the gateway's model-facing pairs, matching OpenAI. A first turn returns only its input; a continuation after a web search returns message, web_search_call, ... as OpenAI does.

• On a later turn the model sees each gateway call rebuilt from its public item, under the public item's ID as call_id:

  • web search: the queries and the sources' URLs and titles, not the result text;
  • code interpreter: status and the combined logs;
  • a search refused by max_tool_calls replays with no results and no refusal text. OpenAI behaves the same: asked on the next turn, its model reports no results and no error for the refused search.

  The inference prefix after a gateway call therefore differs from the live round, so a continuation does not reuse the prompt cache past that point.

• Client-supplied web_search_call and code_interpreter_call input items are lowered to function calls like every other tool item.

• A repeated or reused call_id (two calls, two outputs for one call, a continuation re-sending a stored call) is accepted on every route, as on OpenAI; the gateway refused it before. llm-d no longer refuses one at commit.

• An empty call_id in Responses input (every route, store: false included) or Conversations items is refused with 400 empty_string, param input[N].call_id / items[N].call_id, as on OpenAI. Before, Responses accepted it with store: false and the conversation stored it, failing on a later turn.

• A Responses input item of any kind whose id has the wrong prefix is refused with 400 invalid_value, as on OpenAI; before, only messages were checked.

• Conversations and input_items listings show an omitted function_call status as in_progress and an omitted output status as completed, as on OpenAI.

• Two custom tool calls without an id are no longer refused as duplicate items.

• A custom gateway executor must implement started_output and public_output; one that relied on the None defaults no longer compiles.

• A call to a gateway tool with no executor (file_search) is no longer shown in the response output or stored; the model still receives its error output.

• Public API:

  • GatewayExecutor::started_output and public_output are required (they defaulted to None); web_search::started_output_item and web_search::output_item are public;
  • StorageError::InvalidItemId is replaced by ExecutorError::ClientItem(ClientItemError); ConversationStore::create and create_items no longer validate items (use ConversationHandler); ClientCallId::validate is new;
  • persist_turn(ctx, exec_ctx) replaces persist_turn(ctx, output_items, conv_handler, resp_handler) (one call site in the server);
  • persist_response is removed (use commit);
  • upstream_request takes guidance and withheld (None, None for llm-d).

• Metrics: an empty session turn is now timed as a persist stage; a write that finds no configured storage is no longer timed on any path.

• A multi-agent conversation turn stored after another turn that was stored since it rehydrated fails with 400 conversation_locked and stores nothing, as on OpenAI; the earlier turn's tree and pending client calls survive.

Known limitations (planned for the next PR)

• Response persist still reads the parent row again for its history_item_ids; it can extend the checkpoint's rows instead.
• A plain turn (no tree, no pending call) that commits after a concurrent turn started a new tree becomes the latest turn, so that tree is not restored. Neither turn's rehydration saw a tree, so the executor cannot mark either; catching it would need storage to read the latest turn's metadata.
• Durable and conversation turns with an in-turn compaction still store the outputs the summary absorbed. This predates this PR.

Size

Measured against main, which now includes PR3. Production code excludes integration tests (tests/, *tests.rs), cassettes, benches and docs; unit tests inside src files count as production code. Cassettes include the two recorder scripts.

	Files	Added	Deleted	Net
Production code	42	+954	−1,079	−125
Tests and benches	21	+396	−359	+37
Cassettes	6	+9,243	−1,983	+7,260
Docs	3	+49	−37	+12
Total	73	+10,643	−3,463	+7,180

Stack

The cleanup is split into five stacked PRs; each builds on the previous one and passes CI on its own. This is PR4.

PR	Branch	Carries	Leaves for later
1	refactor/storage-layer	One typed storage layer: one conversion pair per column, one store function per operation, one rehydrate and one persist per handler.	Stored data is unchanged.
2	refactor/storage-schema	What is stored: typed items.metadata, public_id NOT NULL, the responses.snapshot column, recreate/reject for databases from earlier releases (migration 0009).	Conversation turns still use the version check and revision.
3	refactor/append-only-turns	Conversation turns no longer fail with conversation_locked when the conversation changes while they run, except a second answer to the same pending call; a turn stores the history it started from (one ResponseCheckpoint from rehydration to persist); conversations.revision is dropped (migration 0010); concurrent-writes and competing-tool-outputs cassettes.	Multi-agent and compaction concurrency.
4 (this PR)	refactor/public-history	History stores public items only; one lowering point for the model; one persist path in persist.rs; input_items match OpenAI for gateway tools; one CallLedger on the shared client-call types; overlapping multi-agent turns refused as on OpenAI.	Compaction during a concurrent turn (PR5).
5	refactor/typed-request	The checkpoint extended to response chains and sessions, fewer reads and statements per turn, typed request metadata, compaction during a concurrent turn (stale-compaction).	Nothing from this RFC; tenant on RequestContext stays a follow-up.

Test Plan

• Full local CI, before the last scan_calls fold, which then passed clippy, the core unit tests and the shell cassettes (pre-commit, clippy, cargo test --workspace: 3018 passed, 0 failed, 20 ignored; launchers, CLI end-to-end, OpenTelemetry fixtures, embedded code interpreter clippy, build and characterization; MSRV; PostgreSQL schema 4, storage 5, tenant isolation 6)
• After the required public item hooks: workspace clippy clean; cargo test --workspace: 3013 passed, 0 failed, 20 ignored
• After the llm-d call-ID change (1c5bed0): workspace clippy clean; core lib 1366 passed (6 ignored); llm-d split_execution_integration 14; relayed_stream_validation_test 12; terminal_status_test 9; shell_tool_test 26; stateful_responses_integration 5; input_file_repro 41; responses_test 88
• Updated tests: input_items for MCP and web search return only public items; session history holds web_search_call; continuations find the lowered call under the public ID
• New tests: in-turn session compaction keeps MCP discovery and async calls; overlapping multi-agent turns keep the first turn's pending call (fails without the fix)
• OpenAI probe (gpt-5.6, max_tool_calls: 1, two searches): the refused search is stored as web_search_call (searching) and the next turn's model reports no results and no error
• overlapping-multi-agent recorded against OpenAI (gpt-5.6-sol) and the gateway (vLLM, Qwen/Qwen3.8-27B-FP8): on both, B is refused with conversation_locked, C keeps its pending call, and D completes with the call's output (7°C). Gateway concurrent-writes and competing-tool-outputs re-recorded with this PR's code; conversations_api_cassette_test: 13 passed; relayed_stream_validation_test: 12 passed
• Gateway re-recorded through a release build of the final code against vLLM (Qwen/Qwen3.8-27B-FP8). The re-recordings are not committed.
  • Responses (lifecycle, pagination, edge cases, conversation, replayed items; JSON and SSE) and Conversations (all 9 scenarios): match the previous PR's recordings on every turn (method, path, status, error fields, output and input item types and statuses, SSE event sequence, page shape), apart from one opaque pagination cursor.
  • Responses tool-items: input_items no longer lists the gateway's function_call / function_call_output pairs and has the shape of the committed OpenAI reference (message, mcp_list_tools, reasoning, web_search_call, ..., mcp_call, ...); client function, custom and shell calls and their outputs are still listed. Other differences are model variance (which tools the model chose to call, an intermediate assistant message, the order of multi-agent calls).
  • store: false (full-history replay, stored and unstored parents, explicit conversation, conversation items, function replay; JSON and SSE): JSON matches the committed gateway cassettes on every turn; SSE differs only by response.reasoning_part.added / response.reasoning_part.done, which main emits since 474d154 (before the previous PR).
• Gateway re-recorded with this PR's code against vLLM (Qwen/Qwen3.8-27B-FP8), not committed: responses (11/11), store_false (20/20), multi_turn_steering (5 categories; vision needs a vision model) and multi_agent (17/17, including the 5 WebSocket scenarios). No behaviour change comes from this PR. The differences from the committed recordings are earlier main changes (response.reasoning_part.* events, store echoed on response objects, function_call_output listed with status: completed, OpenAI's wording for an output with an unknown call_id, no repeated mcp_list_tools for resumed subagents) or the different model (preamble messages, other tool or agent choices, output-token limits). Replay tests against the new files: responses_api_test 29/29, multi_agent_contract_test, hosted_tool_search_test 38/38, responses_websocket_test 110/110; two tests that pin details of the old recordings (a vLLM stream fault vLLM no longer emits, and an exact response count) fail only against the new files
• Gateway re-recorded after the persist.rs cleanup against vLLM (Qwen/Qwen3.8-27B-FP8), not committed: web_search (2/2), code_interpreter (3/3), responses (11/11) and conversations (12/12), all on the first attempt. No behaviour change comes from this PR; the differences from the committed recordings are earlier main changes (store echoed on response objects, reasoning item statuses, function_call_output listed with status: completed) or the model (multi-agent item order). competing-tool-outputs still differs from OpenAI only in which racing request wins. Replay tests against the new files: responses_api_test 29/29, conversations_api_cassette_test 13/13, conversation_branch_response_test 2/2, accumulator_cassette_test 40/40, code_interpreter_characterization_test 4/4; web_search_tool_test and relayed_stream_validation_test fail only on pins to the old model name and the old recordings' rejected-stream count

[tjtanaa]

One session-continuation case needs attention.

[tjtanaa]

One existing-history compatibility case needs attention.

[maralbahari]

@tjtanaa @franciscojavierarceo any remaining review can be addressed in the next PR in stack. the PR grow to 96 file changes including the new recording yaml files and test files.