Repository navigation
Description
Observed Behavior
OpenAIChatCompletionClient turns one framework Message with several content parts into one request message per part. For a user message with text and an image, the request contains two consecutive user messages: {"role": "user", "content": "What is in this picture?"} followed by {"role": "user", "content": [{"type": "image_url", ...}]}. The same happens for [image, text], for [text, text] (two user messages), and for an assistant message with two text parts (two assistant messages).
I am asking whether this is intended.
In _build_openai_messages (python/packages/openai/agent_framework_openai/_chat_completion_client.py) a fresh args dict is created per content item inside for content in message.contents and appended to all_messages as soon as it has content. Parts are not accumulated into one content list; the only merge is text plus tool calls for assistant messages (mergeable_assistant).
There is an existing test that pins the split for assistant messages: test_prepare_message_with_image_between_text_and_function_call_stays_separate (python/packages/openai/tests/openai/test_openai_chat_completion_client.py), so it may be deliberate there. I did not find a test for user messages.
For contrast, OpenAIChatClient (Responses API) keeps one message: for the same Message it produces a single {"type": "message", "role": "user", "content": [{"type": "input_text", ...}, {"type": "input_image", ...}]}.
Possible impact, stated as a risk and not something I verified: OpenAI itself accepts consecutive user messages, so the difference is invisible there. OpenAI-compatible servers whose chat template requires user/assistant alternation might reject or reorder such a request. I did not run any such server.
Expected Behavior
I would have expected one {"role": "user", "content": [{"type": "text", ...}, {"type": "image_url", ...}]} message for one framework user message, matching what the Responses client does. If the split is intentional, a note in the docs or a code comment on why would help.
If a change is wanted, two options: (a) for user messages (and plain assistant messages), accumulate the non-tool parts into one content list and emit one message, keeping the existing flatten of text-only lists; (b) keep the current behavior and document it. I will wait for maintainer agreement before starting any implementation.
Steps to Reproduce
- Create
OpenAIChatCompletionClientwith a mockedchat.completions.createthat records its keyword arguments. - Call
get_responsewith one userMessagecontaining a text part and an image part. - Print the recorded
messages.
Minimal Reproduction
import asyncio, json
from unittest.mock import MagicMock
from openai.types.chat import ChatCompletion
from agent_framework import Message, Content
from agent_framework_openai import OpenAIChatCompletionClient
sent = []
async def create(**kw):
sent.append(kw)
return ChatCompletion.model_validate({"id":"1","object":"chat.completion","created":1,"model":"m",
"choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"ok"}}],
"usage":{"prompt_tokens":1,"completion_tokens":1,"total_tokens":2}})
oc = MagicMock(); oc.chat.completions.create = create; oc.base_url = "https://api.openai.com/v1"
c = OpenAIChatCompletionClient(model="m", async_client=oc)
async def main():
await c.get_response([Message("user", [Content.from_text("What is in this picture?"),
Content.from_uri("https://example.com/cat.png", media_type="image/png")])])
print(json.dumps(sent[0]["messages"], indent=1))
asyncio.run(main())Output:
[
{
"role": "user",
"content": "What is in this picture?"
},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://example.com/cat.png"
}
}
]
}
]
Error Messages and Stack Traces
None; the request succeeds.
Package Versions
agent-framework-core: 1.26.0, agent-framework-openai: 1.18.2 (at main 92838b6), openai: 3.51.0
Python Version
Python 3.12.11
Operating System
macOS
Regression
Unknown
Additional Context
The behavior is the same since the Chat Completions client was extracted into its own module (5e056b6).
Acknowledgements
- I searched existing issues and did not find a duplicate.
- I personally verified this behavior and the reproduction details are authentic.
- I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.
Thanks a lot for maintaining Agent Framework, and sorry if this is already a known design choice!
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsNo status
Activity
github-actions commentedon Nov 14, 2026
Thanks for the detailed report. I ran an independent version of the repro on
main(92838b6) and observed the same output: one[text, image]userMessageis sent as two consecutiveusermessages byOpenAIChatCompletionClient. Evidence: https://github.com/microsoft/agent-framework/actions/runs/43632129250Support boundary: unclear. I found no maintained spec, ADR or public doc saying whether Chat Completions messages should be merged or split, so whether this is a bug or intended behavior is a maintainer decision. I did not test any alternation-strict OpenAI-compatible server. Routing to the
python/packages/openaiowner.saquibjawedbit commentedon Nov 15, 2026
Reproduced on main (92838b6) at unit level:
_build_openai_messagesinagent_framework_openai/_chat_completion_client.pybuilds one message per content item, so a userMessagewith [text, image] (or [text, text]) goes out as two user messages.Proposed fix: for
usermessages only, put all parts into onecontentlist and emit a single message. Text-only lists still flatten to a string; assistant, tool and system handling (including the #8382 merging) stays unchanged. This does change whatmessage_preparerreceives for user messages, so I'd call that out in the PR.Tests: new unit tests for user [text, image], [image, text] and [text, text], plus the existing suite.
If the reporter planned to fix this, I'll step back. Otherwise, happy to take this if that approach works for you.