Skip to content

Python: [Bug]: Is it intended that the Chat Completions client sends one multimodal user message as several messages? #10365

Open

Description

Observed Behavior

OpenAIChatCompletionClient turns one framework Message with several content parts into one request message per part. For a user message with text and an image, the request contains two consecutive user messages: {"role": "user", "content": "What is in this picture?"} followed by {"role": "user", "content": [{"type": "image_url", ...}]}. The same happens for [image, text], for [text, text] (two user messages), and for an assistant message with two text parts (two assistant messages).

I am asking whether this is intended.

In _build_openai_messages (python/packages/openai/agent_framework_openai/_chat_completion_client.py) a fresh args dict is created per content item inside for content in message.contents and appended to all_messages as soon as it has content. Parts are not accumulated into one content list; the only merge is text plus tool calls for assistant messages (mergeable_assistant).

There is an existing test that pins the split for assistant messages: test_prepare_message_with_image_between_text_and_function_call_stays_separate (python/packages/openai/tests/openai/test_openai_chat_completion_client.py), so it may be deliberate there. I did not find a test for user messages.

For contrast, OpenAIChatClient (Responses API) keeps one message: for the same Message it produces a single {"type": "message", "role": "user", "content": [{"type": "input_text", ...}, {"type": "input_image", ...}]}.

Possible impact, stated as a risk and not something I verified: OpenAI itself accepts consecutive user messages, so the difference is invisible there. OpenAI-compatible servers whose chat template requires user/assistant alternation might reject or reorder such a request. I did not run any such server.

Expected Behavior

I would have expected one {"role": "user", "content": [{"type": "text", ...}, {"type": "image_url", ...}]} message for one framework user message, matching what the Responses client does. If the split is intentional, a note in the docs or a code comment on why would help.

If a change is wanted, two options: (a) for user messages (and plain assistant messages), accumulate the non-tool parts into one content list and emit one message, keeping the existing flatten of text-only lists; (b) keep the current behavior and document it. I will wait for maintainer agreement before starting any implementation.

Steps to Reproduce

  1. Create OpenAIChatCompletionClient with a mocked chat.completions.create that records its keyword arguments.
  2. Call get_response with one user Message containing a text part and an image part.
  3. Print the recorded messages.

Minimal Reproduction

import asyncio, json
from unittest.mock import MagicMock
from openai.types.chat import ChatCompletion
from agent_framework import Message, Content
from agent_framework_openai import OpenAIChatCompletionClient

sent = []
async def create(**kw):
    sent.append(kw)
    return ChatCompletion.model_validate({"id":"1","object":"chat.completion","created":1,"model":"m",
      "choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"ok"}}],
      "usage":{"prompt_tokens":1,"completion_tokens":1,"total_tokens":2}})
oc = MagicMock(); oc.chat.completions.create = create; oc.base_url = "https://api.openai.com/v1"
c = OpenAIChatCompletionClient(model="m", async_client=oc)
async def main():
    await c.get_response([Message("user", [Content.from_text("What is in this picture?"),
                                           Content.from_uri("https://example.com/cat.png", media_type="image/png")])])
    print(json.dumps(sent[0]["messages"], indent=1))
asyncio.run(main())

Output:

[
 {
  "role": "user",
  "content": "What is in this picture?"
 },
 {
  "role": "user",
  "content": [
   {
    "type": "image_url",
    "image_url": {
     "url": "https://example.com/cat.png"
    }
   }
  ]
 }
]

Error Messages and Stack Traces

None; the request succeeds.

Package Versions

agent-framework-core: 1.26.0, agent-framework-openai: 1.18.2 (at main 92838b6), openai: 3.51.0

Python Version

Python 3.12.11

Operating System

macOS

Regression

Unknown

Additional Context

The behavior is the same since the Chat Completions client was extracted into its own module (5e056b6).

Acknowledgements

  • I searched existing issues and did not find a duplicate.
  • I personally verified this behavior and the reproduction details are authentic.
  • I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.

Thanks a lot for maintaining Agent Framework, and sorry if this is already a known design choice!

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 14, 2026
  2. added
    agentsUsage: [Issues, PRs], Target: Single agent
    reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow
    on Nov 14, 2026
  3. github-actions commented on Nov 14, 2026

    @github-actions
    Contributor

    Thanks for the detailed report. I ran an independent version of the repro on main (92838b6) and observed the same output: one [text, image] user Message is sent as two consecutive user messages by OpenAIChatCompletionClient. Evidence: https://github.com/microsoft/agent-framework/actions/runs/43632129250

    Support boundary: unclear. I found no maintained spec, ADR or public doc saying whether Chat Completions messages should be merged or split, so whether this is a bug or intended behavior is a maintainer decision. I did not test any alternation-strict OpenAI-compatible server. Routing to the python/packages/openai owner.

    Generated by Issue Routing for #10365 · copilot · auto · ~19 AIC · ⌖ ~1 AIC · ⊞ 5.2K · ◷

  4. saquibjawedbit commented on Nov 15, 2026

    @saquibjawedbit
    Contributor

    Reproduced on main (92838b6) at unit level: _build_openai_messages in agent_framework_openai/_chat_completion_client.py builds one message per content item, so a user Message with [text, image] (or [text, text]) goes out as two user messages.

    Proposed fix: for user messages only, put all parts into one content list and emit a single message. Text-only lists still flatten to a string; assistant, tool and system handling (including the #8382 merging) stays unchanged. This does change what message_preparer receives for user messages, so I'd call that out in the PR.

    Tests: new unit tests for user [text, image], [image, text] and [text, text], plus the existing suite.

    If the reporter planned to fix this, I'll step back. Otherwise, happy to take this if that approach works for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agentsUsage: [Issues, PRs], Target: Single agentpythonUsage: [Issues, PRs], Target: PythonreproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflowtriageUsage: [Issues], Target: All issues that still need to be triaged

    Type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions