[Bug]: Default user agent is a stale Chrome/116 string that contradicts the bundled Chromium #2493

crawl4ai version

0.9.5 (Docker image unclecode/crawl4ai:0.9.5); the same BrowserConfig code is on develop.

Expected Behavior

The default BrowserConfig.user_agent matches the browser crawl4ai launches.
An empty user_agent (to keep the browser's own User-Agent) sends no forged client hints.

Current Behavior

1. Stale, non-standard default. BrowserConfig.user_agent defaults to
   Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36
   (crawl4ai/async_configs.py, also the fallback of user_agent_generator.py).
   It lacks the (KHTML, like Gecko) token Chrome sends, and announces Chrome 116, while the 0.9.5 image runs Playwright 1.64 with Chromium 156.0.8078.4.
   The generated sec-ch-ua says 116 too, whereas the other request headers (Accept-Encoding: … zstd, Priority: u=0, i) come from Chromium 156.
   The BrowserConfig docstring documents yet another default, a well-formed macOS User-Agent.
   Since Playwright is not pinned (playwright>=1.49.0), the bundled Chromium changes with each image build, so a hard-coded string keeps drifting.
2. Empty sec-ch-ua with an empty User-Agent. BrowserConfig(user_agent="") keeps the browser's native User-Agent, but __init__ still runs self.headers.setdefault("sec-ch-ua", self.browser_hint), and generate_client_hints("") returns '': every request then carries an empty sec-ch-ua header.
3. A worker keeps the real User-Agent. With the default config, CreepJS reports in its ServiceWorkerGlobalScope worker the User-Agent … HeadlessChrome/156.0.0.0 …, while the page says Chrome/116.
   Passing the same string as a launch argument (--user-agent=… in extra_args) makes the worker report it too.
   (Observed on the 0.9.5 image with fix(browser): pass the default chromium channel again, fall back to the shell #2448 backported; the User-Agent code is the same without it.)

Related: #2472 / #2473 touch the same auto-added sec-ch-ua, but #2473 only leaves it out of dump(); __init__ is unchanged.

Is this reproducible?

Yes

Inputs Causing the Bug

- URL(s): https://httpbin.org/headers (headers sent), https://abrahamjuliot.github.io/creepjs/ (worker User-Agent)
- Settings: default BrowserConfig; Docker server /md endpoint

Steps to Reproduce

docker run -d --name c4ai --shm-size=1g -p 11235:11235 -e CRAWL4AI_API_TOKEN=test unclecode/crawl4ai:0.9.5
curl -s -X POST http://localhost:11235/md -H "Authorization: Bearer test" \
  -H "Content-Type: application/json" -d '{"url": "https://httpbin.org/headers", "f": "raw"}'
# "User-Agent" and "Sec-Ch-Ua" announce Chrome 116; the bundled Chromium is 156:
docker exec c4ai sh -c 'grep -A3 "\"chromium\"" /usr/local/lib/python3.12/site-packages/playwright/driver/package/browsers.json'

Code snippets

from crawl4ai import BrowserConfig

print(BrowserConfig().user_agent)
# Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36
print(BrowserConfig(user_agent="").headers)
# {'sec-ch-ua': ''}

Suggested fix

• Default to the launched browser's own User-Agent (with HeadlessChrome replaced by Chrome in headless mode) rather than a hard-coded string; at least add (KHTML, like Gecko) and keep the version in step with the bundled Chromium.
• Skip the sec-ch-ua default when user_agent is empty.
• Also pass the User-Agent as --user-agent at launch, so workers report the same one.

OS

Linux (Docker on WSL2)

Python version

3.12 (image)

Browser

Chromium bundled with Playwright 1.64 (the stock 0.9.5 image launches chrome-headless-shell, see #2382)

Browser version

156.0.8078.4