fix(lexer): normalize narrow literals to UTF-8 execution bytes

[gengjiawen]

Raw non-ASCII text and universal escapes produced different narrow literal values. Encode both as UTF-8 execution bytes, keep numeric byte escapes distinct, consume astral characters as complete code points, and apply the same contract to macro-created string literals.

Validation: equivalent-spelling and multibyte-character regressions, macro stringification, lint, formatting, typecheck, build and all tests.

Mixed narrow/wide concatenation re-lexes each narrow spelling using the final prefix, preserving the difference between byte escapes and Unicode text; incompatible wide encodings receive a diagnostic.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
c-parser	Ready	Preview	Sep 9, 2026 8:25am UTC

[gengjiawen]

Holding this one for your call — the other 45 audit PRs (#41–#88) are merged; this is the only one left.

The change is technically right: a narrow string literal is a byte sequence in the execution character set, so sizeof("café") should be 6, not 5. But it also changes observable AST output for a published package:

parse('char *s = "café";')  // value was 'café', becomes 'café' (UTF-8 bytes as code units)

That is a visible regression for the primary consumer (AST Explorer renders value directly), shipped under a fix: prefix. Worth deciding explicitly whether you want it, and if so whether it should be called out in the changelog / released as a breaking change rather than folded into a routine bump.

Verified: the change combines cleanly with everything now on main — 907/907 tests, typecheck, lint and build all pass. The only merge conflict is in README.md, where both this PR and #57 appended a paragraph after ## License; on main that section now lives under ### Token kinds and serialization before ## Development, so this PR's paragraph should be appended there.