Skip to content

Replace C0 control characters with U+FFFD when writing - #101

Merged
andy128k merged 1 commit into
rust-syndication:masterfrom
emschwartz:replace-control-chars
Nov 16, 2026
Merged

andy128k merged 1 commit into
rust-syndication:masterfrom
emschwartz:replace-control-chars

Conversation

@emschwartz

Copy link
Copy Markdown
Contributor

Writing a feed whose text contains a C0 control character produced a document that strict XML parsers reject.

Before (xmllint --noout fails with "PCDATA invalid Char value 30"):

<title>Ti^^tle</title>                         (raw 0x1E byte)
<summary type="html">&lt;scr^Aipt&gt;</summary>  (raw 0x01 byte)

After (xmllint --noout passes), the character is written as a raw U+FFFD:

<title>Ti�tle</title>
<summary type="html">&lt;scr�ipt&gt;</summary>

XML 1.0 forbids C0 controls other than tab, LF and CR everywhere, even as &#x1E;, so they cannot be escaped. Seen in production on a reader that scraped U+001E and U+0008 into titles. The fix wraps the output of Feed::write_with_config and Entry::write_with_config in a small Write adapter, so every text, attribute, element name and attribute name is covered.

The characters are replaced with U+FFFD rather than dropped so the text on either side cannot join into a new token (<scr\u{1}ipt> must not become <script>). C0 bytes never occur inside multi-byte UTF-8, so the swap keeps the output valid UTF-8. U+FFFE and U+FFFF are not handled.

Tested with write_replaces_control_characters (title, html summary, category term). It asserts the raw output has no forbidden byte, since the crate's own parser is lenient, then parses it back. cargo test --all-targets passes with and without default features and cargo fmt --check is clean.

Test output before the fix
thread 'write_replaces_control_characters' panicked at tests/write.rs:121:5:
assertion failed: !out.bytes().any(|b| b < 0x20 && !matches!(b, b'\t' | b'\n' | b'\r'))

Authored by Claude (claude-sonnet-5-5, claude-opus-5-5)

XML 1.0 forbids C0 control characters other than tab, LF and CR, even as
character references, but the writer emitted them raw, so a stray U+001E
in a title produced a document that strict parsers such as xmllint
reject. Replace them in the output stream, which covers text and
attribute values alike.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@emschwartz
emschwartz force-pushed the replace-control-chars branch from cc15b5c to 188ef62 Compare November 11, 2026 16:27
@codecov

codecov Bot commented Nov 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.25000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.27%. Comparing base (01f0a77) to head (188ef62).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
src/toxml.rs 80.00% 2 Missing ⚠️
src/entry.rs 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #101      +/-   ##
==========================================
+ Coverage   83.85%   84.27%   +0.41%     
==========================================
  Files          14       14              
  Lines        1016     1030      +14     
==========================================
+ Hits          852      868      +16     
+ Misses        164      162       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@DCjanus
DCjanus self-requested a review November 12, 2026 06:43
Comment thread src/toxml.rs Resolved
@andy128k

Copy link
Copy Markdown
Member

Nice. But why not to contribute this change to quick_xml?

@DCjanus
DCjanus removed their request for review November 16, 2026 03:01
@andy128k
andy128k merged commit ef78ce2 into rust-syndication:master Nov 16, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants