[satiricalguru]

Fixes #4475

ScanCode truncates or transliterates URL paths such as the Slovak and Cyrillic Wikipedia links in the issue. Preserve Unicode characters through URL normalization so the output points to the original path.

Normalize typographic quotes, remove invisible formatting marks, and trim trailing Unicode punctuation. If CJK prose joins two bare HTTP links into one candidate, recover the pair when the second scheme and sentence punctuation identify it. Update the stored URLs used for license clue filtering.

Validation

• Finder and clue-filter tests on Python 3.12: 69 passed and 2 existing expected failures. The two regressions for prose and punctuation fail against the previous finder implementation.

• Full license ignorable-clue validation: 37,669 passed. The two failures are the same FSL assertions reported on upstream develop.

• GitHub reported 31 passing checks. The failed Azure job reproduced the same two FSL assertions as upstream, which also caused its parent pipeline to fail.

Tasks

• Reviewed the contribution guidelines.
• Linked the original issue.
• Upstream CI completed for the revised commit. See the results above.
• Updated CHANGELOG.rst and AUTHORS.rst.

AI assistance: Codex assisted with investigation, implementation and local tests.

[knowprasad]

Took a look through the changes in cluecode/finder.py and the updated rule data.

Supporting Unicode in URLs for issue #4475 (like the Cyrillic and Slovak Wikipedia paths) is a great improvement. However, setting ascii_matches=False reveals a couple of edge cases in how URL boundaries and trailing punctuation are handled:

1. In src/cluecode/finder.py, url_body is defined as [^\s<>\[\]"]. With Unicode matching enabled, non-whitespace characters in languages without word spaces get matched directly into the URL. We can see this in cc-by-sa-2.1-jp.LICENSE, where the Japanese sentence (から、クリエイティブ...) immediately following https://creativecommons.org/ was absorbed into the URL.
2. In end_of_url_cleaner(), trailing characters are stripped using string.punctuation, which only covers ASCII symbols. Unicode punctuation like ¬ (\u00ac), † (\u2020), or CJK full stops 。 (\u3002) remain attached to the URL and end up percent-encoded, which is why http://www.mozilla.org/MPL/.%C2%AC%E2%80%A0 appeared in mpl-1.1_or_gpl-2.0-plus_3.RULE.

Having to add percent-encoded sentence fragments and punctuation to .LICENSE and .RULE files usually indicates the extractor over-matched trailing characters. If we update end_of_url_cleaner() to clean Unicode punctuation (for example checking unicodedata.category(char).startswith('P') or explicit CJK punctuation), URLs like http://www.mozilla.org/MPL/ will trim cleanly without needing changes to existing license rules.