[Bhumit267]

Fixes #1510

Description

In Java, strings are UTF-16 encoded. Characters outside the Basic Multilingual Plane (BMP)—such as surrogate pairs (𠮷, emoji, and CJK extension ideographs)—occupy two 16-bit code units.

Following the corresponding JavaScript resolution in #1511, Parser.parse() previously iterated by UTF-16 code units (i = 1; i < length; i++) and extracted feature windows using sentence.substring(...). When a surrogate pair appeared in the text:

1. substring sliced half of the surrogate pair (lone high or low surrogates), which never matched features in models trained using Unicode code points (in Python where len("𠮷") == 1).
2. Even with Character.isSurrogatePair(sentence.charAt(i - 1), sentence.charAt(i)), neighboring n-gram features (such as UW4, BW1, BW2, TW1, TW2) were malformed because slices spanned across partial surrogates.

Solution

• Segment the string into Unicode code points (String[] chars) using codePointAt and Character.charCount.
• Iterate through code points (i < chars.length) while tracking cumulative UTF-16 code unit offset (offset += chars[i - 1].length()), ensuring boundary indices passed to sentence.substring(phraseStart, offset) accurately slice the original Java string without splitting surrogate pairs.
• Consolidate boundary checks into if (i > 2), if (i > 1), if (i + 1 < chars.length), and if (i + 2 < chars.length) blocks, evaluating each bound once per code point and matching the pattern used in budoux/parser.py and javascript/src/parser.ts.
• Extract all unigram (UW1–UW6), bigram (BW1–BW3), and trigram (TW1–TW4) feature windows using Unicode code point strings from chars.

Tests & Verification

• Added testMatchSurrogatePairFeatures in ParserTest.java verifying that features containing surrogate pairs (e.g. UW3: {"𠮷": 1000}, BW2: {"𠮷野": 1000}) are matched and parsed correctly.
• Added testCumulativeOffsetMultipleSurrogatePairs verifying cumulative offset tracking across multiple surrogate pairs ("𠮷野𠮷家").
• Added testOuterWindowFeaturesAcrossSurrogatePair verifying outer window unigrams (UW2, UW5) matching BMP characters across intervening surrogate pairs ("あ𠮷い").
• Ran Maven test suite: all 38 tests in ParserTest and HTMLProcessorTest passed.
• Ran Python test suite: all 62 tests passed.

[Bhumit267]

Hi @tushuhei, when you have a moment, could you please take a look at this PR?

This resolves the Java side of #1510 (following #1511 for JavaScript). It updates Parser.parse() to decompose the string into Unicode code points (String[] chars) and extract n-gram features from code points, while tracking cumulative UTF-16 code unit offsets (offset += chars[i - 1].length()) so substring boundaries accurately slice the original string. Boundary guards are consolidated into if (i > 2), if (i > 1), if (i + 1 < chars.length), and if (i + 2 < chars.length) blocks matching Python and JavaScript.

All 38 Maven tests (including 3 new regression tests for surrogate pairs, multi-surrogate cumulative offsets, and intervening outer window features) and 62 Python tests pass. Thank you!