OPENNLP-1929: Regex removal (7/10): Split BasicContextGenerator features on a literal separator

[krickert]

BasicContextGenerator.getContext split on String.split(separator), so the separator was compiled as a regex on each call: "|" split on each character, "." returned an empty array, and "+" or "(" failed with PatternSyntaxException at prediction time.

Changes:

• Default constructor splits on runs of Unicode whitespace (StringUtil.isUnicodeWhitespace), independent of opennlp.whitespace.mode. "a b" now gives [a, b], not [a, , b].
• BasicContextGenerator(String sep) takes the separator as written.
• Empty predicates are not returned. Null or empty separators, separators with an unpaired surrogate, and a null input throw IllegalArgumentException.
• The manual paragraph in machine-learning.xml states these rules.

Breaking for code that passed a regex: a separator containing a backslash, such as "\\|" or "\\s+", now throws IllegalArgumentException at construction, so the old escapes fail at startup instead of matching no input. "[,;]" is a plain separator. No code in the repository uses the separator constructor; the manual has the migration table.

Tests: BasicContextGeneratorTest, 129 cases, failing on the old code.

OPENNLP-1929

[rzo1]

Little time, so here is a GPT 5.6-sol review instead for now

Compatibility note. No additional splitting bug found; please document the intentional constructor behavior changes.

Validation across the combined stack: 1,856 targeted tests, zero failures, one skipped.

[rzo1]

This intentionally changes existing constructor behavior: regex separators such as \s+ become literal, and empty separators now throw. Please include these changes in the migration notes.

[rzo1]

Here are some additional comments. The literal-separator change breaks existing callers silently, so it needs a deprecation path or a documented maintainer decision.

Blocking

1. BasicContextGenerator.java:81. The default constructor splits with WhitespaceTokenizer.INSTANCE, a shared mutable singleton. TokenizerME.java:195-196 calls setKeepNewLines(...) on it on every call, so once any TokenizerME with keepNewLines=true has run, getContext("a\nb\r\nc") returns [a, \n, b, \r, \n, c]. Across threads this is a data race. Scan with a private loop instead, and add a test that sets INSTANCE.setKeepNewLines(true) first.

2. BasicContextGenerator.java:58-66. A regex separator now silently stops splitting. o.split(sep) was a regex split, so escaping was the only way to split on | or ., and exactly those callers break:

   • "\\|" on a|b: old [a, b], new [a|b]
   • "\\." on a.b: old [a, b], new [a.b]
   • "\\s+" on a b\tc: old [a, b, c], new [a b\tc]
   • "[ \t]" and "[,;]" behave the same way

   The whole line becomes one unknown predicate and the model silently returns its prior outcome. This is public maxent API. Keep the String constructor with regex semantics (compile once in the constructor), mark it @Deprecated(forRemoval = true), and add a literal BasicContextGenerator(char) or a named factory as the replacement. Otherwise, get an explicit decision on dev@ and add it to the 3.0.0 migration notes.

3. BasicContextGenerator.java:50-52, :81. The default constructor changes behavior, and the PR description doesn't mention it. It used to split on one U+0020 and now splits on whitespace runs, following the global opennlp.whitespace.mode, and drops empty parts. On 1M fuzz inputs old vs new, 858,399 differ. U+0085 splits under UNICODE but not LEGACY; U+001C–U+001F behave the other way round. "".split(" ") gave [""]; now []. Document the mode dependency (link StringUtil#isWhitespace) and state the change.

4. PR description. It is out of date:

   • "keeps the String.split shape (a leading occurrence gives an empty first element…)" is wrong; the last commit drops every empty part (",a" gives [a]).
   • The default whitespace split and the IAE for null input are not mentioned.
   • "A dot matched no boundary at all" is wrong: . matched every character, so "a.b".split(".") returned [].

   Rewrite it. Retitle, e.g. "OPENNLP-1929: BasicContextGenerator splits on whitespace by default and takes a custom separator literally".

5. machine-learning.xml:82-94. "the same definition of whitespace as the event streams" is false. FileEventStream uses StringTokenizer (ASCII \t\n\r\f), and RealValueFileEventStream/RealBasicEventStream still use split("\\s+") here; only OPENNLP-1933: Regex removal (5/10): Replace per-call String.split/matches/replaceAll patterns #1279 changes them. Event lines also start with an outcome, so "for the line format of the event streams" is misleading. Cut the claim, condense to 2-3 sentences, and mention the whitespace mode. OPENNLP-1933: Regex removal (5/10): Replace per-call String.split/matches/replaceAll patterns #1279 edits the same section, so coordinate the wording.

Minor

• BasicContextGenerator.java:35-36. "Since 3.0.0 the separator is not a regular expression … (OPENNLP-1929)" is version history in Javadoc. Remove it or reduce it to one line. Same at machine-learning.xml:91-93 ("In releases before 3.0.0 …").
• BasicContextGenerator.java:47-48. "as defined by {@link WhitespaceTokenizer}" hides the mode dependency. Link StringUtil#isWhitespace and name the property.
• BasicContextGenerator.java:83-95. Allocates an ArrayList and an array on every call. Add a fast path when indexOf(separator) == -1.
• ContextGenerator.java:24-29. No @return. Add it.
• BasicContextGeneratorTest.java:102-111. A single-value @ValueSource({""}) next to a separate null @Test. Merge into one @NullAndEmptySource test.
• BasicContextGeneratorTest.java:113-119. One test asserts both constructors. Split it or parameterize it.
• BasicContextGeneratorTest.java:33-66. Only "\\s" pins the regex break. Add "\\|", "\\.", Pattern.quote("|") and "\\s+" rows, so the incompatibility is deliberate and visible.
• BasicContextGeneratorTest.java:75-94. No mode-dependent rows (U+0085, U+001C) and no "a\nb" row. Add them, pinning WhitespaceMode with a reset in @AfterEach.
• This PR uses none of OPENNLP-1928: Regex removal (1/8): Replace trivial patterns with character scans in StringUtil #1275's helpers, and WhitespaceMode is already on main. Rebase onto main and drop the OPENNLP-1928: Regex removal (1/8): Replace trivial patterns with character scans in StringUtil #1275 commits.

Verified: the literal path is correct (1M fuzz inputs vs split(Pattern.quote(sep)) with empty parts removed: 0 differences). There is no in-repo caller of either constructor, opennlp-eval-tests included, so no eval build is needed; the risk is third-party code.

[rzo1]

Follow-up: the shared WhitespaceTokenizer.INSTANCE problem behind blocking item 1 is filed as OPENNLP-1947.

[krickert]

I did a break instead of a deprecated regex constructor. We shouldn't keep a Pattern in this class, I don't think it's justified for exclusion in the 10th regex guard ticket (#1282), and a shim does not stop the silent one-context result.

I think it's cleaner to have a separator with a backslash to throw at construction, so old escapes fails loudly.

43da7b5.