OPENNLP-1975: Regex removal (9/10): Correct the output of TokenSampleStream, ParserTool, ConlluStream, DownloadUtil and FeatureGeneratorUtil

[rzo1]

Part 9 of the regex removal (OPENNLP-1926). Split out of #1275. Based on main; part 1 (#1311, OPENNLP-1928) is merged.

These sites lose their regular expressions like the rest of the epic, but the scans follow what the code was meant to do instead of copying the regex quirks, so the output changes. Each corrected case is pinned by a test.

Behavior changes

TokenSampleStream (English). Tokens are split on runs of Unicode White_Space (StringUtil.splitOnUnicodeWhitespace), independent of the whitespace mode; before, split("\\s+") split on ASCII whitespace only. A leading space no longer yields an empty first token with the span (0,0). A line without a token resets the open-quote state. A token holding a letter or digit of any script is a word; before, [A-Za-z0-9] made non-ASCII words attach to the previous token as punctuation.

ParserTool. Brackets that follow each other are each separated: a))b gives a ) ) b, before a ) )b. A line without a bracket is returned unchanged. Whitespace follows the toolkit whitespace mode. Documented in the parser chapter.

ConlluStream. A multiword token id must be two runs of ASCII digits without a leading zero, joined by one hyphen, with the end not smaller than the start. Any other id fails with an InvalidFormatException naming the id. Before, 3- threw ArrayIndexOutOfBoundsException, -4 and 3--4 threw NumberFormatException, and 4-3 and 1-2-3 were read without an error. The text_xx language code is read with a scan, same output. New CoNLL-U section in the corpora chapter.

DownloadUtil. A blank checksum file is an IOException saying so; before, the model was rejected with "Expected: null". The index page links are read with a scan, same output.

FeatureGeneratorUtil. The cp class needs one capital and one period and no other character. A.\n was cp and is ic now. No other token changes class.

Before merge

• Tokenizer eval build that reads through TokenSampleStream.

Commits by @krickert from #1275, regrouped.

[krickert]

eval should read its samples through opennlp.tools.tokenize.lang.en.TokenSampleStream, since that is the class changed here. I couldn’t find an existing eval using it, so we should (w/the command, corpus, and before/after etc etc etc..)

I don't mind doing this it's a quick task.

[krickert]

I don't mind doing the changes - if they're not in by tomorrow I'll push it :)

1. HTML scanning
2. Unicode capitalization
3. Eval tokenizer stuff

[krickert]

Actually, I'll just fix it :) I see it's not a fork.

[krickert]

Made all the changes requested from my own review. Ready for a review @rzo1 and @mawiesne

[krickert]

I had to merge since main had corpora.xml touched since. I'll run the eval once the other eval is done in about 20 minutes.