[SPARK-57932][SQL][3.5] Fix regexp_instr and regexp_replace returning wrong results for supplementary characters

[sarutak]

What changes were proposed in this pull request?

This PR backports SPARK-57932 (#56998) for branch-3.5 with conflict resolution.

regexp_instr returned matcher.start() + 1 (a UTF-16 code-unit index), and regexp_replace's position argument was used as a UTF-16 offset in matcher.region(pos-1, ...) and its out-of-range guard. Both now use code-point positions via codePointCount/offsetByCodePoints.

Why are the changes needed?

Spark string positions are 1-based code points (instr, locate, position, substring, length). For strings containing supplementary characters (code points > U+FFFF, e.g. emojis), these two functions gave wrong results:

• regexp_instr('😀ab', 'ab') returned 3 instead of 2.
• regexp_replace('😀aXa', 'a', 'Z', 3) returned 😀ZXZ instead of 😀aXZ (replaced a match before position); regexp_replace('😀😀', '😀', 'Z', 3) returned 😀Z instead of the input unchanged. This is the sibling of the LIKE fix in SPARK-55453.

Does this PR introduce any user-facing change?

Yes. regexp_instr and regexp_replace now return code-point-consistent results for strings with supplementary characters.

How was this patch tested?

Added UT.

Was this patch authored or co-authored using generative AI tooling?

Yes.

Stale.

[dongjoon-hyun]

@sarutak . When @holdenk proposed to extend branch-3.5, we decided to allow only security patches. So, this seems to be improper to me.

[dongjoon-hyun]

Please correct me if I'm wrong, @holdenk .

[sarutak]

Oh, I see. I'm OK to close this. Let me wait for @holdenk's reaction for a while.

[sarutak]

Will close as the versioning policy clearly states as follows.

During the 3.5.x extended LTS period, we will only include security fixes.