gh-79478: Update untokenize.rst documentation to reflect current implementation

[ByteFlowing1337]

• Issue: untokenize documentation is not correct #79478

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #32807385 | 📁 Comparing 27239f2 against main (1daad8a)

  🔍 Preview build  

1 file changed

± library/tokenize.html

[robsdedude]

I think the docs could be more explicit about whether the iterator may or may not mix 2- and 5-element sequences.

[ByteFlowing1337]

I added a para describing the behavior when they are mixed. :)

[ByteFlowing1337]

Thank you all for the review!

[ByteFlowing1337]

I replaced the word exactly that I introduced in my first commit.

The original docstring of untokenize once said: Round-trip invariant for full input: Untokenized source will match input source exactly, but that was inaccurate (see #128031), and it was updated to match the docs in #128521.

So even with a full input, the spacing between tokens (column positions) may still change, but it's a rare case.

---

I also replaced must with should , because I found that untokenize() accepts even three- or six-element sequences in iterable, only if the two-element sequence is encountered earlier in the iterable. It's also a rare case.