untokenize documentation is not correct

[csernazs]

BPO	35297
Nosy	@csernazs, @utkarsh2102, @cdonovick

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-11-22.21:25:02.196>
labels = ['3.7', 'docs']
title = 'untokenize documentation is not correct'
updated_at = <Date 2019-10-16.20:26:13.785>
user = 'https://github.com/csernazs'

bugs.python.org fields:

activity = <Date 2019-10-16.20:26:13.785>
actor = 'Zachary McCord'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2018-11-22.21:25:02.196>
creator = 'csernazs'
dependencies = []
files = []
hgrepos = []
issue_num = 35297
keywords = []
message_count = 3.0
messages = ['330281', '340469', '354816']
nosy_count = 5.0
nosy_names = ['csernazs', 'docs@python', 'utkarsh2102', 'donovick', 'Zachary McCord']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue35297'
versions = ['Python 3.6', 'Python 3.7']

Linked PRs

• gh-79478: Update untokenize.rst documentation to reflect current implementation #150225

[csernazs]

untokenize documentation (https://docs.python.org/3/library/tokenize.html#tokenize.untokenize) states the following:

"""
Converts tokens back into Python source code. The iterable must return sequences with at least two elements, the token type and the token string. Any additional sequence elements are ignored.
"""

This last sentence is clearly not true because here:
https://github.com/python/cpython/blob/master/Lib/tokenize.py#L242

The code checks for the length of the input token there, and the code behaves differently, in terms of whitespace, when an iterator of 2-tuples are given and when there are more elements in the tuple. When there are more elements in the tuple, the function renders whitespaces as the same as they were present in the original source.

So this code:
tokenize.untokenize(tokenize.tokenize(source.readline))

And this:
tokenize.untokenize([x[:2] for x in tokenize.tokenize(source.readline)])

Have different results.

I don't know that it is a documentation issue or a bug in the module itself, so I created this bugreport to seek for assistance in this regard.

[utkarsh2102]

I am not sure if that's a documentation problem, is it?
If so, I'll be happy to send a PR :)

[ZacharyMcCord]

I think anyone using the tokenize module to programmatically edit python source wants to use and probably does use the undocumented behavior, which should then be documented.

I ran into this issue because for me this manifested as a crash:

$ python3
>>> import tokenize
>>> tokenize.untokenize([(tokenize.STRING, "''", (1, 0), (1, 0), None)])
"''"
>>> tokenize.untokenize([(tokenize.STRING, "''", None, None, None)])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/<snip>/virtualenv/lib/python3.6/tokenize.py", line 338, in untokenize
    out = ut.untokenize(iterable)
  File "/<snip>/virtualenv/lib/python3.6/tokenize.py", line 272, in untokenize
    self.add_whitespace(start)
  File "/<snip>/virtualenv/lib/python3.6/tokenize.py", line 231, in add_whitespace
    row, col = start
TypeError: 'NoneType' object is not iterable

The second call is giving untokenize() input that is documented to be valid, yet which causes a crash.