IntenseWebs/sphinx
3
0
Fork 0
mirror of https://github.com/sphinx-doc/sphinx.git synced 2025-02-25 18:55:22 -06:00
Code Issues Packages Projects Releases Wiki Activity

Fix: pgen2 tokenizer doesn't recognize ... literal (Ellipsis for py3). Closes #1547

Browse Source 
I think pgen2 code derived from lib2to3 package. Basically, the package only support python2 code then it doesn't recognize `...` literal.
This commit is contained in:
Takayuki Shimizukawa 2014-08-28 22:41:52 +09:00
parent 13bbf44d87
commit f190de75cd
2 changed files with 10 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
CHANGES
View File

@ -215,6 +215,7 @@ Bugs fixed
:rst:dir:`c:function`. Thanks to Takeshi Komiya. :rst:dir:`c:function`. Thanks to Takeshi Komiya.
* PR#278: Fix section entries were shown twice if toctree has been put under * PR#278: Fix section entries were shown twice if toctree has been put under
only directive. Thanks to Takeshi Komiya. only directive. Thanks to Takeshi Komiya.
* #1547: pgen2 tokenizer doesn't recognize `...` literal (Ellipsis for py3).
Documentation Documentation
------------- -------------

11
sphinx/pycode/pgen2/tokenize.py
View File

@ -33,6 +33,7 @@ __credits__ = \
'GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip Montanaro' 'GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip Montanaro'
import string, re import string, re
from six import PY3
from sphinx.pycode.pgen2.token import * from sphinx.pycode.pgen2.token import *
from sphinx.pycode.pgen2 import token from sphinx.pycode.pgen2 import token
@ -84,6 +85,9 @@ Operator = group(r"\*\*=?", r">>=?", r"<<=?", r"<>", r"!=",
Bracket = '[][(){}]' Bracket = '[][(){}]'
Special = group(r'\r?\n', r'[:;.,`@]') Special = group(r'\r?\n', r'[:;.,`@]')
if PY3:
Ellipsis_ = r'\.{3}'
Special = group(Ellipsis_, Special)
Funny = group(Operator, Bracket, Special) Funny = group(Operator, Bracket, Special)
PlainToken = group(Number, Funny, String, Name) PlainToken = group(Number, Funny, String, Name)
@ -356,8 +360,9 @@ def generate_tokens(readline):
spos, epos, pos = (lnum, start), (lnum, end), end spos, epos, pos = (lnum, start), (lnum, end), end
token, initial = line[start:end], line[start] token, initial = line[start:end], line[start]
if initial in numchars or \ if initial in numchars or (
(initial == '.' and token != '.'): # ordinary number initial == '.' and token not in ('.', '...')
): # ordinary number
yield (NUMBER, token, spos, epos, line) yield (NUMBER, token, spos, epos, line)
elif initial in '\r\n': elif initial in '\r\n':
newline = NEWLINE newline = NEWLINE
@ -393,6 +398,8 @@ def generate_tokens(readline):
yield (STRING, token, spos, epos, line) yield (STRING, token, spos, epos, line)
elif initial in namechars: # ordinary name elif initial in namechars: # ordinary name
yield (NAME, token, spos, epos, line) yield (NAME, token, spos, epos, line)
elif token in ('...',): # ordinary name
yield (NAME, token, spos, epos, line)
elif initial == '\\': # continued stmt elif initial == '\\': # continued stmt
# This yield is new; needed for better idempotency: # This yield is new; needed for better idempotency:
yield (NL, token, spos, (lnum, pos), line) yield (NL, token, spos, (lnum, pos), line)