Merge pull request #2321 from sphinx-doc/py-3-highlighting

The default highlight language is now Python 3. Use it in viewcode as well.

CHANGES

@@ -26,6 +26,9 @@ Incompatible changes
   by ``termsep`` node. In new implementation, each terms are converted into individual
   ``term`` nodes and ``termsep`` node is removed.
   By this change, output layout of every builders are changed a bit.
+* The default highlight language is now Python 3.  This means that source code
+  is highlighted as Python 3 (which is mostly a superset of Python 2), and no
+  parsing is attempted to distinguish valid code.
 
 Features added
 --------------

doc/config.rst

@@ -320,11 +320,16 @@ Project information
 .. confval:: highlight_language
 
    The default language to highlight source code in.  The default is
-   ``'python'``.  The value should be a valid Pygments lexer name, see
+   ``'python3'``.  The value should be a valid Pygments lexer name, see
    :ref:`code-examples` for more details.
 
    .. versionadded:: 0.5
 
+   .. versionchanged:: 1.4
+      The default is now ``'python3'``, since it is mostly a superset of
+      ``'python'``.  If you prefer Python 2 only highlighting, you can set
+      it back to ``'python'``.
+
 .. confval:: highlight_options
 
    A dictionary of options that modify how the lexer specified by

doc/ext/napoleon.rst

@@ -11,6 +11,8 @@
 Napoleon - *Marching toward legible docstrings*
 ===============================================
 
+.. highlight:: text
+
 Are you tired of writing docstrings that look like this::
 
     :param path: The path of the file to wrap
@@ -124,7 +126,9 @@ Napoleon supports two styles of docstrings: `Google`_ and `NumPy`_. The
 main difference between the two styles is that Google uses indention to
 separate sections, whereas NumPy uses underlines.
 
-Google style::
+Google style:
+
+.. code-block:: python3
 
     def func(arg1, arg2):
         """Summary line.
@@ -141,7 +145,9 @@ Google style::
         """
         return True
 
-NumPy style::
+NumPy style:
+
+.. code-block:: python3
 
     def func(arg1, arg2):
         """Summary line.

doc/faq.rst

@@ -213,7 +213,7 @@ If you want Emacs to display Info files produced by Sphinx using the value
 Info files, try adding the following Emacs Lisp code to your start-up file,
 ``~/.emacs.d/init.el``.
 
-::
+.. code-block:: elisp
 
    (defadvice info-insert-file-contents (after
                                          sphinx-info-insert-file-contents

doc/intl.rst

@@ -88,14 +88,18 @@ This section describe a easy way to translate with sphinx-intl.
    :confval:`gettext_compact` is set to `False` (the Sphinx document is
    already configured as such).
 
-#. Extract document's translatable messages into pot files::
+#. Extract document's translatable messages into pot files:
+
+   .. code-block:: console
 
       $ make gettext
 
    As a result, many pot files are generated under ``_build/locale``
    directory.
 
-#. Setup/Update your `locale_dir`::
+#. Setup/Update your `locale_dir`:
+
+   .. code-block:: console
 
       $ sphinx-intl update -p _build/locale -l de -l ja
 
@@ -109,7 +113,9 @@ This section describe a easy way to translate with sphinx-intl.
 #. make translated document.
 
    You need a :confval:`language` parameter in ``conf.py`` or you may also
-   specify the parameter on the command line::
+   specify the parameter on the command line:
+
+   .. code-block:: console
 
       $ make -e SPHINXOPTS="-D language='de'" html
 
@@ -163,7 +169,7 @@ and to apply differences to translated po files.
 In order to apply the updating difference of a pot file to po file,
 use the :command:`sphinx-intl update` command.
 
-.. code-block:: bash
+.. code-block:: console
 
    $ sphinx-intl update -p _build/locale
 
@@ -182,7 +188,7 @@ easy to fetch and push translations.
 
    You need :command:`tx` command to upload resources (pot files).
 
-   .. code-block:: bash
+   .. code-block:: console
 
       $ pip install transifex-client
 
@@ -206,7 +212,7 @@ easy to fetch and push translations.
    This process will create ``.tx/config`` in the current directory, as well as
    a ``~/.transifexrc`` file that includes auth information.
 
-   .. code-block:: bash
+   .. code-block:: console
 
       $ tx init
       Creating .tx folder...
@@ -221,7 +227,7 @@ easy to fetch and push translations.
 
    Register pot files to ``.tx/config`` file:
 
-   .. code-block:: bash
+   .. code-block:: console
 
       $ cd /your/document/root
       $ sphinx-intl update-txconfig-resources --pot-dir _build/locale \
@@ -229,7 +235,7 @@ easy to fetch and push translations.
 
    and upload pot files:
 
-   .. code-block:: bash
+   .. code-block:: console
 
       $ tx push -s
       Pushing translations for resource sphinx-document-test_1_0.builders:
@@ -248,7 +254,7 @@ easy to fetch and push translations.
 
    Get translated catalogs and build mo files (ex. for 'de'):
 
-   .. code-block:: bash
+   .. code-block:: console
 
       $ cd /your/document/root
       $ tx pull -l de
@@ -257,7 +263,9 @@ easy to fetch and push translations.
       ...
       Done.
 
-   Invoke make html::
+   Invoke make html:
+
+   .. code-block:: console
 
       $ make -e SPHINXOPTS="-D language='de'" html
 

doc/invocation.rst

@@ -6,7 +6,9 @@ Invocation of sphinx-quickstart
 ===============================
 
 The :program:`sphinx-quickstart` script generates a Sphinx documentation set.
-It is called like this::
+It is called like this:
+
+   .. code-block:: console
 
      $ sphinx-quickstart [options] [projectdir]
 
@@ -141,9 +143,11 @@ Invocation of sphinx-build
 ==========================
 
 The :program:`sphinx-build` script builds a Sphinx documentation set.  It is
-called like this::
+called like this:
 
-     $ sphinx-build [options] sourcedir builddir [filenames]
+.. code-block:: console
+
+    $ sphinx-build [options] sourcedir builddir [filenames]
 
 where *sourcedir* is the :term:`source directory`, and *builddir* is the
 directory in which you want to place the built documentation.  Most of the time,
@@ -380,9 +384,11 @@ Invocation of sphinx-apidoc
 ===========================
 
 The :program:`sphinx-apidoc` generates completely automatic API documentation
-for a Python package.  It is called like this::
+for a Python package.  It is called like this:
 
-     $ sphinx-apidoc [options] -o outputdir packagedir [pathnames]
+.. code-block:: console
+
+    $ sphinx-apidoc [options] -o outputdir packagedir [pathnames]
 
 where *packagedir* is the path to the package to document, and *outputdir* is
 the directory where the generated sources are placed.  Any *pathnames* given

sphinx/config.py

@@ -65,7 +65,7 @@ class Config(object):
         trim_footnote_reference_space = (False, 'env'),
         show_authors = (False, 'env'),
         pygments_style = (None, 'html', [str]),
-        highlight_language = ('python', 'env'),
+        highlight_language = ('python3', 'env'),
         highlight_options = ({}, 'env'),
         templates_path = ([], 'html'),
         template_bridge = (None, 'html', [str]),

sphinx/ext/viewcode.py

@@ -139,7 +139,11 @@ def collect_pages(app):
         # construct a page name for the highlighted source
         pagename = '_modules/' + modname.replace('.', '/')
         # highlight the source using the builder's highlighter
-        highlighted = highlighter.highlight_block(code, 'python', linenos=False)
+        if env.config.highlight_language == 'python3':
+            lexer = 'python3'
+        else:
+            lexer = 'python'
+        highlighted = highlighter.highlight_block(code, lexer, linenos=False)
         # split the code into lines
         lines = highlighted.splitlines()
         # split off wrap markup from the first line of the actual code

sphinx/highlighting.py

@@ -25,8 +25,8 @@ from sphinx.util.texescape import tex_hl_escape_map_new
 from sphinx.ext import doctest
 
 from pygments import highlight
-from pygments.lexers import PythonLexer, PythonConsoleLexer, CLexer, \
-    TextLexer, RstLexer
+from pygments.lexers import PythonLexer, Python3Lexer, PythonConsoleLexer, \
+    CLexer, TextLexer, RstLexer
 from pygments.lexers import get_lexer_by_name, guess_lexer
 from pygments.formatters import HtmlFormatter, LatexFormatter
 from pygments.filters import ErrorToken
@@ -37,6 +37,7 @@ from sphinx.pygments_styles import SphinxStyle, NoneStyle
 lexers = dict(
     none = TextLexer(stripnl=False),
     python = PythonLexer(stripnl=False),
+    python3 = Python3Lexer(stripnl=False),
     pycon = PythonConsoleLexer(stripnl=False),
     pycon3 = PythonConsoleLexer(python3=True, stripnl=False),
     rest = RstLexer(stripnl=False),
@@ -151,9 +152,11 @@ class PygmentsBridge(object):
                     lexer = lexers['none']
             else:
                 lexer = lexers['python']
-        elif lang in ('python3', 'py3') and source.startswith('>>>'):
-            # for py3, recognize interactive sessions, but do not try parsing...
-            lexer = lexers['pycon3']
+        elif lang in ('py3', 'python3'):
+            if source.startswith('>>>'):
+                lexer = lexers['pycon3']
+            else:
+                lexer = lexers['python3']
         elif lang == 'guess':
             try:
                 lexer = guess_lexer(source)
@@ -187,7 +190,8 @@ class PygmentsBridge(object):
             # this is most probably not the selected language,
             # so let it pass unhighlighted
             if warn:
-                warn('Could not parse literal_block as "%s". highlighting skipped.' % lang)
+                warn('Could not lex literal_block as "%s". '
+                     'Highlighting skipped.' % lang)
             else:
                 raise exc
             hlsource = highlight(source, lexers['none'], formatter)

tests/root/includes.txt

@@ -17,10 +17,12 @@ Test file and literal inclusion
 
 .. should give a warning
 .. literalinclude:: wrongenc.inc
+   :language: none
 
 .. should succeed
 .. literalinclude:: wrongenc.inc
    :encoding: latin-1
+   :language: none
 .. include:: wrongenc.inc
    :encoding: latin-1
 

tests/root/markup.txt

@@ -23,6 +23,8 @@ Generic reST
 
 A |subst| (the definition is in rst_epilog).
 
+.. highlight:: none
+
 .. _label:
 
 ::

tests/test_build_html.py

@@ -31,16 +31,16 @@ http://www.python.org/logo.png
 reading included file u'.*?wrongenc.inc' seems to be wrong, try giving an \
 :encoding: option\\n?
 %(root)s/includes.txt:4: WARNING: download file not readable: .*?nonexisting.png
-(%(root)s/markup.txt:357: WARNING: invalid single index entry u'')?
+(%(root)s/markup.txt:359: WARNING: invalid single index entry u'')?
 (%(root)s/undecodable.txt:3: WARNING: undecodable source characters, replacing \
 with "\\?": b?'here: >>>(\\\\|/)xbb<<<'
 )?"""
 
 HTML_WARNINGS = ENV_WARNINGS + """\
 %(root)s/images.txt:20: WARNING: no matching candidate for image URI u'foo.\\*'
-%(root)s/markup.txt:269: WARNING: Could not parse literal_block as "c". highlighting skipped.
+%(root)s/markup.txt:271: WARNING: Could not lex literal_block as "c". Highlighting skipped.
 %(root)s/footnote.txt:60: WARNING: citation not found: missing
-%(root)s/markup.txt:158: WARNING: unknown option: &option
+%(root)s/markup.txt:160: WARNING: unknown option: &option
 """
 
 if PY3:

tests/test_build_latex.py

@@ -24,10 +24,10 @@ from test_build_html import ENV_WARNINGS
 
 
 LATEX_WARNINGS = ENV_WARNINGS + """\
-%(root)s/markup.txt:158: WARNING: unknown option: &option
+%(root)s/markup.txt:160: WARNING: unknown option: &option
 %(root)s/footnote.txt:60: WARNING: citation not found: missing
 %(root)s/images.txt:20: WARNING: no matching candidate for image URI u'foo.\\*'
-%(root)s/markup.txt:269: WARNING: Could not parse literal_block as "c". highlighting skipped.
+%(root)s/markup.txt:271: WARNING: Could not lex literal_block as "c". Highlighting skipped.
 """
 
 if PY3:

tests/test_build_texinfo.py

@@ -23,7 +23,7 @@ from test_build_html import ENV_WARNINGS
 
 
 TEXINFO_WARNINGS = ENV_WARNINGS + """\
-%(root)s/markup.txt:158: WARNING: unknown option: &option
+%(root)s/markup.txt:160: WARNING: unknown option: &option
 %(root)s/footnote.txt:60: WARNING: citation not found: missing
 %(root)s/images.txt:20: WARNING: no matching candidate for image URI u'foo.\\*'
 %(root)s/images.txt:29: WARNING: no matching candidate for image URI u'svgimg.\\*'