Merge branch 'master' into 4976_location_for_logger.info

.travis.yml

@@ -30,7 +30,7 @@ matrix:
       env: TOXENV=docs
     - python: '3.6'
       env: TOXENV=mypy
-    - python: '2.7'
+    - python: '3.6'
       env: TOXENV=flake8
 
 install:

CHANGES

@@ -4,6 +4,11 @@ Release 1.8.0 (in development)
 Dependencies
 ------------
 
+* LaTeX: :confval:`latex_use_xindy`, if ``True`` (default for
+  ``xelatex/lualatex``), instructs ``make latexpdf`` to use :program:`xindy`
+  for general index.  Make sure your LaTeX distribution includes it.
+  (refs: #5134)
+
 Incompatible changes
 --------------------
 
@@ -39,6 +44,10 @@ Incompatible changes
   Please use ``app.add_js_file()`` instead.
 * #5072: Save environment object also with only new documents
 * #5035: qthelp builder allows dashes in :confval:`qthelp_namespace`
+* LaTeX: with lualatex or xelatex use by default :program:`xindy` as
+  UTF-8 able replacement of :program:`makeindex` (refs: #5134).  After
+  upgrading Sphinx, please clean latex build repertory of existing project
+  before new build.
 
 Deprecated
 ----------
@@ -143,6 +152,8 @@ Features added
   for mathjax
 * #4362: latex: Don't overwrite .tex file if document not changed
 * #1431: latex: Add alphanumeric enumerated list support
+* Add :confval:`latex_use_xindy` for UTF-8 savvy indexing, defaults to ``True``
+  if :confval:`latex_engine` is ``'xelatex'`` or ``'lualatex'``.
 * #4976: ``SphinxLoggerAdapter.info()`` now supports ``location`` parameter
 
 Bugs fixed
@@ -153,6 +164,8 @@ Bugs fixed
 * #4945: i18n: fix lang_COUNTRY not fallback correctly for IndexBuilder. Thanks
   to Shengjing Zhu.
 * #4983: productionlist directive generates invalid IDs for the tokens
+* #5132: (lualatex) PDF build fails if indexed word starts with Unicode character
+* #5133: latex: index headings "Symbols" and "Numbers" not internationalized
 
 Testing
 --------
@@ -203,6 +216,15 @@ Bugs fixed
 * #5091: latex: curly braces in index entries are not handled correctly
 * #5070: epub: Wrong internal href fragment links
 * #5104: apidoc: Interface of ``sphinx.apidoc:main()`` has changed
+* #5076: napoleon raises RuntimeError with python 3.7
+* #5125: sphinx-build: Interface of ``sphinx:main()`` has changed
+* sphinx-build: ``sphinx.cmd.build.main()`` refers ``sys.argv`` instead of given
+  argument
+* #5146: autosummary: warning is emitted when the first line of docstring ends
+  with literal notation
+* autosummary: warnings of autosummary indicates wrong location (refs: #5146)
+* #5143: autodoc: crashed on inspecting dict like object which does not support
+  sorting
 
 Testing
 --------

Makefile

@@ -1,4 +1,4 @@
-PYTHON ?= python
+PYTHON ?= python3
 
 .PHONY: all
 all: clean-pyc clean-backupfiles style-check type-check test

doc/Makefile

@@ -1,6 +1,6 @@
 # Makefile for Sphinx documentation
 #
-PYTHON ?= python
+PYTHON ?= python3
 
 # You can set these variables from the command line.
 SPHINXOPTS   =

doc/conf.py

@@ -72,6 +72,7 @@ latex_elements = {
 ''',
 }
 latex_show_urls = 'footnote'
+latex_use_xindy = True
 
 autodoc_member_order = 'groupwise'
 todo_include_todos = True

doc/contents.rst

@@ -12,16 +12,16 @@ Sphinx documentation contents
    usage/restructuredtext/index
    usage/markdown
    usage/configuration
+   usage/builders/index
+   usage/extensions/index
 
    intro
    man/index
-   builders
    intl
    theming
    setuptools
    templating
    latex
-   extensions
    extdev/index
    websupport
 

doc/ext/builtins.rst

@@ -1,25 +0,0 @@
-Builtin Sphinx extensions
--------------------------
-
-These extensions are built in and can be activated by respective entries in the
-:confval:`extensions` configuration value:
-
-.. toctree::
-
-   autodoc
-   autosectionlabel
-   autosummary
-   coverage
-   doctest
-   extlinks
-   githubpages
-   graphviz
-   ifconfig
-   imgconverter
-   inheritance
-   intersphinx
-   linkcode
-   math
-   napoleon
-   todo
-   viewcode

doc/extdev/parserapi.rst

@@ -12,7 +12,7 @@ __ http://docutils.sourceforge.net/docs/dev/hacking.html#parsing-the-document
 
 In Sphinx, the parser modules works as same as docutils.  The parsers are
 registered to Sphinx by extensions using Application APIs;
-:meth:`Sphinx.add_source_suffix()` and :meth:`Sphinx.add_source_parsers()`.
+:meth:`.Sphinx.add_source_suffix()` and :meth:`.Sphinx.add_source_parsers()`.
 
 The *source suffix* is a mapping from file suffix to file type.  For example,
 ``.rst`` file is mapped to ``'restructuredtext'`` type.  Sphinx uses the

doc/extensions.rst

@@ -1,16 +0,0 @@
-.. _extensions:
-
-Sphinx Extensions
-=================
-
-Since many projects will need special features in their documentation, Sphinx
-allows adding "extensions" to the build process, each of which can modify almost
-any aspect of document processing.
-
-This chapter describes the extensions bundled with Sphinx.  For the API
-documentation on writing your own extension, see :ref:`dev-extensions`.
-
-.. toctree::
-
-   ext/builtins
-   ext/thirdparty

doc/glossary.rst

@@ -12,7 +12,8 @@ Glossary
       use the builder builders that e.g. check for broken links in the
       documentation, or build coverage information.
 
-      See :ref:`builders` for an overview over Sphinx's built-in builders.
+      See :doc:`/usage/builders/index` for an overview over Sphinx's built-in
+      builders.
 
    configuration directory
       The directory containing :file:`conf.py`.  By default, this is the same as
@@ -66,6 +67,12 @@ Glossary
       parsing stage, so that successive runs only need to read and parse new and
       changed documents.
 
+   extension
+     A custom :term:`role`, :term:`directive` or other aspect of Sphinx that
+     allows users to modify any aspect of the build process within Sphinx.
+
+     For more information, refer to :doc:`/usage/extensions/index`.
+
    master document
       The document that contains the root :rst:dir:`toctree` directive.
 

doc/man/sphinx-build.rst

@@ -86,8 +86,8 @@ Options
      Build compact pretty-printed "pseudo-XML" files displaying the
      internal structure of the intermediate document trees.
 
-   See :ref:`builders` for a list of all builders shipped with Sphinx.
+   See :doc:`/usage/builders/index` for a list of all builders shipped with
-   Extensions can add their own builders.
+   Sphinx.  Extensions can add their own builders.
 
 .. _make_mode:
 
@@ -96,7 +96,7 @@ Options
    Alternative to :option:`-b`. Uses the Sphinx :program:`make_mode` module,
    which provides the same build functionality as a default :ref:`Makefile or
    Make.bat <makefile_options>`. In addition to all Sphinx
-   :ref:`builders <builders>`, the following build pipelines are available:
+   :doc:`/usage/builders/index`, the following build pipelines are available:
 
    **latexpdf**
      Build LaTeX files and run them through :program:`pdflatex`, or as per

doc/usage/builders/index.rst

@@ -1,13 +1,14 @@
 .. _builders:
 
-Available builders
+========
-==================
+Builders
+========
 
 .. module:: sphinx.builders
    :synopsis: Available built-in builder classes.
 
 These are the built-in Sphinx builders.  More builders can be added by
-:ref:`extensions <extensions>`.
+:doc:`extensions </usage/extensions/index>`.
 
 The builder's "name" must be given to the **-b** command-line option of
 :program:`sphinx-build` to select a builder.
@@ -75,8 +76,8 @@ The builder's "name" must be given to the **-b** command-line option of
 .. class:: QtHelpBuilder
 
    This builder produces the same output as the standalone HTML builder, but
-   also generates `Qt help`_ collection support files that allow
+   also generates `Qt help`_ collection support files that allow the Qt
-   the Qt collection generator to compile them.
+   collection generator to compile them.
 
    .. autoattribute:: name
 
@@ -144,7 +145,8 @@ The builder's "name" must be given to the **-b** command-line option of
 
    .. versionchanged:: 1.5
 
-      Since Sphinx-1.5, the epub3 builder is used for the default builder of epub.
+      Since Sphinx-1.5, the epub3 builder is used for the default builder of
+      epub.
 
 .. module:: sphinx.builders.latex
 .. class:: LaTeXBuilder
@@ -159,12 +161,12 @@ The builder's "name" must be given to the **-b** command-line option of
    in a "minimal" TeX distribution installation.  For example, on Ubuntu, the
    following packages need to be installed for successful PDF builds:
 
-   * texlive-latex-recommended
+   * ``texlive-latex-recommended``
-   * texlive-fonts-recommended
+   * ``texlive-fonts-recommended``
-   * texlive-latex-extra
+   * ``texlive-latex-extra``
-   * latexmk (for ``make latexpdf`` on GNU/Linux and MacOS X)
+   * ``latexmk`` (for ``make latexpdf`` on GNU/Linux and MacOS X)
-   * latex-xcolor (old Ubuntu)
+   * ``latex-xcolor`` (old Ubuntu)
-   * texlive-luatex, texlive-xetex (see :confval:`latex_engine`)
+   * ``texlive-luatex``, ``texlive-xetex`` (see :confval:`latex_engine`)
 
    The testing of Sphinx LaTeX is done on Ubuntu trusty with the above
    mentioned packages, which are from a TeXLive 2013 snapshot dated
@@ -178,7 +180,7 @@ The builder's "name" must be given to the **-b** command-line option of
       Since 1.6, ``make latexpdf`` uses ``latexmk`` (not on Windows).  This
       makes sure the needed number of runs is automatically executed to get
       the cross-references, bookmarks, indices, and tables of contents right.
-  
+
       One can pass to ``latexmk`` options via the ``LATEXMKOPTS``
       Makefile variable. For example:
 
@@ -311,7 +313,6 @@ name is ``rinoh``. Refer to the `rinohtype manual`_ for details.
 
       The filename for the search index Sphinx generates.
 
-
    See :ref:`serialization-details` for details about the output format.
 
    .. versionadded:: 0.5

doc/usage/configuration.rst

@@ -65,8 +65,9 @@ General configuration
 
 .. confval:: extensions
 
-   A list of strings that are module names of :ref:`extensions`. These can be
+   A list of strings that are module names of :doc:`extensions
-   extensions coming with Sphinx (named ``sphinx.ext.*``) or custom ones.
+   <extensions/index>`. These can be extensions coming with Sphinx (named
+   ``sphinx.ext.*``) or custom ones.
 
    Note that you can extend :data:`sys.path` within the conf file if your
    extensions live in another directory -- but make sure you use absolute paths.
@@ -142,7 +143,7 @@ General configuration
    .. versionadded:: 1.3
 
    .. deprecated:: 1.8
-      Now Sphinx provides an API :meth:`Sphinx.add_source_parser` to register
+      Now Sphinx provides an API :meth:`.Sphinx.add_source_parser` to register
       a source parser.  Please use it instead.
 
 .. confval:: master_doc
@@ -1871,6 +1872,40 @@ information.
 
    .. versionadded:: 1.6
 
+.. confval:: latex_use_xindy
+
+   If ``True``, the PDF build from the LaTeX files created by Sphinx
+   will use :program:`xindy` (doc__) rather than :program:`makeindex`
+   for preparing the index of general terms (from :rst:dir:`index`
+   usage).  This means that words with UTF-8 characters will get
+   ordered correctly for the :confval:`language`.
+
+   __ http://xindy.sourceforge.net/
+
+   - This option is ignored if :confval:`latex_engine` is ``'platex'``
+     (Japanese documents) as :program:`mendex` is used in that case.
+
+   - The default is ``True`` for ``'xelatex'`` or ``'lualatex'`` as
+     :program:`makeindex`, if any indexed term starts with a non-ascii
+     character, creates ``.ind`` file containing invalid bytes for
+     UTF-8 encoding. With ``'lualatex'`` this then breaks the PDF
+     build.  Notice that :program:`xindy` supports most but not
+     all European languages.
+
+   - The default is ``False`` for ``'pdflatex'`` but ``True`` is
+     recommended for non-English documents as soon as some indexed
+     terms use non-ascii characters from the language script.
+     Cyrillic scripts are (transparently) supported with
+     ``'pdflatex'`` thanks to a specific Sphinx-contributed ``xindy``
+     style file :file:`cyrLICRutf8.xdy`.
+
+     As :program:`xindy` does not support the same range of languages
+     as ``LaTeX/babel`` does, the default :program:`makeindex` for
+     ``'pdflatex'`` may be preferred in some circumstances, although
+     the index will be ill-formed probably.
+
+   .. versionadded:: 1.8
+
 .. confval:: latex_elements
 
    .. versionadded:: 0.5

doc/usage/extensions/autodoc.rst

@@ -405,6 +405,7 @@ There are also new config values that you can set:
 
    .. versionadded:: 1.7
 
+
 Docstring preprocessing
 -----------------------
 

doc/usage/extensions/autosectionlabel.rst

@@ -27,6 +27,7 @@ default. The ``autosectionlabel_prefix_document`` configuration variable can be
 used to make headings which appear multiple times but in different documents
 unique.
 
+
 Configuration
 -------------
 

doc/usage/extensions/autosummary.rst

@@ -25,7 +25,6 @@ The :mod:`sphinx.ext.autosummary` extension does this in two parts:
    These files by default contain only the corresponding
    :mod:`sphinx.ext.autodoc` directive, but can be customized with templates.
 
-
 .. rst:directive:: autosummary
 
    Insert a table that contains links to documented items, and a short summary
@@ -59,7 +58,6 @@ The :mod:`sphinx.ext.autosummary` extension does this in two parts:
    :event:`autodoc-process-docstring` and :event:`autodoc-process-signature`
    hooks as :mod:`~sphinx.ext.autodoc`.
 
-
    **Options**
 
    * If you want the :rst:dir:`autosummary` table to also serve as a

doc/usage/extensions/coverage.rst

@@ -4,7 +4,6 @@
 .. module:: sphinx.ext.coverage
    :synopsis: Check Python modules and C API for coverage in the documentation.
 
-
 This extension features one additional builder, the :class:`CoverageBuilder`.
 
 .. class:: CoverageBuilder
@@ -14,7 +13,6 @@ This extension features one additional builder, the :class:`CoverageBuilder`.
 
 .. todo:: Write this section.
 
-
 Several new configuration values can be used to specify what the builder
 should check:
 

doc/usage/extensions/doctest.rst

@@ -165,7 +165,6 @@ a comma-separated list of group names.
 
          Output text.
 
-
 The following is an example for the usage of the directives.  The test via
 :rst:dir:`doctest` and the test via :rst:dir:`testcode` and
 :rst:dir:`testoutput` are equivalent. ::

doc/usage/extensions/extlinks.rst

@@ -7,7 +7,6 @@
 
 .. versionadded:: 1.0
 
-
 This extension is meant to help with the common pattern of having many external
 links that point to URLs on one and the same site, e.g. links to bug trackers,
 version control web interfaces, or simply subpages in other websites.  It does

doc/usage/extensions/githubpages.rst

@@ -1,5 +1,3 @@
-.. highlight:: rest
-
 :mod:`sphinx.ext.githubpages` -- Publish HTML docs in GitHub Pages
 ==================================================================
 

doc/usage/extensions/graphviz.rst

@@ -13,7 +13,6 @@ your documents.
 
 It adds these directives:
 
-
 .. rst:directive:: graphviz
 
    Directive to embed graphviz code.  The input code for ``dot`` is given as the

doc/usage/extensions/imgconverter.rst

@@ -1,28 +1,30 @@
-.. highlight:: rest
-
 .. _sphinx.ext.imgconverter:
 
-:mod:`sphinx.ext.imgconverter` -- A reference implementation for image converter using Imagemagick
+:mod:`sphinx.ext.imgconverter` -- A reference image converter using Imagemagick
-==================================================================================================
+===============================================================================
 
 .. module:: sphinx.ext.imgconverter
    :synopsis: Convert images to appropriate format for builders
 
 .. versionadded:: 1.6
 
-This extension converts images in your document to appropriate format for builders.
+This extension converts images in your document to appropriate format for
-For example, it allows you to use SVG images with LaTeX builder.
+builders.  For example, it allows you to use SVG images with LaTeX builder.
 As a result, you don't mind what image format the builder supports.
 
 Internally, this extension uses Imagemagick_ to convert images.
 
 .. _Imagemagick: https://www.imagemagick.org/script/index.php
 
-.. note:: Imagemagick rasterizes a SVG image on conversion.  As a result, the image
+.. note::
-          becomes not scalable.  To avoid that, please use other image converters
+
-          like sphinxcontrib-svg2pdfconverter_ (which uses Inkscape or rsvg-convert).
+   Imagemagick rasterizes a SVG image on conversion.  As a result, the image
+   becomes not scalable.  To avoid that, please use other image converters like
+   `sphinxcontrib-svg2pdfconverter`__ (which uses Inkscape or
+   ``rsvg-convert``).
+
+.. __: https://github.com/missinglinkelectronics/sphinxcontrib-svg2pdfconverter
 
-.. _sphinxcontrib-svg2pdfconverter: https://github.com/missinglinkelectronics/sphinxcontrib-svg2pdfconverter
 
 Configuration
 -------------

doc/usage/extensions/index.rst

@@ -1,9 +1,50 @@
+==========
+Extensions
+==========
+
+Since many projects will need special features in their documentation, Sphinx
+allows adding "extensions" to the build process, each of which can modify
+almost any aspect of document processing.
+
+This chapter describes the extensions bundled with Sphinx.  For the API
+documentation on writing your own extension, refer to :ref:`dev-extensions`.
+
+
+Built-in extensions
+-------------------
+
+These extensions are built in and can be activated by respective entries in the
+:confval:`extensions` configuration value:
+
+.. toctree::
+
+   autodoc
+   autosectionlabel
+   autosummary
+   coverage
+   doctest
+   extlinks
+   githubpages
+   graphviz
+   ifconfig
+   imgconverter
+   inheritance
+   intersphinx
+   linkcode
+   math
+   napoleon
+   todo
+   viewcode
+
+
 Third-party extensions
 ----------------------
 
+.. todo:: This should reference the GitHub organization now
+
 You can find several extensions contributed by users in the `Sphinx Contrib`_
-repository.  It is open for anyone who wants to maintain an extension
+repository.  It is open for anyone who wants to maintain an extension publicly;
-publicly; just send a short message asking for write permissions.
+just send a short message asking for write permissions.
 
 There are also several extensions hosted elsewhere.  The `Sphinx extension
 survey <https://sphinxext-survey.readthedocs.io/>`__ and `awesome-sphinxdoc
@@ -16,15 +57,13 @@ list (`join here <https://groups.google.com/forum/#!forum/sphinx-dev>`_).
 
 .. _Sphinx Contrib: https://bitbucket.org/birkenfeld/sphinx-contrib
 
-
 Where to put your own extensions?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Extensions local to a project should be put within the project's directory
 structure.  Set Python's module search path, ``sys.path``, accordingly so that
-Sphinx can find them.
+Sphinx can find them.  For example, if your extension ``foo.py`` lies in the
-E.g., if your extension ``foo.py`` lies in the ``exts`` subdirectory of the
+``exts`` subdirectory of the project root, put into :file:`conf.py`::
-project root, put into :file:`conf.py`::
 
    import sys, os
 

doc/usage/extensions/inheritance.rst

@@ -91,7 +91,9 @@ It adds this directive:
    .. versionchanged:: 1.7
       Added ``top-classes`` option to limit the scope of inheritance graphs.
 
-New config values are:
+
+Configuration
+-------------
 
 .. confval:: inheritance_graph_attrs
 

doc/usage/extensions/intersphinx.rst

@@ -38,8 +38,9 @@ Behind the scenes, this works as follows:
   specified individually, e.g. if the docs should be buildable without Internet
   access.
 
-Configuring Intersphinx
+
------------------------
+Configuration
+-------------
 
 To use Intersphinx linking, add ``'sphinx.ext.intersphinx'`` to your
 :confval:`extensions` config value, and use these new config values to activate
@@ -136,12 +137,14 @@ linking:
       exception is raised if the server has not issued a response for timeout
       seconds.
 
+
 Showing all links of an Intersphinx mapping file
 ------------------------------------------------
 
-To show all Intersphinx links and their targets of an Intersphinx mapping file, run
+To show all Intersphinx links and their targets of an Intersphinx mapping file,
-``python -msphinx.ext.intersphinx url-or-path``.  This is helpful when searching for the root cause of a broken
+run ``python -msphinx.ext.intersphinx url-or-path``.  This is helpful when
-Intersphinx link in a documentation project. The following example prints the Intersphinx mapping of the Python 3
+searching for the root cause of a broken Intersphinx link in a documentation
+project. The following example prints the Intersphinx mapping of the Python 3
 documentation::
 
    $ python -msphinx.ext.intersphinx https://docs.python.org/3/objects.inv

doc/usage/extensions/linkcode.rst

@@ -16,6 +16,10 @@ found somewhere on the Internet.
 In your configuration, you need to specify a :confval:`linkcode_resolve`
 function that returns an URL based on the object.
 
+
+Configuration
+-------------
+
 .. confval:: linkcode_resolve
 
    This is a function ``linkcode_resolve(domain, info)``,

doc/usage/extensions/napoleon.rst

@@ -1,5 +1,5 @@
 :mod:`sphinx.ext.napoleon` -- Support for NumPy and Google style docstrings
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+===========================================================================
 
 .. module:: sphinx.ext.napoleon
    :synopsis: Support for NumPy and Google style docstrings
@@ -8,8 +8,8 @@
 
 .. versionadded:: 1.3
 
-Napoleon - *Marching toward legible docstrings*
+Overview
-===============================================
+--------
 
 .. highlight:: text
 
@@ -25,7 +25,7 @@ Are you tired of writing docstrings that look like this::
     :returns: A buffered writable file descriptor
     :rtype: BufferedFileStorage
 
-`ReStructuredText`_ is great, but it creates visually dense, hard to read
+`reStructuredText`_ is great, but it creates visually dense, hard to read
 `docstrings`_. Compare the jumble above to the same thing rewritten
 according to the `Google Python Style Guide`_::
 
@@ -40,8 +40,8 @@ according to the `Google Python Style Guide`_::
 
 Much more legible, no?
 
-Napoleon is a :doc:`../extensions` that enables Sphinx to parse both `NumPy`_
+Napoleon is a :term:`extension` that enables Sphinx to parse both `NumPy`_ and
-and `Google`_ style docstrings - the style recommended by `Khan Academy`_.
+`Google`_ style docstrings - the style recommended by `Khan Academy`_.
 
 Napoleon is a pre-processor that parses `NumPy`_ and `Google`_ style
 docstrings and converts them to reStructuredText before Sphinx attempts to
@@ -61,7 +61,7 @@ source code files.
    https://github.com/Khan/style-guides/blob/master/style/python.md#docstrings
 
 Getting Started
----------------
+~~~~~~~~~~~~~~~
 
 1. After :doc:`setting up Sphinx </usage/quickstart>` to build your docs,
    enable napoleon in the Sphinx `conf.py` file::
@@ -77,7 +77,7 @@ Getting Started
 
 
 Docstrings
-----------
+~~~~~~~~~~
 
 Napoleon interprets every docstring that :mod:`autodoc <sphinx.ext.autodoc>`
 can find, including docstrings on: ``modules``, ``classes``, ``attributes``,
@@ -91,7 +91,7 @@ All standard reStructuredText formatting still works as expected.
 .. _Sections:
 
 Docstring Sections
-------------------
+~~~~~~~~~~~~~~~~~~
 
 All of the following section headers are supported:
 
@@ -127,7 +127,7 @@ All of the following section headers are supported:
     * ``Yields``
 
 Google vs NumPy
----------------
+~~~~~~~~~~~~~~~
 
 Napoleon supports two styles of docstrings: `Google`_ and `NumPy`_. The
 main difference between the two styles is that Google uses indention to
@@ -195,7 +195,7 @@ not be mixed. Choose one style for your project and be consistent with it.
 
 
 Type Annotations
-----------------
+~~~~~~~~~~~~~~~~
 
 `PEP 484`_ introduced a standard way to express types in Python code.
 This is an alternative to expressing types directly in docstrings.
@@ -249,7 +249,7 @@ Google style with types in docstrings::
 
 
 Configuration
-=============
+-------------
 
 Listed below are all the settings used by napoleon and their default
 values. These settings can be changed in the Sphinx `conf.py` file. Make
@@ -278,8 +278,6 @@ sure that "sphinx.ext.napoleon" is enabled in `conf.py`::
 .. _NumPy style:
    https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt
 
-
-
 .. confval:: napoleon_google_docstring
 
    True to parse `Google style`_ docstrings. False to disable support

doc/usage/extensions/todo.rst

@@ -27,7 +27,10 @@ There are two additional directives when using this extension:
    documentation, if :confval:`todo_include_todos` is ``True``.
 
 
-There is also an additional config value:
+These can be configured as seen below.
+
+Configuration
+-------------
 
 .. confval:: todo_include_todos
 

doc/usage/extensions/viewcode.rst

@@ -7,9 +7,8 @@
 
 .. versionadded:: 1.0
 
-
+This extension looks at your Python object descriptions (``.. class::``, ``..
-This extension looks at your Python object descriptions (``.. class::``,
+function::`` etc.) and tries to find the source files where the objects are
-``.. function::`` etc.) and tries to find the source files where the objects are
 contained.  When found, a separate HTML page will be output for each module with
 a highlighted version of the source code, and a link will be added to all object
 descriptions that leads to the source code of the described object.  A link back
@@ -33,14 +32,15 @@ This extension works only on HTML related builders like ``html``,
 ``singlehtml``. By default ``epub`` builder doesn't
 support this extension (see :confval:`viewcode_enable_epub`).
 
-There is an additional config value:
+Configuration
+-------------
 
 .. confval:: viewcode_follow_imported_members
 
    If this is ``True``, viewcode extension will follow alias objects that
-   imported from another module such as functions, classes and attributes.
+   imported from another module such as functions, classes and attributes.  As
-   As side effects, this option
+   side effects, this option else they produce nothing.  The default is
-   else they produce nothing.  The default is ``True``.
+   ``True``.
 
    .. versionadded:: 1.3
 

doc/usage/quickstart.rst

@@ -127,7 +127,7 @@ directory in which you want to place the built documentation.
 The :option:`-b <sphinx-build -b>` option selects a builder; in this example
 Sphinx will build HTML files.
 
-|more| Refer to the :program:`sphinx-build man page <sphinx-build>` for all
+|more| Refer to the :doc:`sphinx-build man page </man/sphinx-build>` for all
 options that :program:`sphinx-build` supports.
 
 However, :program:`sphinx-quickstart` script creates a :file:`Makefile` and a
@@ -318,12 +318,7 @@ features of intersphinx.
 More topics to be covered
 -------------------------
 
-- :doc:`Other extensions </extensions>`:
+- :doc:`Other extensions </usage/extensions/index>`:
-
-  * :doc:`/ext/math`,
-  * :doc:`/ext/viewcode`,
-  * :doc:`/ext/doctest`,
-  * ...
 - Static files
 - :doc:`Selecting a theme </theming>`
 - :doc:`/setuptools`
@@ -336,7 +331,7 @@ More topics to be covered
 
 .. [#] This is the usual layout.  However, :file:`conf.py` can also live in
        another directory, the :term:`configuration directory`.  Refer to the
-       :program:`sphinx-build man page <sphinx-build>` for more information.
+       :doc:`sphinx-build man page </man/sphinx-build>` for more information.
 
 .. |more| image:: /_static/more.png
           :align: middle

doc/usage/restructuredtext/basics.rst

@@ -444,7 +444,7 @@ candidates.  For instance, if the file name ``gnu.*`` was given and two files
 :file:`gnu.pdf` and :file:`gnu.png` existed in the source tree, the LaTeX
 builder would choose the former, while the HTML builder would prefer the
 latter.  Supported image types and choosing priority are defined at
-:ref:`builders`.
+:doc:`/usage/builders/index`.
 
 Note that image file names should not contain spaces.
 

doc/usage/restructuredtext/roles.rst

@@ -62,7 +62,7 @@ Cross-referencing anything
      by :rst:role:`doc`, :rst:role:`ref` or :rst:role:`option`.
 
      Custom objects added to the standard domain by extensions (see
-     :meth:`Sphinx.add_object_type`) are also searched.
+     :meth:`.Sphinx.add_object_type`) are also searched.
 
    * Then, it looks for objects (targets) in all loaded domains.  It is up to
      the domains how specific a match must be.  For example, in the Python

setup.cfg

@@ -32,7 +32,7 @@ directory = sphinx/locale/
 [flake8]
 max-line-length = 95
 ignore = E116,E241,E251,E741,I101
-exclude = .git,.tox,.venv,tests/*,build/*,doc/_build/*,sphinx/search/*,doc/ext/example*.py
+exclude = .git,.tox,.venv,tests/*,build/*,doc/_build/*,sphinx/search/*,doc/usage/extensions/example*.py
 application-import-names = sphinx
 import-order-style = smarkets
 

sphinx/__init__.py

@@ -71,16 +71,16 @@ if __version__.endswith('+'):
         pass
 
 
-def main(*args, **kwargs):
+def main(argv=sys.argv):  # type: ignore
-    # type: (Any, Any) -> int
+    # type: (List[unicode]) -> int
     from .cmd import build
     warnings.warn(
         '`sphinx.main()` has moved to `sphinx.cmd.build.main()`.',
         RemovedInSphinx20Warning,
         stacklevel=2,
     )
-    args = args[1:]  # skip first argument to adjust arguments (refs: #4615)
+    argv = argv[1:]  # skip first argument to adjust arguments (refs: #4615)
-    return build.main(*args, **kwargs)
+    return build.main(argv)
 
 
 def build_main(argv=sys.argv):

sphinx/application.py

@@ -642,7 +642,7 @@ class Sphinx(object):
         a title.
 
         Other keyword arguments are used for node visitor functions. See the
-        :meth:`Sphinx.add_node` for details.
+        :meth:`.Sphinx.add_node` for details.
 
         .. versionadded:: 1.4
         """
@@ -850,8 +850,9 @@ class Sphinx(object):
           object.  It will automatically add index entries if *indextemplate*
           is nonempty; if given, it must contain exactly one instance of
           ``%s``.  See the example below for how the template will be
-          interpreted.  * Create a new role (called *rolename*) to
+          interpreted.
-          cross-reference to these object descriptions.
+        - Create a new role (called *rolename*) to cross-reference to these
+          object descriptions.
         - If you provide *parse_node*, it must be a function that takes a
           string and a docutils node, and it must populate the node with
           children parsed from the string.  It must then return the name of the

sphinx/builders/latex/__init__.py

@@ -44,6 +44,67 @@ if False:
     from sphinx.config import Config  # NOQA
 
 
+XINDY_LANG_OPTIONS = {
+    # language codes from docutils.writers.latex2e.Babel
+    # ! xindy language names may differ from those in use by LaTeX/babel
+    # ! xindy does not support all Latin scripts as recognized by LaTeX/babel
+    # ! not all xindy-supported languages appear in Babel.language_codes
+    # cd /usr/local/texlive/2018/texmf-dist/xindy/modules/lang
+    # find . -name '*utf8.xdy'
+    # LATIN
+    'sq': '-L albanian -C utf8 ',
+    'hr': '-L croatian -C utf8 ',
+    'cs': '-L czech -C utf8 ',
+    'da': '-L danish -C utf8 ',
+    'nl': '-L dutch -C ij-as-ij-utf8 ',
+    'en': '-L english -C utf8 ',
+    'eo': '-L esperanto -C utf8 ',
+    'et': '-L estonian -C utf8 ',
+    'fi': '-L finnish -C utf8 ',
+    'fr': '-L french -C utf8 ',
+    'de': '-L german -C din5007-utf8 ',
+    'is': '-L icelandic -C utf8 ',
+    'it': '-L italian -C utf8 ',
+    'la': '-L latin -C utf8 ',
+    'lv': '-L latvian -C utf8 ',
+    'lt': '-L lithuanian -C utf8 ',
+    'dsb': '-L lower-sorbian -C utf8 ',
+    'ds': '-L lower-sorbian -C utf8 ',   # trick, no conflict
+    'nb': '-L norwegian -C utf8 ',
+    'no': '-L norwegian -C utf8 ',       # and what about nynorsk?
+    'pl': '-L polish -C utf8 ',
+    'pt': '-L portuguese -C utf8 ',
+    'ro': '-L romanian -C utf8 ',
+    'sk': '-L slovak -C small-utf8 ',    # there is also slovak-large
+    'sl': '-L slovenian -C utf8 ',
+    'es': '-L spanish -C modern-utf8 ',  # there is also spanish-traditional
+    'sv': '-L swedish -C utf8 ',
+    'tr': '-L turkish -C utf8 ',
+    'hsb': '-L upper-sorbian -C utf8 ',
+    'hs': '-L upper-sorbian -C utf8 ',   # trick, no conflict
+    'vi': '-L vietnamese -C utf8 ',
+    # CYRILLIC
+    # for usage with pdflatex, needs also cyrLICRutf8.xdy module
+    'be': '-L belarusian -C utf8 ',
+    'bg': '-L bulgarian -C utf8 ',
+    'mk': '-L macedonian -C utf8 ',
+    'mn': '-L mongolian -C cyrillic-utf8 ',
+    'ru': '-L russian -C utf8 ',
+    'sr': '-L serbian -C utf8 ',
+    'sh-cyrl': '-L serbian -C utf8 ',
+    'sh': '-L serbian -C utf8 ',         # trick, no conflict
+    'uk': '-L ukrainian -C utf8 ',
+    # GREEK
+    # can work only with xelatex/lualatex, not supported by texindy+pdflatex
+    'el': '-L greek -C utf8 ',
+    # FIXME, not compatible with [:2] slice but does Sphinx support Greek ?
+    'el-polyton': '-L greek -C polytonic-utf8 ',
+}  # type: Dict[unicode, unicode]
+
+XINDY_CYRILLIC_SCRIPTS = [
+    'be', 'bg', 'mk', 'mn', 'ru', 'sr', 'sh', 'uk',
+]  # type: List[unicode]
+
 logger = logging.getLogger(__name__)
 
 
@@ -232,7 +293,23 @@ class LaTeXBuilder(Builder):
         self.copy_image_files()
 
         # copy TeX support files from texinputs
-        context = {'latex_engine': self.config.latex_engine}
+        # configure usage of xindy (impacts Makefile and latexmkrc)
+        # FIXME: convert this rather to a confval with suitable default
+        #        according to language ? but would require extra documentation
+        if self.config.language:
+            xindy_lang_option = \
+                XINDY_LANG_OPTIONS.get(self.config.language[:2],
+                                       '-L general -C utf8 ')
+            xindy_cyrillic = self.config.language[:2] in XINDY_CYRILLIC_SCRIPTS
+        else:
+            xindy_lang_option = '-L english -C utf8 '
+            xindy_cyrillic = False
+        context = {
+            'latex_engine':      self.config.latex_engine,
+            'xindy_use':         self.config.latex_use_xindy,
+            'xindy_lang_option': xindy_lang_option,
+            'xindy_cyrillic':    xindy_cyrillic,
+        }
         logger.info(bold(__('copying TeX support files...')))
         staticdirname = path.join(package_dir, 'texinputs')
         for filename in os.listdir(staticdirname):
@@ -317,6 +394,12 @@ def default_latex_docclass(config):
         return {}
 
 
+def default_latex_use_xindy(config):
+    # type: (Config) -> bool
+    """ Better default latex_use_xindy settings for specific engines. """
+    return config.latex_engine in {'xelatex', 'lualatex'}
+
+
 def setup(app):
     # type: (Sphinx) -> Dict[unicode, Any]
     app.add_builder(LaTeXBuilder)
@@ -334,6 +417,7 @@ def setup(app):
     app.add_config_value('latex_logo', None, None, string_classes)
     app.add_config_value('latex_appendices', [], None)
     app.add_config_value('latex_use_latex_multicolumn', False, None)
+    app.add_config_value('latex_use_xindy', default_latex_use_xindy, None)
     app.add_config_value('latex_toplevel_sectioning', None, None,
                          ENUM(None, 'part', 'chapter', 'section'))
     app.add_config_value('latex_domain_indices', True, None, [list])

sphinx/builders/qthelp.py

@@ -140,7 +140,7 @@ class QtHelpBuilder(StandaloneHTMLBuilder):
         else:
             nspace = 'org.sphinx.%s.%s' % (outname, self.config.version)
 
-        nspace = re.sub('[^a-zA-Z0-9.\-]', '', nspace)
+        nspace = re.sub(r'[^a-zA-Z0-9.\-]', '', nspace)
         nspace = re.sub(r'\.+', '.', nspace).strip('.')
         nspace = nspace.lower()
 

sphinx/cmd/build.py

@@ -311,7 +311,7 @@ def main(argv=sys.argv[1:]):  # type: ignore
     locale.setlocale(locale.LC_ALL, '')
     sphinx.locale.init_console(os.path.join(package_dir, 'locale'), 'sphinx')
 
-    if sys.argv[1:2] == ['-M']:
+    if argv[:1] == ['-M']:
         return make_main(argv)
     else:
         return build_main(argv)

sphinx/config.py

@@ -129,7 +129,7 @@ class Config(object):
         rst_epilog = (None, 'env', string_classes),
         rst_prolog = (None, 'env', string_classes),
         trim_doctest_flags = (True, 'env', []),
-        primary_domain = ('py', 'env', [NoneType]),
+        primary_domain = ('py', 'env', [NoneType]),   # type: ignore
         needs_sphinx = (None, None, string_classes),
         needs_extensions = ({}, None, []),
         manpages_url = (None, 'env', []),

sphinx/directives/other.py

@@ -45,7 +45,7 @@ locale.versionlabels = DeprecatedDict(
     RemovedInSphinx30Warning
 )
 
-glob_re = re.compile('.*[*?\[].*')
+glob_re = re.compile(r'.*[*?\[].*')
 
 
 def int_or_nothing(argument):

sphinx/ext/autosummary/__init__.py

@@ -78,7 +78,9 @@ from sphinx.ext.autodoc.importer import import_module
 from sphinx.locale import __
 from sphinx.pycode import ModuleAnalyzer, PycodeError
 from sphinx.util import import_object, rst, logging
-from sphinx.util.docutils import NullReporter, SphinxDirective, new_document
+from sphinx.util.docutils import (
+    NullReporter, SphinxDirective, new_document, switch_source_input
+)
 
 if False:
     # For type annotation
@@ -92,6 +94,7 @@ logger = logging.getLogger(__name__)
 
 
 periods_re = re.compile(r'\.(?:\s+)')
+literal_re = re.compile(r'::\s*$')
 
 
 # -- autosummary_toc node ------------------------------------------------------
@@ -373,17 +376,19 @@ class Autosummary(SphinxDirective):
         def append_row(*column_texts):
             # type: (unicode) -> None
             row = nodes.row('')
+            source, line = self.state_machine.get_source_and_line()
             for text in column_texts:
                 node = nodes.paragraph('')
                 vl = ViewList()
-                vl.append(text, '<autosummary>')
+                vl.append(text, '%s:%d:<autosummary>' % (source, line))
-                self.state.nested_parse(vl, 0, node)
+                with switch_source_input(self.state, vl):
-                try:
+                    self.state.nested_parse(vl, 0, node)
-                    if isinstance(node[0], nodes.paragraph):
+                    try:
-                        node = node[0]
+                        if isinstance(node[0], nodes.paragraph):
-                except IndexError:
+                            node = node[0]
-                    pass
+                    except IndexError:
-                row.append(nodes.entry('', node))
+                        pass
+                    row.append(nodes.entry('', node))
             body.append(row)
 
         for name, sig, summary, real_name in items:
@@ -484,6 +489,9 @@ def extract_summary(doc, document):
                 # considered as that splitting by period does not break inline markups
                 break
 
+    # strip literal notation mark ``::`` from tail of summary
+    summary = literal_re.sub('.', summary)
+
     return summary
 
 

sphinx/ext/autosummary/generate.py

@@ -124,7 +124,7 @@ def generate_autosummary_docs(sources, output_dir=None, suffix='.rst',
     else:
         if template_dir:
             template_dirs.insert(0, template_dir)
-        template_loader = FileSystemLoader(template_dirs)  # type: ignore
+        template_loader = FileSystemLoader(template_dirs)
     template_env = SandboxedEnvironment(loader=template_loader)
     template_env.filters['underline'] = _underline
 

sphinx/ext/napoleon/docstring.py

@@ -556,7 +556,14 @@ class GoogleDocstring(UnicodeMixin):
         self._parsed_lines = self._consume_empty()
 
         if self._name and (self._what == 'attribute' or self._what == 'data'):
-            self._parsed_lines.extend(self._parse_attribute_docstring())
+            # Implicit stop using StopIteration no longer allowed in
+            # Python 3.7; see PEP 479
+            res = []  # type: List[unicode]
+            try:
+                res = self._parse_attribute_docstring()
+            except StopIteration:
+                pass
+            self._parsed_lines.extend(res)
             return
 
         while self._line_iter.has_next():

sphinx/locale/__init__.py

@@ -64,7 +64,7 @@ class _TranslationProxy(UserString, object):
     # replace function from UserString; it instantiates a self.__class__
     # for the encoding result
 
-    def encode(self, encoding=None, errors=None):
+    def encode(self, encoding=None, errors=None):  # type: ignore
         # type: (unicode, unicode) -> str
         if encoding:
             if errors:
@@ -88,7 +88,7 @@ class _TranslationProxy(UserString, object):
         return dir(text_type)
 
     def __iter__(self):
-        # type: () -> Iterator[unicode]
+        # type: () -> Iterator
         return iter(self.data)
 
     def __len__(self):
@@ -103,15 +103,15 @@ class _TranslationProxy(UserString, object):
         # type: () -> unicode
         return text_type(self.data)
 
-    def __add__(self, other):
+    def __add__(self, other):  # type: ignore
         # type: (unicode) -> unicode
         return self.data + other
 
-    def __radd__(self, other):
+    def __radd__(self, other):  # type: ignore
         # type: (unicode) -> unicode
         return other + self.data
 
-    def __mod__(self, other):
+    def __mod__(self, other):  # type: ignore
         # type: (unicode) -> unicode
         return self.data % other
 
@@ -119,11 +119,11 @@ class _TranslationProxy(UserString, object):
         # type: (unicode) -> unicode
         return other % self.data
 
-    def __mul__(self, other):
+    def __mul__(self, other):  # type: ignore
         # type: (Any) -> unicode
         return self.data * other
 
-    def __rmul__(self, other):
+    def __rmul__(self, other):  # type: ignore
         # type: (Any) -> unicode
         return other * self.data
 
@@ -165,7 +165,7 @@ class _TranslationProxy(UserString, object):
         # type: (Tuple[Callable, Tuple[unicode]]) -> None
         self._func, self._args = tup
 
-    def __getitem__(self, key):
+    def __getitem__(self, key):  # type: ignore
         # type: (Any) -> unicode
         return self.data[key]
 

sphinx/templates/latex/latex.tex_t

@@ -13,6 +13,11 @@
 \ifdefined\pdfpxdimen
    \let\sphinxpxdimen\pdfpxdimen\else\newdimen\sphinxpxdimen
 \fi \sphinxpxdimen=<%= pxunit %>\relax
+<% if use_xindy -%>
+%% turn off hyperref patch of \index as sphinx.xdy xindy module takes care of
+%% suitable \hyperpage mark-up, working around hyperref-xindy incompatibility
+\PassOptionsToPackage{hyperindex=false}{hyperref}
+<% endif -%>
 <%= passoptionstopackages %>
 \PassOptionsToPackage{warn}{textcomp}
 <%= inputenc %>
@@ -31,7 +36,7 @@
 <%= hyperref %>
 <%= contentsname %>
 <%= numfig_format %>
-<%= literalblockpto %>
+<%= translatablestrings %>
 <%= pageautorefname %>
 <%= tocdepth %>
 <%= secnumdepth %>

sphinx/testing/path.py

@@ -223,7 +223,7 @@ class path(text_type):
         """
         Joins the path with the argument given and returns the result.
         """
-        return self.__class__(os.path.join(self, *map(self.__class__, args)))  # type: ignore  # NOQA
+        return self.__class__(os.path.join(self, *map(self.__class__, args)))
 
     def listdir(self):
         # type: () -> List[unicode]

sphinx/texinputs/Makefile_t

@@ -22,6 +22,15 @@ export LATEXOPTS =
 # or on command line for faster builds.
 {% endif -%}
 LATEXMKOPTS =
+{% if xindy_use -%}
+export XINDYOPTS = {{ xindy_lang_option }} -M sphinx.xdy
+{% if latex_engine == 'pdflatex' and xindy_cyrillic -%}
+XINDYOPTS += -M cyrLICRutf8.xdy
+{% endif -%}
+{% if latex_engine == 'xelatex' or latex_engine == 'lualatex' -%}
+XINDYOPTS += -I xelatex
+{% endif -%}
+{% endif -%}
 # format: pdf or dvi (used only by archive targets)
 FMT = pdf
 

sphinx/texinputs/cyrLICRutf8.xdy

@@ -0,0 +1,103 @@
+;; -*- coding: utf-8; mode: Lisp; -*-
+;; style file for xindy
+;; filename: cyrLICRutf8.xdy
+;; description: style file for xindy which maps back LaTeX Internal
+;;   Character Representation of Cyrillic to utf-8
+;; usage: for use with pdflatex produced .idx files.
+;; This module must be loaded after the rule for suppressing space
+;; characters has been executed, hence after module sphinx.xdy.
+;; Contributed by the Sphinx team, July 2018.
+(merge-rule "\IeC{\'\CYRG}" "Ѓ" :string)
+(merge-rule "\IeC{\'\CYRK}" "Ќ" :string)
+(merge-rule "\IeC{\'\cyrg}" "ѓ" :string)
+(merge-rule "\IeC{\'\cyrk}" "ќ" :string)
+(merge-rule "\IeC{\CYRA}" "А" :string)
+(merge-rule "\IeC{\CYRB}" "Б" :string)
+(merge-rule "\IeC{\CYRC}" "Ц" :string)
+(merge-rule "\IeC{\CYRCH}" "Ч" :string)
+(merge-rule "\IeC{\CYRD}" "Д" :string)
+(merge-rule "\IeC{\CYRDJE}" "Ђ" :string)
+(merge-rule "\IeC{\CYRDZE}" "Ѕ" :string)
+(merge-rule "\IeC{\CYRDZHE}" "Џ" :string)
+(merge-rule "\IeC{\CYRE}" "Е" :string)
+(merge-rule "\IeC{\CYREREV}" "Э" :string)
+(merge-rule "\IeC{\CYRERY}" "Ы" :string)
+(merge-rule "\IeC{\CYRF}" "Ф" :string)
+(merge-rule "\IeC{\CYRG}" "Г" :string)
+(merge-rule "\IeC{\CYRGUP}" "Ґ" :string)
+(merge-rule "\IeC{\CYRH}" "Х" :string)
+(merge-rule "\IeC{\CYRHRDSN}" "Ъ" :string)
+(merge-rule "\IeC{\CYRI}" "И" :string)
+(merge-rule "\IeC{\CYRIE}" "Є" :string)
+(merge-rule "\IeC{\CYRII}" "І" :string)
+(merge-rule "\IeC{\CYRISHRT}" "Й" :string)
+(merge-rule "\IeC{\CYRJE}" "Ј" :string)
+(merge-rule "\IeC{\CYRK}" "К" :string)
+(merge-rule "\IeC{\CYRL}" "Л" :string)
+(merge-rule "\IeC{\CYRLJE}" "Љ" :string)
+(merge-rule "\IeC{\CYRM}" "М" :string)
+(merge-rule "\IeC{\CYRN}" "Н" :string)
+(merge-rule "\IeC{\CYRNJE}" "Њ" :string)
+(merge-rule "\IeC{\CYRO}" "О" :string)
+(merge-rule "\IeC{\CYRP}" "П" :string)
+(merge-rule "\IeC{\CYRR}" "Р" :string)
+(merge-rule "\IeC{\CYRS}" "С" :string)
+(merge-rule "\IeC{\CYRSFTSN}" "Ь" :string)
+(merge-rule "\IeC{\CYRSH}" "Ш" :string)
+(merge-rule "\IeC{\CYRSHCH}" "Щ" :string)
+(merge-rule "\IeC{\CYRT}" "Т" :string)
+(merge-rule "\IeC{\CYRTSHE}" "Ћ" :string)
+(merge-rule "\IeC{\CYRU}" "У" :string)
+(merge-rule "\IeC{\CYRUSHRT}" "Ў" :string)
+(merge-rule "\IeC{\CYRV}" "В" :string)
+(merge-rule "\IeC{\CYRYA}" "Я" :string)
+(merge-rule "\IeC{\CYRYI}" "Ї" :string)
+(merge-rule "\IeC{\CYRYO}" "Ё" :string)
+(merge-rule "\IeC{\CYRYU}" "Ю" :string)
+(merge-rule "\IeC{\CYRZ}" "З" :string)
+(merge-rule "\IeC{\CYRZH}" "Ж" :string)
+(merge-rule "\IeC{\cyra}" "а" :string)
+(merge-rule "\IeC{\cyrb}" "б" :string)
+(merge-rule "\IeC{\cyrc}" "ц" :string)
+(merge-rule "\IeC{\cyrch}" "ч" :string)
+(merge-rule "\IeC{\cyrd}" "д" :string)
+(merge-rule "\IeC{\cyrdje}" "ђ" :string)
+(merge-rule "\IeC{\cyrdze}" "ѕ" :string)
+(merge-rule "\IeC{\cyrdzhe}" "џ" :string)
+(merge-rule "\IeC{\cyre}" "е" :string)
+(merge-rule "\IeC{\cyrerev}" "э" :string)
+(merge-rule "\IeC{\cyrery}" "ы" :string)
+(merge-rule "\IeC{\cyrf}" "ф" :string)
+(merge-rule "\IeC{\cyrg}" "г" :string)
+(merge-rule "\IeC{\cyrgup}" "ґ" :string)
+(merge-rule "\IeC{\cyrh}" "х" :string)
+(merge-rule "\IeC{\cyrhrdsn}" "ъ" :string)
+(merge-rule "\IeC{\cyri}" "и" :string)
+(merge-rule "\IeC{\cyrie}" "є" :string)
+(merge-rule "\IeC{\cyrii}" "і" :string)
+(merge-rule "\IeC{\cyrishrt}" "й" :string)
+(merge-rule "\IeC{\cyrje}" "ј" :string)
+(merge-rule "\IeC{\cyrk}" "к" :string)
+(merge-rule "\IeC{\cyrl}" "л" :string)
+(merge-rule "\IeC{\cyrlje}" "љ" :string)
+(merge-rule "\IeC{\cyrm}" "м" :string)
+(merge-rule "\IeC{\cyrn}" "н" :string)
+(merge-rule "\IeC{\cyrnje}" "њ" :string)
+(merge-rule "\IeC{\cyro}" "о" :string)
+(merge-rule "\IeC{\cyrp}" "п" :string)
+(merge-rule "\IeC{\cyrr}" "р" :string)
+(merge-rule "\IeC{\cyrs}" "с" :string)
+(merge-rule "\IeC{\cyrsftsn}" "ь" :string)
+(merge-rule "\IeC{\cyrsh}" "ш" :string)
+(merge-rule "\IeC{\cyrshch}" "щ" :string)
+(merge-rule "\IeC{\cyrt}" "т" :string)
+(merge-rule "\IeC{\cyrtshe}" "ћ" :string)
+(merge-rule "\IeC{\cyru}" "у" :string)
+(merge-rule "\IeC{\cyrushrt}" "ў" :string)
+(merge-rule "\IeC{\cyrv}" "в" :string)
+(merge-rule "\IeC{\cyrya}" "я" :string)
+(merge-rule "\IeC{\cyryi}" "ї" :string)
+(merge-rule "\IeC{\cyryo}" "ё" :string)
+(merge-rule "\IeC{\cyryu}" "ю" :string)
+(merge-rule "\IeC{\cyrz}" "з" :string)
+(merge-rule "\IeC{\cyrzh}" "ж" :string)

sphinx/texinputs/latexmkrc_t

@@ -10,7 +10,11 @@ $pdflatex = 'xelatex ' . $ENV{'LATEXOPTS'} . ' %O %S';
 {% endif -%}
 $lualatex = 'lualatex ' . $ENV{'LATEXOPTS'} . ' %O %S';
 $xelatex = 'xelatex --no-pdf ' . $ENV{'LATEXOPTS'} . ' %O %S';
+{% if xindy_use -%}
+$makeindex = 'xindy ' . $ENV{'XINDYOPTS'} . ' %O -o %D %S';
+{% else -%}
 $makeindex = 'makeindex -s python.ist %O -o %D %S';
+{% endif -%}
 add_cus_dep( "glo", "gls", 0, "makeglo" );
 sub makeglo {
  return system( "makeindex -s gglo.ist -o '$_[0].gls' '$_[0].glo'" );

sphinx/texinputs/python.ist

@@ -3,11 +3,11 @@ headings_flag 1
 heading_prefix "  \\bigletter "
 
 preamble "\\begin{sphinxtheindex}
-\\def\\bigletter#1{{\\Large\\sffamily#1}\\nopagebreak\\vspace{1mm}}
+\\let\\bigletter\\sphinxstyleindexlettergroup
 
 "
 
 postamble "\n\n\\end{sphinxtheindex}\n"
 
-symhead_positive "{Symbols}"
+symhead_positive "{\\sphinxsymbolsname}"
-numhead_positive "{Numbers}"
+numhead_positive "{\\sphinxnumbersname}"

sphinx/texinputs/sphinx.sty

@@ -6,7 +6,7 @@
 %
 
 \NeedsTeXFormat{LaTeX2e}[1995/12/01]
-\ProvidesPackage{sphinx}[2018/03/28 v1.8 LaTeX package (Sphinx markup)]
+\ProvidesPackage{sphinx}[2018/06/30 v1.8 LaTeX package (Sphinx markup)]
 
 % provides \ltx@ifundefined
 % (many packages load ltxcmds: graphicx does for pdftex and lualatex but
@@ -470,6 +470,11 @@
  {\newenvironment{sphinxtheindex}{\begin{theindex}}{\end{theindex}}}%
  {}% else clause of \ltx@ifundefined
 
+% for usage with xindy: this string gets internationalized in preamble
+\newcommand*{\sphinxnonalphabeticalgroupname}{}
+% redefined in preamble, headings for makeindex produced index
+\newcommand*{\sphinxsymbolsname}{}
+\newcommand*{\sphinxnumbersname}{}
 
 %% COLOR (general)
 %
@@ -1591,10 +1596,13 @@
   {\textnormal{\Large[}}{#1}\hspace{0.5mm}{\textnormal{\Large]}}}
 
 % additional customizable styling
-% FIXME: convert this to package options ?
+\def\sphinxstyleindexentry   #1{\texttt{#1}}
-\protected\def\sphinxstyleindexentry   #1{\texttt{#1}}
+\def\sphinxstyleindexextra   #1{ \emph{(#1)}}
-\protected\def\sphinxstyleindexextra   #1{ \emph{(#1)}}
+\def\sphinxstyleindexpageref #1{, \pageref{#1}}
-\protected\def\sphinxstyleindexpageref #1{, \pageref{#1}}
+\def\sphinxstyleindexlettergroup #1%
+    {{\Large\sffamily#1}\nopagebreak\vspace{1mm}}
+\def\sphinxstyleindexlettergroupDefault #1%
+    {{\Large\sffamily\sphinxnonalphabeticalgroupname}\nopagebreak\vspace{1mm}}
 \protected\def\sphinxstyletopictitle   #1{\textbf{#1}\par\medskip}
 \let\sphinxstylesidebartitle\sphinxstyletopictitle
 \protected\def\sphinxstyleothertitle   #1{\textbf{#1}}

sphinx/texinputs/sphinx.xdy

@@ -0,0 +1,133 @@
+;;; -*- mode: lisp; coding: utf-8; -*-
+
+;; Unfortunately xindy is out-of-the-box hyperref-incompatible.  This
+;; configuration is a workaround, which requires to pass option
+;; hyperindex=false to hyperref.
+;; textit and emph not currently used by Sphinx LaTeX writer.
+(define-attributes (("textbf" "textit" "emph" "default")))
+(markup-locref :open "\textbf{\hyperpage{" :close "}}" :attr "textbf")
+(markup-locref :open "\textit{\hyperpage{" :close "}}" :attr "textit")
+(markup-locref :open "\emph{\hyperpage{" :close "}}" :attr "emph")
+(markup-locref :open "\hyperpage{" :close "}" :attr "default")
+
+(require "numeric-sort.xdy")
+
+;; xindy base module latex.xdy loads tex.xdy and the latter instructs
+;; xindy to ignore **all** TeX macros in .idx entries, except those
+;; explicitely described in merge rule.  But when after applying all
+;; merge rules an empty string results, xindy raises an error:
+
+;;   ERROR: CHAR: index 0 should be less than the length of the string
+
+;; For example when using pdflatex with utf-8 characters the index
+;; file will contain \IeC macros and they will get ignored except if
+;; suitable merge rules are loaded early.  The texindy script coming
+;; with xindy provides this, but only for Latin scripts.  The texindy
+;; man page says to use rather xelatex or lualatex in case of Cyrillic
+;; scripts.
+
+;; Sphinx contributes cyrLICRutf8.xdy to provide support for Cyrillic
+;; scripts for the pdflatex engine.
+
+;; Another issue caused by xindy ignoring all TeX macros except those
+;; explicitely declared reveals itself when attempting to index ">>>",
+;; as the ">" is converted to "\textgreater{}" by Sphinx's LaTeX
+;; escaping.
+
+;; To fix this, Sphinx does **not** use texindy, and does not even
+;; load the xindy latex.xdy base module.
+
+;(require "latex.xdy")
+
+;; Rather it incorporates some suitable extracts from latex.xdy and
+;; tex.xdy with additional Sphinx contributed rules.
+
+;;;;;;;; extracts from tex.xdy (discarding most original comments):
+
+;;;
+;;; TeX conventions
+;;;
+
+;; Discard leading and trailing white space. Collapse multiple white
+;; space characters to blank.
+
+(merge-rule "^ +" "" :eregexp)
+(merge-rule " +$" "" :eregexp)
+(merge-rule "  +" " " :eregexp)
+
+;; Handle TeX markup
+
+(merge-rule "\\([{}$%&#])" "\1" :eregexp)
+
+;;;;;;;; end of extracts from xindy's tex.xdy
+
+;;;;;;;; extracts from latex.xdy:
+
+;; Standard location classes: arabic and roman numbers, and alphabets.
+
+(define-location-class "arabic-page-numbers" ("arabic-numbers"))
+(define-location-class "roman-page-numbers"  ("roman-numbers-lowercase"))
+(define-location-class "Roman-page-numbers"  ("roman-numbers-uppercase"))
+(define-location-class "alpha-page-numbers"  ("alpha"))
+(define-location-class "Alpha-page-numbers"  ("ALPHA"))
+
+;; Output Markup
+
+(markup-letter-group-list :sep "~n~n  \indexspace~n")
+
+(markup-indexentry :open "~n  \item "           :depth 0)
+(markup-indexentry :open "~n    \subitem "      :depth 1)
+(markup-indexentry :open "~n      \subsubitem " :depth 2)
+
+(markup-locclass-list :open ", " :sep ", ")
+(markup-locref-list   :sep ", ")
+
+;;;;;;;; end of extracts from latex.xdy
+
+;; Sphinx additions, cf sphinx.util.texescape for rationale
+;;
+;; blanks are already ignored from above merge-rules, so no space
+;; character after TeX control words, despite the fact that they will
+;; be present in .idx file.
+
+(merge-rule "\\sphinxleftcurlybrace\{\}" "{")
+(merge-rule "\\sphinxrightcurlybrace\{\}" "}")
+(merge-rule "\\_" "_")
+(merge-rule "\{\[\}" "[")
+(merge-rule "\{\]\}" "]")
+(merge-rule "\{\}`" "`")
+(merge-rule "\\textbackslash\{\}" "\\")
+(merge-rule "\\textasciitilde\{\}" "~~")
+(merge-rule "\\textless\{\}" "<")
+(merge-rule "\\textgreater\{\}" ">")
+(merge-rule "\\textasciicircum\{\}" "^")
+(merge-rule "\\P\{\}" "¶")
+(merge-rule "\\S\{\}" "§")
+(merge-rule "\\texteuro\{\}" "€")
+(merge-rule "\\\(\\infty\\\)" "∞")
+(merge-rule "\\\(\\pm\\\)" "±")
+(merge-rule "\\\(\\rightarrow\\\)" "→")
+(merge-rule "\\\(\\checkmark\\\)" "✓")
+(merge-rule "\\textendash\{\}" "–")
+(merge-rule "\\textbar\{\}" "|")
+
+;; This xindy module provides some basic support for "see"
+(require "makeindex.xdy")
+
+;; This creates one-letter headings and works fine with utf-8 letters.
+;; For Cyrillic and pdflatex necessitates cyrLICRutf8.xdy to be loaded too.
+(require "latin-lettergroups.xdy")
+
+;; currently we don't (know how to easily) separate "Numbers" from
+;; "Symbols" with xindy as is the case with makeindex.
+(markup-index :open  "\begin{sphinxtheindex}
+\let\lettergroup\sphinxstyleindexlettergroup
+\let\lettergroupDefault\sphinxstyleindexlettergroupDefault
+
+"
+	      :close "
+
+\end{sphinxtheindex}
+"
+	      :tree)
+

sphinx/util/inspect.py

@@ -241,7 +241,7 @@ def object_description(object):
     if isinstance(object, dict):
         try:
             sorted_keys = sorted(object)
-        except TypeError:
+        except Exception:
             pass  # Cannot sort dict keys, fall back to generic repr
         else:
             items = ("%s: %s" %

sphinx/writers/latex.py

@@ -130,7 +130,7 @@ DEFAULT_SETTINGS = {
     'tocdepth':        '',
     'secnumdepth':     '',
     'pageautorefname': '',
-    'literalblockpto': '',
+    'translatablestrings': '',
 }  # type: Dict[unicode, unicode]
 
 ADDITIONAL_SETTINGS = {
@@ -255,7 +255,7 @@ class ExtBabel(Babel):
 
     def get_mainlanguage_options(self):
         # type: () -> unicode
-        """Return options for polyglossia's ``\setmainlanguage``."""
+        """Return options for polyglossia's ``\\setmainlanguage``."""
         if self.use_polyglossia is False:
             return None
         elif self.language == 'german':
@@ -497,6 +497,7 @@ class LaTeXTranslator(nodes.NodeVisitor):
             'release':      self.encode(builder.config.release),
             'author':       document.settings.author,   # treat as a raw LaTeX code
             'indexname':    _('Index'),
+            'use_xindy':    builder.config.latex_use_xindy,
         })
         if not self.elements['releasename'] and self.elements['release']:
             self.elements.update({
@@ -667,12 +668,21 @@ class LaTeXTranslator(nodes.NodeVisitor):
         if self.elements['extraclassoptions']:
             self.elements['classoptions'] += ',' + \
                                              self.elements['extraclassoptions']
-        self.elements['literalblockpto'] = (
+        self.elements['translatablestrings'] = (
             self.babel_renewcommand(
                 '\\literalblockcontinuedname', self.encode(_('continued from previous page'))
             ) +
             self.babel_renewcommand(
                 '\\literalblockcontinuesname', self.encode(_('continues on next page'))
+            ) +
+            self.babel_renewcommand(
+                '\\sphinxnonalphabeticalgroupname', self.encode(_('Non-alphabetical'))
+            ) +
+            self.babel_renewcommand(
+                '\\sphinxsymbolsname', self.encode(_('Symbols'))
+            ) +
+            self.babel_renewcommand(
+                '\\sphinxnumbersname', self.encode(_('Numbers'))
             )
         )
         self.elements['pageautorefname'] = \
@@ -856,8 +866,7 @@ class LaTeXTranslator(nodes.NodeVisitor):
         def generate(content, collapsed):
             # type: (List[Tuple[unicode, List[Tuple[unicode, unicode, unicode, unicode, unicode]]]], bool) -> None  # NOQA
             ret.append('\\begin{sphinxtheindex}\n')
-            ret.append('\\def\\bigletter#1{{\\Large\\sffamily#1}'
+            ret.append('\\let\\bigletter\\sphinxstyleindexlettergroup\n')
-                       '\\nopagebreak\\vspace{1mm}}\n')
             for i, (letter, entries) in enumerate(content):
                 if i > 0:
                     ret.append('\\indexspace\n')
@@ -866,7 +875,8 @@ class LaTeXTranslator(nodes.NodeVisitor):
                 for entry in entries:
                     if not entry[3]:
                         continue
-                    ret.append('\\item {\\sphinxstyleindexentry{%s}}' % self.encode(entry[0]))
+                    ret.append('\\item\\relax\\sphinxstyleindexentry{%s}' %
+                               self.encode(entry[0]))
                     if entry[4]:
                         # add "extra" info
                         ret.append('\\sphinxstyleindexextra{%s}' % self.encode(entry[4]))
@@ -1919,8 +1929,11 @@ class LaTeXTranslator(nodes.NodeVisitor):
         # type: (nodes.Node, Pattern) -> None
         def escape(value):
             value = self.encode(value)
-            value = value.replace(r'\{', r'{\sphinxleftcurlybrace}')
+            value = value.replace(r'\{', r'\sphinxleftcurlybrace{}')
-            value = value.replace(r'\}', r'{\sphinxrightcurlybrace}')
+            value = value.replace(r'\}', r'\sphinxrightcurlybrace{}')
+            value = value.replace('"', '""')
+            value = value.replace('@', '"@')
+            value = value.replace('!', '"!')
             return value
 
         if not node.get('inline', True):

tests/test_build_latex.py

@@ -1201,7 +1201,7 @@ def test_latex_index(app, status, warning):
     result = (app.outdir / 'Python.tex').text(encoding='utf8')
     assert 'A \\index{famous}famous \\index{equation}equation:\n' in result
     assert '\n\\index{Einstein}\\index{relativity}\\ignorespaces \nand' in result
-    assert '\n\\index{main {\\sphinxleftcurlybrace}}\\ignorespaces ' in result
+    assert '\n\\index{main \\sphinxleftcurlybrace{}}\\ignorespaces ' in result
 
 
 @pytest.mark.sphinx('latex', testroot='latex-equations')

tests/test_ext_autosummary.py

@@ -81,6 +81,10 @@ def test_extract_summary(capsys):
     doc = ['Blabla, i.e. bla.']
     assert extract_summary(doc, document) == 'Blabla, i.e.'
 
+    # literal
+    doc = ['blah blah::']
+    assert extract_summary(doc, document) == 'blah blah.'
+
     _, err = capsys.readouterr()
     assert err == ''
 

tox.ini

@@ -27,12 +27,14 @@ commands=
     pytest -Wall --durations 25 {posargs}
 
 [testenv:flake8]
+basepython = python3
 description =
     Run style checks.
 commands =
     flake8
 
 [testenv:pylint]
+basepython = python3
 description =
     Run source code analyzer.
 deps =
@@ -42,6 +44,7 @@ commands =
     pylint --rcfile utils/pylintrc sphinx
 
 [testenv:coverage]
+basepython = python3
 description =
     Run code coverage checks.
 setenv =
@@ -51,6 +54,7 @@ commands =
     coverage report
 
 [testenv:mypy]
+basepython = python3
 description =
     Run type checks.
 deps =
@@ -59,6 +63,7 @@ commands=
     mypy sphinx/
 
 [testenv:docs]
+basepython = python3
 description =
     Build documentation.
 commands =