Merge branch 'master' into deprecate_highlightlang_directive

CHANGES

@@ -17,6 +17,12 @@ Incompatible changes
   a ``conf.py`` file sphinx-build generates.
 * The ``gettext_compact`` attribute is removed from ``document.settings``
   object.  Please use ``config.gettext_compact`` instead.
+* The processing order on reading phase is changed.  smart_quotes, sphinx
+  domains, :event:`doctree-read` event and versioning doctrees are invoked
+  earlier than so far.  For more details, please read a description of
+  :py:meth:`.Sphinx.add_transform()`
+* #4827: All ``substitution_definition`` nodes are removed from doctree on
+  reading phase
 
 Deprecated
 ----------
@@ -36,6 +42,12 @@ Deprecated
 * ``env._nitpick_ignore`` is deprecated
 * ``app.override_domain()`` is deprecated
 * ``app.add_stylesheet()`` is deprecated
+* ``sphinx.versioning.prepare()`` is deprecated
+* ``Config.__init__()`` has changed;  the *dirname*, *filename* and *tags*
+  argument has been deprecated
+* ``Config.check_types()`` is deprecated
+* ``Config.check_unicode()`` is deprecated
+* ``sphinx.application.CONFIG_FILENAME`` is deprecated
 * ``highlightlang`` directive is deprecated
 
 For more details, see `deprecation APIs list
@@ -63,6 +75,13 @@ Features added
 * Add :confval:`html_css_files` and :confval:`epub_css_files` for adding CSS
   files from configuration
 * #4834: Ensure set object descriptions are reproducible.
+* #4828: Allow to override :confval:`numfig_format` partially.  Full definition
+  is not needed.
+* Improve warning messages during including (refs: #4818)
+* LaTeX: separate customizability of :rst:role:`guilabel` and
+  :rst:role:`menuselection` (refs: #4830)
+* Add ``Config.read()`` classmethod to create a new config object from
+  configuration file
 
 Bugs fixed
 ----------
@@ -77,7 +96,7 @@ Features removed
 
 * ``sphinx.ext.pngmath`` extension
 
-Release 1.7.3 (in development)
+Release 1.7.5 (in development)
 ==============================
 
 Dependencies
@@ -95,6 +114,24 @@ Features added
 Bugs fixed
 ----------
 
+Testing
+--------
+
+Release 1.7.4 (released Apr 25, 2018)
+=====================================
+
+Bugs fixed
+----------
+
+* #4885, #4887: domains: Crashed with duplicated objects
+* #4889: latex: sphinx.writers.latex causes recusrive import
+
+Release 1.7.3 (released Apr 23, 2018)
+=====================================
+
+Bugs fixed
+----------
+
 * #4769: autodoc loses the first staticmethod parameter
 * #4790: autosummary: too wide two column tables in PDF builds
 * #4795: Latex customization via ``_templates/longtable.tex_t`` is broken
@@ -105,9 +142,14 @@ Bugs fixed
 * #4817: wrong URLs on warning messages
 * #4784: latex: :confval:`latex_show_urls` assigns incorrect footnote numbers if
   hyperlinks exists inside substitutions
-
+* #4837: latex with class memoir Error: Font command ``\sf`` is not supported
-Testing
+* #4803: latex: too slow in proportion to number of auto numbered footnotes
---------
+* #4838: htmlhelp: The entries in .hhp file is not ordered
+* toctree directive tries to glob for URL having query_string
+* #4871: html search: Upper characters problem in German
+* #4717: latex: Compilation for German docs failed with LuaLaTeX and XeLaTeX
+* #4459: duplicated labels detector does not work well in parallel build
+* #4878: Crashed with extension which returns invalid metadata
 
 Release 1.7.2 (released Mar 21, 2018)
 =====================================
@@ -139,6 +181,7 @@ Bugs fixed
 * #1435: qthelp builder should htmlescape keywords
 * epub: Fix docTitle elements of toc.ncx is not escaped
 * #4520: apidoc: Subpackage not in toc (introduced in 1.6.6) now fixed
+* #4767: html: search highlighting breaks mathjax equations
 
 Release 1.7.1 (released Feb 23, 2018)
 =====================================
@@ -214,6 +257,8 @@ Incompatible changes
   (refs: #4295)
 * #4246: Limit width of text body for all themes. Conifigurable via theme
   options ``body_min_width`` and ``body_max_width``.
+* #4771: apidoc: The ``exclude_patterns`` arguments are ignored if they are
+  placed just after command line options
 
 1.7.0b2
 

EXAMPLES

@@ -317,7 +317,7 @@ Documentation using a custom theme or integrated in a website
 * PSI4: http://www.psicode.org/psi4manual/master/index.html
 * PyMOTW: https://pymotw.com/2/
 * python-aspectlib: https://python-aspectlib.readthedocs.io/
-  (`sphinx_py3doc_enhanced_theme <https://pypi.python.org/pypi/sphinx_py3doc_enhanced_theme>`__)
+  (`sphinx_py3doc_enhanced_theme <https://pypi.org/project/sphinx_py3doc_enhanced_theme/>`__)
 * QGIS: https://qgis.org/en/docs/index.html
 * qooxdoo: http://www.qooxdoo.org/current/
 * Roundup: http://www.roundup-tracker.org/

README.rst

@@ -3,7 +3,7 @@
 ========
 
 .. image:: https://img.shields.io/pypi/v/sphinx.svg
-   :target: https://pypi.python.org/pypi/Sphinx
+   :target: https://pypi.org/project/Sphinx/
    :alt: Package on PyPi
 
 .. image:: https://readthedocs.org/projects/sphinx/badge/
@@ -71,7 +71,7 @@ We also publish beta releases::
 If you wish to install `Sphinx` for development purposes, refer to `the
 contributors guide`__.
 
-__ https://pypi.python.org/pypi/Sphinx
+__ https://pypi.org/project/Sphinx/
 __ http://www.sphinx-doc.org/en/master/devguide.html
 
 Documentation

doc/_templates/indexsidebar.html

@@ -8,11 +8,11 @@
   not released yet.{%endtrans%}</p>
 <p>{%trans%}You can use it from the
   <a href="https://github.com/sphinx-doc/sphinx/">Git repo</a> or look for
-  released versions in the <a href="https://pypi.python.org/pypi/Sphinx">Python
+  released versions in the <a href="https://pypi.org/project/Sphinx/">Python
     Package Index</a>.{%endtrans%}</p>
 {% else %}
 <p>{%trans%}Current version: <b><a href="changes.html">{{ version }}</a></b>{%endtrans%}</p>
-<p>{%trans%}Get Sphinx from the <a href="https://pypi.python.org/pypi/Sphinx">Python Package
+<p>{%trans%}Get Sphinx from the <a href="https://pypi.org/project/Sphinx/">Python Package
 Index</a>, or install it with:{%endtrans%}</p>
 <pre>pip install -U Sphinx</pre>
 {% endif %}

doc/_themes/sphinx13/static/sphinx13.css

@@ -302,6 +302,10 @@ cite, code, tt {
     letter-spacing: -0.02em;
 }
 
+table.deprecated code.literal {
+    word-break: break-all;
+}
+
 tt {
     background-color: #f2f2f2;
     border: 1px solid #ddd;

doc/builders.rst

@@ -288,7 +288,7 @@ name is ``rinoh``. Refer to the `rinohtype manual`_ for details.
             globalcontext_filename = 'globalcontext.phpdump'
             searchindex_filename = 'searchindex.phpdump'
 
-   .. _PHP serialization: https://pypi.python.org/pypi/phpserialize
+   .. _PHP serialization: https://pypi.org/project/phpserialize/
 
    .. attribute:: implementation
 

doc/config.rst

@@ -192,8 +192,9 @@ General configuration
    .. index:: pair: global; substitutions
 
    A string of reStructuredText that will be included at the end of every source
-   file that is read.  This is the right place to add substitutions that should
+   file that is read.  This is a possible place to add substitutions that should
-   be available in every file.  An example::
+   be available in every file (another being :confval:`rst_prolog`).  An
+   example::
 
       rst_epilog = """
       .. |psf| replace:: Python Software Foundation
@@ -203,8 +204,16 @@ General configuration
 
 .. confval:: rst_prolog
 
+   .. index:: pair: global; substitutions
+
    A string of reStructuredText that will be included at the beginning of every
-   source file that is read.
+   source file that is read.  This is a possible place to add substitutions that
+   should be available in every file (another being :confval:`rst_epilog`).  An
+   example::
+
+      rst_prolog = """
+      .. |psf| replace:: Python Software Foundation
+      """
 
    .. versionadded:: 1.0
 
@@ -868,6 +877,18 @@ that use Sphinx's HTMLWriter class.
    named :file:`default.css` will overwrite the theme's
    :file:`default.css`.
 
+   .. note::
+
+      For security reason, dotfiles under ``html_static_path`` will
+      not be copied.  If you'd like to copy them intentionally, please
+      add them each filepath to this setting::
+
+          html_static_path = ['_static', '_static/.htaccess']
+
+      Another way to do that, you can also use
+      :confval:`html_extra_path`.  It allows to copy dotfiles under
+      the directories.
+
    .. versionchanged:: 0.4
       The paths in :confval:`html_static_path` can now contain subdirectories.
 
@@ -1138,8 +1159,8 @@ that use Sphinx's HTMLWriter class.
       Sphinx uses a Python implementation by default.  You can use a C
       implementation to accelerate building the index file.
 
-      * `PorterStemmer <https://pypi.python.org/pypi/PorterStemmer>`_ (``en``)
+      * `PorterStemmer <https://pypi.org/project/PorterStemmer/>`_ (``en``)
-      * `PyStemmer <https://pypi.python.org/pypi/PyStemmer>`_ (all languages)
+      * `PyStemmer <https://pypi.org/project/PyStemmer/>`_ (all languages)
 
    .. versionadded:: 1.1
       With support for ``en`` and ``ja``.
@@ -1171,10 +1192,12 @@ that use Sphinx's HTMLWriter class.
          library ('libmecab.so' for linux, 'libmecab.dll' for windows) is required.
       :'sphinx.search.ja.JanomeSplitter':
          Janome binding. To use this splitter,
-         `Janome <https://pypi.python.org/pypi/Janome>`_ is required.
+         `Janome <https://pypi.org/project/Janome/>`_ is required.
 
-      To keep compatibility, ``'mecab'``, ``'janome'`` and ``'default'`` are also
+      .. deprecated:: 1.6
-      acceptable. However it will be deprecated in Sphinx-1.6.
+         ``'mecab'``, ``'janome'`` and ``'default'`` is deprecated.
+         To keep compatibility, ``'mecab'``, ``'janome'`` and ``'default'`` are
+         also acceptable.
 
 
    Other option values depend on splitter value which you choose.

doc/develop.rst

@@ -141,5 +141,5 @@ own extensions.
 .. _inlinesyntaxhighlight: https://sphinxcontrib-inlinesyntaxhighlight.readthedocs.io/
 .. _CMake: https://cmake.org
 .. _domaintools: https://bitbucket.org/klorenz/sphinxcontrib-domaintools
-.. _restbuilder: https://pypi.python.org/pypi/sphinxcontrib-restbuilder
+.. _restbuilder: https://pypi.org/project/sphinxcontrib-restbuilder/
 .. _Lasso: http://www.lassosoft.com/

doc/ext/imgconverter.rst

@@ -1,7 +1,9 @@
 .. highlight:: rest
 
-:mod:`sphinx.ext.imgconverter` -- Convert images to appropriate format for builders
+.. _sphinx.ext.imgconverter:
-===================================================================================
+
+:mod:`sphinx.ext.imgconverter` -- A reference implementation for image converter using Imagemagick
+==================================================================================================
 
 .. module:: sphinx.ext.imgconverter
    :synopsis: Convert images to appropriate format for builders
@@ -16,6 +18,12 @@ Internally, this extension uses Imagemagick_ to convert images.
 
 .. _Imagemagick: https://www.imagemagick.org/script/index.php
 
+.. note:: Imagemagick rasterizes a SVG image on conversion.  As a result, the image
+          becomes not scalable.  To avoid that, please use other image converters
+          like sphinxcontrib-svg2pdfconverter_ (which uses Inkscape or rsvg-convert).
+
+.. _sphinxcontrib-svg2pdfconverter: https://github.com/missinglinkelectronics/sphinxcontrib-svg2pdfconverter
+
 Configuration
 -------------
 

doc/extdev/index.rst

@@ -108,6 +108,8 @@ The following is a list of deprecated interface.
 
 .. list-table:: deprecated APIs
    :header-rows: 1
+   :class: deprecated
+   :widths: 40, 10, 10, 40
 
    * - Target
      - Deprecated
@@ -124,6 +126,37 @@ The following is a list of deprecated interface.
      - 4.0
      - :meth:`~sphinx.application.Sphinx.add_css_file()`
 
+   * - ``sphinx.application.CONFIG_FILENAME``
+     - 1.8
+     - 3.0
+     - ``sphinx.config.CONFIG_FILENAME``
+
+   * - ``Config.check_unicode()``
+     - 1.8
+     - 3.0
+     - ``sphinx.config.check_unicode()``
+
+   * - ``Config.check_types()``
+     - 1.8
+     - 3.0
+     - ``sphinx.config.check_confval_types()``
+
+   * - ``dirname``, ``filename`` and ``tags`` arguments of
+       ``Config.__init__()``
+     - 1.8
+     - 3.0
+     - ``Config.read()``
+
+   * - The value of :confval:`html_search_options`
+     - 1.8
+     - 3.0
+     - see :confval:`html_search_options`
+
+   * - ``sphinx.versioning.prepare()``
+     - 1.8
+     - 3.0
+     - ``sphinx.versioning.UIDTransform``
+
    * - ``sphinx.application.Sphinx.override_domain()``
      - 1.8
      - 3.0
@@ -273,7 +306,7 @@ The following is a list of deprecated interface.
    * - ``sphinx.websupport``
      - 1.6
      - 2.0
-     - `sphinxcontrib-websupport <https://pypi.python.org/pypi/sphinxcontrib-websupport>`_
+     - `sphinxcontrib-websupport <https://pypi.org/project/sphinxcontrib-websupport/>`_
 
    * - ``StandaloneHTMLBuilder.css_files``
      - 1.6

doc/extdev/utils.rst

@@ -17,3 +17,6 @@ components (e.g. :class:`.Config`, :class:`.BuildEnvironment` and so on) easily.
 
 .. autoclass:: sphinx.util.docutils.SphinxDirective
    :members:
+
+.. autoclass:: sphinx.transforms.post_transforms.images.ImageConverter
+   :members:

doc/faq.rst

@@ -23,7 +23,7 @@ How do I...
    Use themes, see :doc:`theming`.
 
 ... add global substitutions or includes?
-   Add them in the :confval:`rst_epilog` config value.
+   Add them in the :confval:`rst_prolog` or :confval:`rst_epilog` config value.
 
 ... display the whole TOC tree in the sidebar?
    Use the :data:`toctree` callable in a custom layout template, probably in the
@@ -72,7 +72,7 @@ SCons
 
 PyPI
    Jannis Leidel wrote a `setuptools command
-   <https://pypi.python.org/pypi/Sphinx-PyPI-upload>`_ that automatically
+   <https://pypi.org/project/Sphinx-PyPI-upload/>`_ that automatically
    uploads Sphinx documentation to the PyPI package documentation area at
    https://pythonhosted.org/.
 

doc/intl.rst

@@ -322,8 +322,8 @@ There is `sphinx translation page`_ for Sphinx (master) documentation.
 .. [2] Because nobody expects the Spanish Inquisition!
 
 
-.. _`transifex-client`: https://pypi.python.org/pypi/transifex-client
+.. _`transifex-client`: https://pypi.org/project/transifex-client/
-.. _`sphinx-intl`: https://pypi.python.org/pypi/sphinx-intl
+.. _`sphinx-intl`: https://pypi.org/project/sphinx-intl/
 .. _Transifex: https://www.transifex.com/
 .. _`sphinx translation page`: https://www.transifex.com/sphinx-doc/sphinx-doc/
 .. _`Transifex Client documentation`: http://docs.transifex.com/developer/client/

doc/intro.rst

@@ -29,7 +29,7 @@ This section is intended to collect helpful hints for those wanting to migrate
 to reStructuredText/Sphinx from other documentation systems.
 
 * Gerard Flanagan has written a script to convert pure HTML to reST; it can be
-  found at the `Python Package Index <https://pypi.python.org/pypi/html2rest>`_.
+  found at the `Python Package Index <https://pypi.org/project/html2rest/>`_.
 
 * For converting the old Python docs to Sphinx, a converter was written which
   can be found at `the Python SVN repository
@@ -40,7 +40,7 @@ to reStructuredText/Sphinx from other documentation systems.
   markup; it is at `GitHub <https://github.com/wojdyr/db2rst>`_.
 
 * Christophe de Vienne wrote a tool to convert from Open/LibreOffice documents
-  to Sphinx: `odt2sphinx <https://pypi.python.org/pypi/odt2sphinx/>`_.
+  to Sphinx: `odt2sphinx <https://pypi.org/project/odt2sphinx/>`_.
 
 * To convert different markups, `Pandoc <https://pandoc.org/>`_ is
   a very helpful tool.

doc/theming.rst

@@ -134,7 +134,7 @@ These themes are:
   Check out at its `installation page`_ how to set up properly
   :confval:`html_sidebars` for its use.
 
-  .. _Alabaster theme: https://pypi.python.org/pypi/alabaster
+  .. _Alabaster theme: https://pypi.org/project/alabaster/
   .. _installation page: https://alabaster.readthedocs.io/en/latest/installation.html
 
 * **classic** -- This is the classic theme, which looks like `the Python 2
@@ -418,7 +418,7 @@ Third Party Themes
   View a working demo over on readthedocs.org. You can get install and options
   information at `Read the Docs Sphinx Theme`_ page.
 
-  .. _Read the Docs Sphinx Theme: https://pypi.python.org/pypi/sphinx_rtd_theme
+  .. _Read the Docs Sphinx Theme: https://pypi.org/project/sphinx_rtd_theme/
 
   .. versionchanged:: 1.4
      **sphinx_rtd_theme** has become optional.
@@ -427,6 +427,6 @@ Third Party Themes
 Besides this, there are a lot of third party themes.  You can find them on
 PyPI__, GitHub__, sphinx-themes.org__ and so on.
 
-.. __: https://pypi.python.org/pypi?:action=browse&c=599
+.. __: https://pypi.org/search/?q=&o=&c=Framework+%3A%3A+Sphinx+%3A%3A+Theme
 .. __: https://github.com/search?utf8=%E2%9C%93&q=sphinx+theme&type=
 .. __: https://sphinx-themes.org/

doc/usage/installation.rst

@@ -132,7 +132,7 @@ Installation from PyPI
 ----------------------
 
 Sphinx packages are published on the `Python Package Index
-<https://pypi.python.org/pypi/Sphinx>`_.  The preferred tool for installing
+<https://pypi.org/project/Sphinx/>`_.  The preferred tool for installing
 packages from *PyPI* is :command:`pip`.  This tool is provided with all modern
 versions of Python.
 

doc/usage/restructuredtext/basics.rst

@@ -508,11 +508,14 @@ or this::
 See the :duref:`reST reference for substitutions <substitution-definitions>`
 for details.
 
+.. index:: ! pair: global; substitutions
+
 If you want to use some substitutions for all documents, put them into
-:confval:`rst_prolog` or put them into a separate file and include it into all
+:confval:`rst_prolog` or :confval:`rst_epilog` or put them into a separate file
-documents you want to use them in, using the :rst:dir:`include` directive.  (Be
+and include it into all documents you want to use them in, using the
-sure to give the include file a file name extension differing from that of
+:rst:dir:`include` directive.  (Be sure to give the include file a file name
-other source files, to avoid Sphinx finding it as a standalone document.)
+extension differing from that of other source files, to avoid Sphinx finding it
+as a standalone document.)
 
 Sphinx defines some default substitutions, see :ref:`default-substitutions`.
 

doc/usage/restructuredtext/domains.rst

@@ -1287,18 +1287,18 @@ Jinja_, Operation_, and Scala_.
 
 .. _sphinx-contrib: https://bitbucket.org/birkenfeld/sphinx-contrib/
 
-.. _Ada: https://pypi.python.org/pypi/sphinxcontrib-adadomain
+.. _Ada: https://pypi.org/project/sphinxcontrib-adadomain/
-.. _Chapel: https://pypi.python.org/pypi/sphinxcontrib-chapeldomain
+.. _Chapel: https://pypi.org/project/sphinxcontrib-chapeldomain/
-.. _CoffeeScript: https://pypi.python.org/pypi/sphinxcontrib-coffee
+.. _CoffeeScript: https://pypi.org/project/sphinxcontrib-coffee/
-.. _Common Lisp: https://pypi.python.org/pypi/sphinxcontrib-cldomain
+.. _Common Lisp: https://pypi.org/project/sphinxcontrib-cldomain/
-.. _dqn: https://pypi.python.org/pypi/sphinxcontrib-dqndomain
+.. _dqn: https://pypi.org/project/sphinxcontrib-dqndomain/
-.. _Erlang: https://pypi.python.org/pypi/sphinxcontrib-erlangdomain
+.. _Erlang: https://pypi.org/project/sphinxcontrib-erlangdomain/
-.. _Go: https://pypi.python.org/pypi/sphinxcontrib-golangdomain
+.. _Go: https://pypi.org/project/sphinxcontrib-golangdomain/
-.. _HTTP: https://pypi.python.org/pypi/sphinxcontrib-httpdomain
+.. _HTTP: https://pypi.org/project/sphinxcontrib-httpdomain/
-.. _Jinja: https://pypi.python.org/pypi/sphinxcontrib-jinjadomain
+.. _Jinja: https://pypi.org/project/sphinxcontrib-jinjadomain/
-.. _Lasso: https://pypi.python.org/pypi/sphinxcontrib-lassodomain
+.. _Lasso: https://pypi.org/project/sphinxcontrib-lassodomain/
-.. _MATLAB: https://pypi.python.org/pypi/sphinxcontrib-matlabdomain
+.. _MATLAB: https://pypi.org/project/sphinxcontrib-matlabdomain/
-.. _Operation: https://pypi.python.org/pypi/sphinxcontrib-operationdomain
+.. _Operation: https://pypi.org/project/sphinxcontrib-operationdomain/
-.. _PHP: https://pypi.python.org/pypi/sphinxcontrib-phpdomain
+.. _PHP: https://pypi.org/project/sphinxcontrib-phpdomain/
 .. _Ruby: https://bitbucket.org/birkenfeld/sphinx-contrib/src/default/rubydomain
-.. _Scala: https://pypi.python.org/pypi/sphinxcontrib-scaladomain
+.. _Scala: https://pypi.org/project/sphinxcontrib-scaladomain/

setup.py

@@ -176,7 +176,7 @@ setup(
     name='Sphinx',
     version=sphinx.__version__,
     url='http://sphinx-doc.org/',
-    download_url='https://pypi.python.org/pypi/Sphinx',
+    download_url='https://pypi.org/project/Sphinx/',
     license='BSD',
     author='Georg Brandl',
     author_email='georg@python.org',

sphinx/application.py

@@ -26,14 +26,13 @@ from six.moves import cStringIO
 
 import sphinx
 from sphinx import package_dir, locale
-from sphinx.config import Config
+from sphinx.config import Config, check_unicode
+from sphinx.config import CONFIG_FILENAME  # NOQA # for compatibility (RemovedInSphinx30)
 from sphinx.deprecation import (
     RemovedInSphinx20Warning, RemovedInSphinx30Warning, RemovedInSphinx40Warning
 )
 from sphinx.environment import BuildEnvironment
-from sphinx.errors import (
+from sphinx.errors import ApplicationError, ConfigError, VersionRequirementError
-    ApplicationError, ConfigError, ExtensionError, VersionRequirementError
-)
 from sphinx.events import EventManager
 from sphinx.locale import __
 from sphinx.registry import SphinxComponentRegistry
@@ -110,7 +109,6 @@ builtin_extensions = (
     'alabaster',
 )  # type: Tuple[unicode, ...]
 
-CONFIG_FILENAME = 'conf.py'
 ENV_PICKLE_FILENAME = 'environment.pickle'
 
 logger = logging.getLogger(__name__)
@@ -189,10 +187,11 @@ class Sphinx(object):
 
         # read config
         self.tags = Tags(tags)
-        self.config = Config(self.confdir, CONFIG_FILENAME,
+        if self.confdir is None:
-                             confoverrides or {}, self.tags)
+            self.config = Config({}, confoverrides or {})
-        self.config.check_unicode()
+        else:
-        # defer checking types until i18n has been initialized
+            self.config = Config.read(self.confdir, confoverrides or {}, self.tags)
+            check_unicode(self.config)
 
         # initialize some limited config variables before initialize i18n and loading
         # extensions
@@ -250,8 +249,6 @@ class Sphinx(object):
 
         # create the builder
         self.builder = self.create_builder(buildername)
-        # check all configuration values for permissible types
-        self.config.check_types()
         # set up the build environment
         self._init_env(freshenv)
         # set up the builder
@@ -561,8 +558,6 @@ class Sphinx(object):
         """
         logger.debug('[app] adding config value: %r',
                      (name, default, rebuild) + ((types,) if types else ()))  # type: ignore
-        if name in self.config:
-            raise ExtensionError(__('Config value %r already present') % name)
         if rebuild in (False, True):
             rebuild = rebuild and 'env' or ''
         self.config.add(name, default, rebuild, types)
@@ -970,7 +965,30 @@ class Sphinx(object):
         Add the standard docutils :class:`Transform` subclass *transform* to
         the list of transforms that are applied after Sphinx parses a reST
         document.
-        """
+
+        .. list-table:: priority range categories for Sphinx transforms
+
+           * - Priority
+             - Main purpose in Sphinx
+           * - 0-99
+             - Fix invalid nodes by docutils. Translate a doctree.
+           * - 100-299
+             - Preparation
+           * - 300-399
+             - early
+           * - 400-699
+             - main
+           * - 700-799
+             - Post processing. Deadline to modify text and referencing.
+           * - 800-899
+             - Collect referencing and referenced nodes. Domain processing.
+           * - 900-999
+             - Finalize and clean up.
+
+        refs: `Transform Priority Range Categories`__
+
+        __ http://docutils.sourceforge.net/docs/ref/transforms.html#transform-priority-range-categories
+        """  # NOQA
         self.registry.add_transform(transform)
 
     def add_post_transform(self, transform):

sphinx/builders/htmlhelp.py

@@ -248,6 +248,8 @@ class HTMLHelpBuilder(StandaloneHTMLBuilder):
                 outdir += os.sep
             olen = len(outdir)
             for root, dirs, files in os.walk(outdir):
+                dirs.sort()
+                files.sort()
                 staticdir = root.startswith(path.join(outdir, '_static'))
                 for fn in sorted(files):
                     if (staticdir and not fn.endswith('.js')) or \

sphinx/builders/latex/__init__.py

@@ -25,7 +25,6 @@ from sphinx.environment.adapters.asset import ImageAdapter
 from sphinx.errors import SphinxError, ConfigError
 from sphinx.locale import _, __
 from sphinx.transforms import SphinxTransformer
-from sphinx.transforms.references import SubstitutionDefinitionsRemover
 from sphinx.util import texescape, logging, status_iterator
 from sphinx.util.console import bold, darkgreen  # type: ignore
 from sphinx.util.docutils import new_document
@@ -217,7 +216,6 @@ class LaTeXBuilder(Builder):
         # type: (nodes.document) -> None
         transformer = SphinxTransformer(doctree)
         transformer.set_environment(self.env)
-        transformer.add_transforms([SubstitutionDefinitionsRemover])
         transformer.apply_transforms()
 
     def finish(self):
@@ -325,7 +323,7 @@ def setup(app):
     app.add_config_value('latex_appendices', [], None)
     app.add_config_value('latex_use_latex_multicolumn', False, None)
     app.add_config_value('latex_toplevel_sectioning', None, None,
-                         ENUM('part', 'chapter', 'section'))
+                         ENUM(None, 'part', 'chapter', 'section'))
     app.add_config_value('latex_domain_indices', True, None, [list])
     app.add_config_value('latex_show_urls', 'no', None)
     app.add_config_value('latex_show_pagerefs', False, None)

sphinx/builders/latex/transforms.py

@@ -0,0 +1,161 @@
+# -*- coding: utf-8 -*-
+"""
+    sphinx.builders.latex.transforms
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    Transforms for LaTeX builder.
+
+    :copyright: Copyright 2007-2018 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+from docutils import nodes
+
+from sphinx.transforms import SphinxTransform
+
+if False:
+    # For type annotation
+    from typing import Dict, List, Set, Union # NOQA
+
+URI_SCHEMES = ('mailto:', 'http:', 'https:', 'ftp:')
+
+
+class ShowUrlsTransform(SphinxTransform):
+    """Expand references to inline text or footnotes.
+
+    For more information, see :confval:`latex_show_urls`.
+    """
+    default_priority = 400
+
+    # references are expanded to footnotes (or not)
+    expanded = False
+
+    def apply(self):
+        # type: () -> None
+        # replace id_prefix temporarily
+        id_prefix = self.document.settings.id_prefix
+        self.document.settings.id_prefix = 'show_urls'
+
+        self.expand_show_urls()
+        if self.expanded:
+            self.renumber_footnotes()
+
+        # restore id_prefix
+        self.document.settings.id_prefix = id_prefix
+
+    def expand_show_urls(self):
+        # type: () -> None
+        show_urls = self.document.settings.env.config.latex_show_urls
+        if show_urls is False or show_urls == 'no':
+            return
+
+        for node in self.document.traverse(nodes.reference):
+            uri = node.get('refuri', '')
+            if uri.startswith(URI_SCHEMES):
+                if uri.startswith('mailto:'):
+                    uri = uri[7:]
+                if node.astext() != uri:
+                    index = node.parent.index(node)
+                    if show_urls == 'footnote':
+                        footnote_nodes = self.create_footnote(uri)
+                        for i, fn in enumerate(footnote_nodes):
+                            node.parent.insert(index + i + 1, fn)
+
+                        self.expanded = True
+                    else:  # all other true values (b/w compat)
+                        textnode = nodes.Text(" (%s)" % uri)
+                        node.parent.insert(index + 1, textnode)
+
+    def create_footnote(self, uri):
+        # type: (unicode) -> List[Union[nodes.footnote, nodes.footnote_ref]]
+        label = nodes.label('', '#')
+        para = nodes.paragraph()
+        para.append(nodes.reference('', nodes.Text(uri), refuri=uri, nolinkurl=True))
+        footnote = nodes.footnote(uri, label, para, auto=1)
+        footnote['names'].append('#')
+        self.document.note_autofootnote(footnote)
+
+        label = nodes.Text('#')
+        footnote_ref = nodes.footnote_reference('[#]_', label, auto=1,
+                                                refid=footnote['ids'][0])
+        self.document.note_autofootnote_ref(footnote_ref)
+        footnote.add_backref(footnote_ref['ids'][0])
+
+        return [footnote, footnote_ref]
+
+    def renumber_footnotes(self):
+        # type: () -> None
+        collector = FootnoteCollector(self.document)
+        self.document.walkabout(collector)
+
+        num = 0
+        for document, footnote in collector.auto_footnotes:
+            # search unused footnote number
+            while True:
+                num += 1
+                if str(num) not in collector.used_footnote_numbers:
+                    break
+
+            # assign new footnote number
+            old_label = footnote[0].astext()
+            footnote[0].replace_self(nodes.label('', str(num)))
+            if old_label in footnote['names']:
+                footnote['names'].remove(old_label)
+            footnote['names'].append(str(num))
+
+            # update footnote_references by new footnote number
+            for ref in collector.footnote_refs.get(document, []):
+                if footnote['ids'][0] == ref['refid']:
+                    ref.remove(ref[0])
+                    ref += nodes.Text(str(num))
+
+
+class FootnoteCollector(nodes.NodeVisitor):
+    """Collect footnotes and footnote references on the document"""
+
+    def __init__(self, document):
+        # type: (nodes.document) -> None
+        self.auto_footnotes = []            # type: List[nodes.footnote]
+        self.used_footnote_numbers = set()  # type: Set[unicode]
+        self.footnote_refs = {}             # type: Dict[nodes.Node, List[nodes.footnote_reference]]  # NOQA
+        self.current_document = []          # type: List[nodes.Node]
+        nodes.NodeVisitor.__init__(self, document)
+
+    def visit_document(self, node):
+        # type: (nodes.Node) -> None
+        self.current_document.append(node)
+
+    def depart_document(self, node):
+        # type: (nodes.Node) -> None
+        self.current_document.pop()
+
+    def visit_start_of_file(self, node):
+        # type: (nodes.Node) -> None
+        self.current_document.append(node)
+
+    def depart_start_of_file(self, node):
+        # type: (nodes.Node) -> None
+        self.current_document.pop()
+
+    def unknown_visit(self, node):
+        # type: (nodes.Node) -> None
+        pass
+
+    def visit_footnote(self, node):
+        # type: (nodes.footnote) -> None
+        document = self.current_document[-1]
+        if node.get('auto'):
+            self.auto_footnotes.append((document, node))
+        else:
+            for name in node['names']:
+                self.used_footnote_numbers.add(name)
+
+    def visit_footnote_reference(self, node):
+        # type: (nodes.footnote_reference) -> None
+        document = self.current_document[-1]
+        footnote_refs = self.footnote_refs.setdefault(document, [])
+        footnote_refs.append(node)
+
+    def unknown_departure(self, node):
+        # type: (nodes.Node) -> None
+        pass

sphinx/config.py

@@ -11,13 +11,15 @@
 
 import re
 import traceback
+import warnings
 from collections import OrderedDict
 from os import path, getenv
 from typing import Any, NamedTuple, Union
 
 from six import PY2, PY3, iteritems, string_types, binary_type, text_type, integer_types
 
-from sphinx.errors import ConfigError
+from sphinx.deprecation import RemovedInSphinx30Warning
+from sphinx.errors import ConfigError, ExtensionError
 from sphinx.locale import _, __
 from sphinx.util import logging
 from sphinx.util.i18n import format_date
@@ -26,28 +28,15 @@ from sphinx.util.pycompat import execfile_, NoneType
 
 if False:
     # For type annotation
-    from typing import Any, Callable, Dict, Iterable, Iterator, List, Tuple, Union  # NOQA
+    from typing import Any, Callable, Dict, Generator, Iterator, List, Tuple, Union  # NOQA
     from sphinx.application import Sphinx  # NOQA
     from sphinx.util.tags import Tags  # NOQA
 
 logger = logging.getLogger(__name__)
 
-nonascii_re = re.compile(br'[\x80-\xff]')
+CONFIG_FILENAME = 'conf.py'
 copyright_year_re = re.compile(r'^((\d{4}-)?)(\d{4})(?=[ ,])')
 
-CONFIG_SYNTAX_ERROR = __("There is a syntax error in your configuration file: %s")
-if PY3:
-    CONFIG_SYNTAX_ERROR += __("\nDid you change the syntax from 2.x to 3.x?")
-CONFIG_ERROR = __("There is a programable error in your configuration file:\n\n%s")
-CONFIG_EXIT_ERROR = __("The configuration file (or one of the modules it imports) "
-                       "called sys.exit()")
-CONFIG_ENUM_WARNING = __("The config value `{name}` has to be a one of {candidates}, "
-                         "but `{current}` is given.")
-CONFIG_PERMITTED_TYPE_WARNING = __("The config value `{name}' has type `{current.__name__}', "
-                                   "expected to {permitted}.")
-CONFIG_TYPE_WARNING = __("The config value `{name}' has type `{current.__name__}', "
-                         "defaults to `{default.__name__}'.")
-
 if PY3:
     unicode = str  # special alias for static typing...
 
@@ -144,11 +133,7 @@ class Config(object):
         nitpick_ignore = ([], None),
         numfig = (False, 'env'),
         numfig_secnum_depth = (1, 'env'),
-        numfig_format = ({'section': _('Section %s'),
+        numfig_format = ({}, 'env'),  # will be initialized in init_numfig_format()
-                          'figure': _('Fig. %s'),
-                          'table': _('Table %s'),
-                          'code-block': _('Listing %s')},
-                         'env'),
 
         tls_verify = (True, 'env'),
         tls_cacerts = (None, 'env'),
@@ -159,30 +144,30 @@ class Config(object):
                                 'env'),
     )  # type: Dict[unicode, Tuple]
 
-    def __init__(self, dirname, filename, overrides, tags):
+    def __init__(self, *args):
-        # type: (unicode, unicode, Dict, Tags) -> None
+        # type: (Any) -> None
+        if len(args) == 4:
+            # old style arguments: (dirname, filename, overrides, tags)
+            warnings.warn('The argument of Config() class has been changed. '
+                          'Use Config.read() to read configuration from conf.py.',
+                          RemovedInSphinx30Warning)
+            dirname, filename, overrides, tags = args
+            if dirname is None:
+                config = {}  # type: Dict[unicode, Any]
+            else:
+                config = eval_config_file(path.join(dirname, filename), tags)
+        else:
+            # new style arguments: (config={}, overrides={})
+            if len(args) == 0:
+                config, overrides = {}, {}
+            elif len(args) == 1:
+                config, overrides = args[0], {}
+            else:
+                config, overrides = args[:2]
+
         self.overrides = overrides
         self.values = Config.config_values.copy()
-        config = {}  # type: Dict[unicode, Any]
-        if dirname is not None:
-            config_file = path.join(dirname, filename)
-            config['__file__'] = config_file
-            config['tags'] = tags
-            with cd(dirname):
-                # we promise to have the config dir as current dir while the
-                # config file is executed
-                try:
-                    execfile_(filename, config)
-                except SyntaxError as err:
-                    raise ConfigError(CONFIG_SYNTAX_ERROR % err)
-                except SystemExit:
-                    raise ConfigError(CONFIG_EXIT_ERROR)
-                except Exception:
-                    raise ConfigError(CONFIG_ERROR % traceback.format_exc())
-
         self._raw_config = config
-        # these two must be preinitialized because extensions can add their
-        # own config values
         self.setup = config.get('setup', None)  # type: Callable
 
         if 'extensions' in overrides:
@@ -192,69 +177,25 @@ class Config(object):
                 config['extensions'] = overrides.pop('extensions')
         self.extensions = config.get('extensions', [])  # type: List[unicode]
 
-        # correct values of copyright year that are not coherent with
+    @classmethod
-        # the SOURCE_DATE_EPOCH environment variable (if set)
+    def read(cls, confdir, overrides=None, tags=None):
-        # See https://reproducible-builds.org/specs/source-date-epoch/
+        # type: (unicode, Dict, Tags) -> Config
-        if getenv('SOURCE_DATE_EPOCH') is not None:
+        """Create a Config object from configuration file."""
-            for k in ('copyright', 'epub_copyright'):
+        filename = path.join(confdir, CONFIG_FILENAME)
-                if k in config:
+        namespace = eval_config_file(filename, tags)
-                    config[k] = copyright_year_re.sub(r'\g<1>%s' % format_date('%Y'),
+        return cls(namespace, overrides or {})
-                                                      config[k])
 
     def check_types(self):
         # type: () -> None
-        # check all values for deviation from the default value's type, since
+        warnings.warn('Config.check_types() is deprecated. Use check_confval_types() instead.',
-        # that can result in TypeErrors all over the place
+                      RemovedInSphinx30Warning)
-        # NB. since config values might use _() we have to wait with calling
+        check_confval_types(None, self)
-        # this method until i18n is initialized
-        for name in self._raw_config:
-            if name not in self.values:
-                continue  # we don't know a default value
-            settings = self.values[name]
-            default, dummy_rebuild = settings[:2]
-            permitted = settings[2] if len(settings) == 3 else ()
-
-            if hasattr(default, '__call__'):
-                default = default(self)  # could invoke _()
-            if default is None and not permitted:
-                continue  # neither inferrable nor expliclitly permitted types
-            current = self[name]
-            if permitted is Any:
-                # any type of value is accepted
-                pass
-            elif isinstance(permitted, ENUM):
-                if not permitted.match(current):
-                    logger.warning(CONFIG_ENUM_WARNING.format(
-                        name=name, current=current, candidates=permitted.candidates))
-            else:
-                if type(current) is type(default):
-                    continue
-                if type(current) in permitted:
-                    continue
-
-                common_bases = (set(type(current).__bases__ + (type(current),)) &
-                                set(type(default).__bases__))
-                common_bases.discard(object)
-                if common_bases:
-                    continue  # at least we share a non-trivial base class
-
-                if permitted:
-                    logger.warning(CONFIG_PERMITTED_TYPE_WARNING.format(
-                        name=name, current=type(current),
-                        permitted=str([cls.__name__ for cls in permitted])))
-                else:
-                    logger.warning(CONFIG_TYPE_WARNING.format(
-                        name=name, current=type(current), default=type(default)))
 
     def check_unicode(self):
         # type: () -> None
-        # check all string values for non-ASCII characters in bytestrings,
+        warnings.warn('Config.check_unicode() is deprecated. Use check_unicode() instead.',
-        # since that can result in UnicodeErrors all over the place
+                      RemovedInSphinx30Warning)
-        for name, value in iteritems(self._raw_config):
+        check_unicode(self)
-            if isinstance(value, binary_type) and nonascii_re.search(value):
-                logger.warning(__('the config value %r is set to a string with non-ASCII '
-                                  'characters; this can lead to Unicode errors occurring. '
-                                  'Please use Unicode strings, e.g. %r.'), name, u'Content')
 
     def convert_overrides(self, name, value):
         # type: (unicode, Any) -> Any
@@ -350,19 +291,49 @@ class Config(object):
         return name in self.values
 
     def __iter__(self):
-        # type: () -> Iterable[ConfigValue]
+        # type: () -> Generator[ConfigValue, None, None]
         for name, value in iteritems(self.values):
             yield ConfigValue(name, getattr(self, name), value[1])  # type: ignore
 
     def add(self, name, default, rebuild, types):
         # type: (unicode, Any, Union[bool, unicode], Any) -> None
+        if name in self.values:
+            raise ExtensionError(__('Config value %r already present') % name)
+        else:
             self.values[name] = (default, rebuild, types)
 
     def filter(self, rebuild):
         # type: (Union[unicode, List[unicode]]) -> Iterator[ConfigValue]
         if isinstance(rebuild, string_types):
             rebuild = [rebuild]
-        return (value for value in self if value.rebuild in rebuild)  # type: ignore
+        return (value for value in self if value.rebuild in rebuild)
+
+
+def eval_config_file(filename, tags):
+    # type: (unicode, Tags) -> Dict[unicode, Any]
+    """Evaluate a config file."""
+    namespace = {}  # type: Dict[unicode, Any]
+    namespace['__file__'] = filename
+    namespace['tags'] = tags
+
+    with cd(path.dirname(filename)):
+        # during executing config file, current dir is changed to ``confdir``.
+        try:
+            execfile_(filename, namespace)
+        except SyntaxError as err:
+            msg = __("There is a syntax error in your configuration file: %s")
+            if PY3:
+                msg += __("\nDid you change the syntax from 2.x to 3.x?")
+            raise ConfigError(msg % err)
+        except SystemExit:
+            msg = __("The configuration file (or one of the modules it imports) "
+                     "called sys.exit()")
+            raise ConfigError(msg)
+        except Exception:
+            msg = __("There is a programable error in your configuration file:\n\n%s")
+            raise ConfigError(msg % traceback.format_exc())
+
+    return namespace
 
 
 def convert_source_suffix(app, config):
@@ -391,9 +362,104 @@ def convert_source_suffix(app, config):
                           "But `%r' is given." % source_suffix))
 
 
+def init_numfig_format(app, config):
+    # type: (Sphinx, Config) -> None
+    """Initialize :confval:`numfig_format`."""
+    numfig_format = {'section': _('Section %s'),
+                     'figure': _('Fig. %s'),
+                     'table': _('Table %s'),
+                     'code-block': _('Listing %s')}
+
+    # override default labels by configuration
+    numfig_format.update(config.numfig_format)
+    config.numfig_format = numfig_format  # type: ignore
+
+
+def correct_copyright_year(app, config):
+    # type: (Sphinx, Config) -> None
+    """correct values of copyright year that are not coherent with
+    the SOURCE_DATE_EPOCH environment variable (if set)
+
+    See https://reproducible-builds.org/specs/source-date-epoch/
+    """
+    if getenv('SOURCE_DATE_EPOCH') is not None:
+        for k in ('copyright', 'epub_copyright'):
+            if k in config:
+                replace = r'\g<1>%s' % format_date('%Y')
+                config[k] = copyright_year_re.sub(replace, config[k])  # type: ignore
+
+
+def check_confval_types(app, config):
+    # type: (Sphinx, Config) -> None
+    """check all values for deviation from the default value's type, since
+    that can result in TypeErrors all over the place NB.
+    """
+    for confval in config:
+        settings = config.values[confval.name]
+        default = settings[0]
+        annotations = settings[2] if len(settings) == 3 else ()
+
+        if hasattr(default, '__call__'):
+            default = default(config)  # evaluate default value
+        if default is None and not annotations:
+            continue  # neither inferrable nor expliclitly annotated types
+
+        if annotations is Any:
+            # any type of value is accepted
+            pass
+        elif isinstance(annotations, ENUM):
+            if not annotations.match(confval.value):
+                msg = __("The config value `{name}` has to be a one of {candidates}, "
+                         "but `{current}` is given.")
+                logger.warning(msg.format(name=confval.name,
+                                          current=confval.value,
+                                          candidates=annotations.candidates))
+        else:
+            if type(confval.value) is type(default):
+                continue
+            if type(confval.value) in annotations:
+                continue
+
+            common_bases = (set(type(confval.value).__bases__ + (type(confval.value),)) &
+                            set(type(default).__bases__))
+            common_bases.discard(object)
+            if common_bases:
+                continue  # at least we share a non-trivial base class
+
+            if annotations:
+                msg = __("The config value `{name}' has type `{current.__name__}', "
+                         "expected to {permitted}.")
+                logger.warning(msg.format(name=confval.name,
+                                          current=type(confval.value),
+                                          permitted=str([c.__name__ for c in annotations])))
+            else:
+                msg = __("The config value `{name}' has type `{current.__name__}', "
+                         "defaults to `{default.__name__}'.")
+                logger.warning(msg.format(name=confval.name,
+                                          current=type(confval.value),
+                                          default=type(default)))
+
+
+def check_unicode(config):
+    # type: (Config) -> None
+    """check all string values for non-ASCII characters in bytestrings,
+    since that can result in UnicodeErrors all over the place
+    """
+    nonascii_re = re.compile(br'[\x80-\xff]')
+
+    for name, value in iteritems(config._raw_config):
+        if isinstance(value, binary_type) and nonascii_re.search(value):
+            logger.warning(__('the config value %r is set to a string with non-ASCII '
+                              'characters; this can lead to Unicode errors occurring. '
+                              'Please use Unicode strings, e.g. %r.'), name, u'Content')
+
+
 def setup(app):
     # type: (Sphinx) -> Dict[unicode, Any]
     app.connect('config-inited', convert_source_suffix)
+    app.connect('config-inited', init_numfig_format)
+    app.connect('config-inited', correct_copyright_year)
+    app.connect('config-inited', check_confval_types)
 
     return {
         'version': 'builtin',

sphinx/directives/other.py

@@ -7,6 +7,9 @@
     :license: BSD, see LICENSE for details.
 """
 
+import re
+from contextlib import contextmanager
+
 from docutils import nodes
 from docutils.parsers.rst import directives
 from docutils.parsers.rst.directives.admonitions import BaseAdmonition
@@ -42,6 +45,8 @@ locale.versionlabels = DeprecatedDict(
     RemovedInSphinx30Warning
 )
 
+glob_re = re.compile('.*[*?\[].*')
+
 
 def int_or_nothing(argument):
     # type: (unicode) -> int
@@ -73,29 +78,50 @@ class TocTree(SphinxDirective):
 
     def run(self):
         # type: () -> List[nodes.Node]
-        suffixes = self.config.source_suffix
+        subnode = addnodes.toctree()
-        glob = 'glob' in self.options
+        subnode['parent'] = self.env.docname
 
-        ret = []
         # (title, ref) pairs, where ref may be a document, or an external link,
         # and title may be None if the document's title is to be used
-        entries = []        # type: List[Tuple[unicode, unicode]]
+        subnode['entries'] = []
-        includefiles = []
+        subnode['includefiles'] = []
+        subnode['maxdepth'] = self.options.get('maxdepth', -1)
+        subnode['caption'] = self.options.get('caption')
+        subnode['glob'] = 'glob' in self.options
+        subnode['hidden'] = 'hidden' in self.options
+        subnode['includehidden'] = 'includehidden' in self.options
+        subnode['numbered'] = self.options.get('numbered', 0)
+        subnode['titlesonly'] = 'titlesonly' in self.options
+        set_source_info(self, subnode)
+        wrappernode = nodes.compound(classes=['toctree-wrapper'])
+        wrappernode.append(subnode)
+        self.add_name(wrappernode)
+
+        ret = self.parse_content(subnode)
+        ret.append(wrappernode)
+        return ret
+
+    def parse_content(self, toctree):
+        suffixes = self.config.source_suffix
+
+        # glob target documents
         all_docnames = self.env.found_docs.copy()
-        # don't add the currently visited file in catch-all patterns
+        all_docnames.remove(self.env.docname)  # remove current document
-        all_docnames.remove(self.env.docname)
+
+        ret = []
         for entry in self.content:
             if not entry:
                 continue
             # look for explicit titles ("Some Title <document>")
             explicit = explicit_title_re.match(entry)
-            if glob and ('*' in entry or '?' in entry or '[' in entry) and not explicit:
+            if (toctree['glob'] and glob_re.match(entry) and
+                    not explicit and not url_re.match(entry)):
                 patname = docname_join(self.env.docname, entry)
-                docnames = sorted(patfilter(all_docnames, patname))  # type: ignore
+                docnames = sorted(patfilter(all_docnames, patname))
                 for docname in docnames:
                     all_docnames.remove(docname)  # don't include it again
-                    entries.append((None, docname))
+                    toctree['entries'].append((None, docname))
-                    includefiles.append(docname)
+                    toctree['includefiles'].append(docname)
                 if not docnames:
                     ret.append(self.state.document.reporter.warning(
                         'toctree glob pattern %r didn\'t match any documents'
@@ -116,7 +142,7 @@ class TocTree(SphinxDirective):
                 # absolutize filenames
                 docname = docname_join(self.env.docname, docname)
                 if url_re.match(ref) or ref == 'self':
-                    entries.append((title, ref))
+                    toctree['entries'].append((title, ref))
                 elif docname not in self.env.found_docs:
                     ret.append(self.state.document.reporter.warning(
                         'toctree contains reference to nonexisting '
@@ -124,28 +150,13 @@ class TocTree(SphinxDirective):
                     self.env.note_reread()
                 else:
                     all_docnames.discard(docname)
-                    entries.append((title, docname))
+                    toctree['entries'].append((title, docname))
-                    includefiles.append(docname)
+                    toctree['includefiles'].append(docname)
-        subnode = addnodes.toctree()
+
-        subnode['parent'] = self.env.docname
         # entries contains all entries (self references, external links etc.)
         if 'reversed' in self.options:
-            entries.reverse()
+            toctree['entries'] = list(reversed(toctree['entries']))
-        subnode['entries'] = entries
+
-        # includefiles only entries that are documents
-        subnode['includefiles'] = includefiles
-        subnode['maxdepth'] = self.options.get('maxdepth', -1)
-        subnode['caption'] = self.options.get('caption')
-        subnode['glob'] = glob
-        subnode['hidden'] = 'hidden' in self.options
-        subnode['includehidden'] = 'includehidden' in self.options
-        subnode['numbered'] = self.options.get('numbered', 0)
-        subnode['titlesonly'] = 'titlesonly' in self.options
-        set_source_info(self, subnode)
-        wrappernode = nodes.compound(classes=['toctree-wrapper'])
-        wrappernode.append(subnode)
-        self.add_name(wrappernode)
-        ret.append(wrappernode)
         return ret
 
 
@@ -426,6 +437,7 @@ class Include(BaseInclude, SphinxDirective):
 
     def run(self):
         # type: () -> List[nodes.Node]
+        current_filename = self.env.doc2path(self.env.docname)
         if self.arguments[0].startswith('<') and \
            self.arguments[0].endswith('>'):
             # docutils "standard" includes, do not do path processing
@@ -433,9 +445,29 @@ class Include(BaseInclude, SphinxDirective):
         rel_filename, filename = self.env.relfn2path(self.arguments[0])
         self.arguments[0] = filename
         self.env.note_included(filename)
+        with patched_warnings(self, current_filename):
             return BaseInclude.run(self)
 
 
+@contextmanager
+def patched_warnings(directive, parent_filename):
+    # type: (BaseInclude, unicode) -> Generator[None, None, None]
+    """Add includee filename to the warnings during inclusion."""
+    try:
+        original = directive.state_machine.insert_input
+
+        def insert_input(input_lines, source):
+            # type: (Any, unicode) -> None
+            source += ' <included from %s>' % parent_filename
+            original(input_lines, source)
+
+        # patch insert_input() temporarily
+        directive.state_machine.insert_input = insert_input
+        yield
+    finally:
+        directive.state_machine.insert_input = original
+
+
 def setup(app):
     # type: (Sphinx) -> Dict[unicode, Any]
     directives.register_directive('toctree', TocTree)

sphinx/domains/python.py

@@ -579,9 +579,10 @@ class PyModule(Directive):
         env.ref_context['py:module'] = modname
         ret = []
         if not noindex:
-            env.domaindata['py']['modules'][modname] = \
+            env.domaindata['py']['modules'][modname] = (env.docname,
-                (env.docname, self.options.get('synopsis', ''),
+                                                        self.options.get('synopsis', ''),
-                 self.options.get('platform', ''), 'deprecated' in self.options)
+                                                        self.options.get('platform', ''),
+                                                        'deprecated' in self.options)
             # make a duplicate entry in 'objects' to facilitate searching for
             # the module in PythonDomain.find_obj()
             env.domaindata['py']['objects'][modname] = (env.docname, 'module')

sphinx/environment/__init__.py

@@ -21,10 +21,10 @@ from os import path
 
 from docutils.frontend import OptionParser
 from docutils.utils import Reporter, get_source_line
-from six import BytesIO, itervalues, class_types, next
+from six import BytesIO, class_types, next
 from six.moves import cPickle as pickle
 
-from sphinx import addnodes, versioning
+from sphinx import addnodes
 from sphinx.deprecation import RemovedInSphinx20Warning, RemovedInSphinx30Warning
 from sphinx.environment.adapters.indexentries import IndexEntries
 from sphinx.environment.adapters.toctree import TocTree
@@ -570,14 +570,6 @@ class BuildEnvironment(object):
         with sphinx_domains(self), rst.default_role(docname, self.config.default_role):
             doctree = read_doc(self.app, self, self.doc2path(docname))
 
-        # post-processing
-        for domain in itervalues(self.domains):
-            domain.process_doc(self, docname, doctree)
-
-        # allow extension-specific post-processing
-        if app:
-            app.emit('doctree-read', doctree)
-
         # store time of reading, for outdated files detection
         # (Some filesystems have coarse timestamp resolution;
         # therefore time.time() can be older than filesystem's timestamp.
@@ -585,10 +577,6 @@ class BuildEnvironment(object):
         self.all_docs[docname] = max(
             time.time(), path.getmtime(self.doc2path(docname)))
 
-        if self.versioning_condition:
-            # add uids for versioning
-            versioning.prepare(doctree)
-
         # cleanup
         self.temp_data.clear()
         self.ref_context.clear()

sphinx/ext/autodoc/__init__.py

@@ -20,8 +20,8 @@ from docutils.statemachine import ViewList
 from six import iteritems, itervalues, text_type, class_types, string_types
 
 import sphinx
-from sphinx.application import ExtensionError
 from sphinx.deprecation import RemovedInSphinx20Warning
+from sphinx.errors import ExtensionError
 from sphinx.ext.autodoc.importer import mock, import_object, get_object_members
 from sphinx.ext.autodoc.importer import _MockImporter  # to keep compatibility  # NOQA
 from sphinx.ext.autodoc.inspector import format_annotation, formatargspec  # to keep compatibility  # NOQA

sphinx/ext/ifconfig.py

@@ -57,7 +57,7 @@ class IfConfig(Directive):
 
 def process_ifconfig_nodes(app, doctree, docname):
     # type: (Sphinx, nodes.Node, unicode) -> None
-    ns = dict((confval.name, confval.value) for confval in app.config)  # type: ignore
+    ns = dict((confval.name, confval.value) for confval in app.config)
     ns.update(app.config.__dict__.copy())
     ns['builder'] = app.builder.name
     for node in doctree.traverse(ifconfig):

sphinx/ext/jsmath.py

@@ -13,7 +13,7 @@
 from docutils import nodes
 
 import sphinx
-from sphinx.application import ExtensionError
+from sphinx.errors import ExtensionError
 from sphinx.ext.mathbase import get_node_equation_number
 from sphinx.ext.mathbase import setup_math as mathbase_setup
 from sphinx.locale import _
@@ -26,7 +26,7 @@ if False:
 
 def html_visit_math(self, node):
     # type: (nodes.NodeVisitor, nodes.Node) -> None
-    self.body.append(self.starttag(node, 'span', '', CLASS='math notranslate'))
+    self.body.append(self.starttag(node, 'span', '', CLASS='math notranslate nohighlight'))
     self.body.append(self.encode(node['latex']) + '</span>')
     raise nodes.SkipNode
 
@@ -34,7 +34,7 @@ def html_visit_math(self, node):
 def html_visit_displaymath(self, node):
     # type: (nodes.NodeVisitor, nodes.Node) -> None
     if node['nowrap']:
-        self.body.append(self.starttag(node, 'div', CLASS='math notranslate'))
+        self.body.append(self.starttag(node, 'div', CLASS='math notranslate nohighlight'))
         self.body.append(self.encode(node['latex']))
         self.body.append('</div>')
         raise nodes.SkipNode
@@ -47,7 +47,7 @@ def html_visit_displaymath(self, node):
                 self.body.append('<span class="eqno">(%s)' % number)
                 self.add_permalink_ref(node, _('Permalink to this equation'))
                 self.body.append('</span>')
-            self.body.append(self.starttag(node, 'div', CLASS='math notranslate'))
+            self.body.append(self.starttag(node, 'div', CLASS='math notranslate nohighlight'))
         else:
             # but only once!
             self.body.append('<div class="math">')

sphinx/ext/mathjax.py

@@ -27,7 +27,7 @@ if False:
 
 def html_visit_math(self, node):
     # type: (nodes.NodeVisitor, nodes.Node) -> None
-    self.body.append(self.starttag(node, 'span', '', CLASS='math notranslate'))
+    self.body.append(self.starttag(node, 'span', '', CLASS='math notranslate nohighlight'))
     self.body.append(self.builder.config.mathjax_inline[0] +
                      self.encode(node['latex']) +
                      self.builder.config.mathjax_inline[1] + '</span>')
@@ -36,7 +36,7 @@ def html_visit_math(self, node):
 
 def html_visit_displaymath(self, node):
     # type: (nodes.NodeVisitor, nodes.Node) -> None
-    self.body.append(self.starttag(node, 'div', CLASS='math notranslate'))
+    self.body.append(self.starttag(node, 'div', CLASS='math notranslate nohighlight'))
     if node['nowrap']:
         self.body.append(self.encode(node['latex']))
         self.body.append('</div>')

sphinx/io.py

@@ -24,15 +24,17 @@ from sphinx.transforms import (
     ApplySourceWorkaround, ExtraTranslatableNodes, CitationReferences,
     DefaultSubstitutions, MoveModuleTargets, HandleCodeBlocks, SortIds,
     AutoNumbering, AutoIndexUpgrader, FilterSystemMessages,
-    UnreferencedFootnotesDetector, SphinxSmartQuotes, ManpageLink
+    UnreferencedFootnotesDetector, SphinxSmartQuotes, DoctreeReadEvent, ManpageLink
 )
 from sphinx.transforms import SphinxTransformer
 from sphinx.transforms.compact_bullet_list import RefOnlyBulletListTransform
 from sphinx.transforms.i18n import (
     PreserveTranslatableMessages, Locale, RemoveTranslatableInline,
 )
+from sphinx.transforms.references import SphinxDomains, SubstitutionDefinitionsRemover
 from sphinx.util import logging
 from sphinx.util.docutils import LoggingReporter
+from sphinx.versioning import UIDTransform
 
 if False:
     # For type annotation
@@ -93,7 +95,9 @@ class SphinxStandaloneReader(SphinxBaseReader):
                   Locale, CitationReferences, DefaultSubstitutions, MoveModuleTargets,
                   HandleCodeBlocks, AutoNumbering, AutoIndexUpgrader, SortIds,
                   RemoveTranslatableInline, FilterSystemMessages, RefOnlyBulletListTransform,
-                  UnreferencedFootnotesDetector, SphinxSmartQuotes, ManpageLink
+                  UnreferencedFootnotesDetector, SphinxSmartQuotes, ManpageLink,
+                  SphinxDomains, SubstitutionDefinitionsRemover, DoctreeReadEvent,
+                  UIDTransform,
                   ]  # type: List[Transform]
 
     def __init__(self, app, *args, **kwargs):
@@ -116,7 +120,8 @@ class SphinxI18nReader(SphinxBaseReader):
                   DefaultSubstitutions, MoveModuleTargets, HandleCodeBlocks,
                   AutoNumbering, SortIds, RemoveTranslatableInline,
                   FilterSystemMessages, RefOnlyBulletListTransform,
-                  UnreferencedFootnotesDetector, SphinxSmartQuotes, ManpageLink]
+                  UnreferencedFootnotesDetector, SphinxSmartQuotes, ManpageLink,
+                  SubstitutionDefinitionsRemover]
 
     def set_lineno_for_reporter(self, lineno):
         # type: (int) -> None

sphinx/locale/zh_CN/LC_MESSAGES/sphinx.po

@@ -425,7 +425,7 @@ msgstr "行规范 %r：未能从包含文件 %r 中拉取行"
 
 #: sphinx/directives/other.py:157
 msgid "Section author: "
-msgstr "节作者："
+msgstr "章节作者："
 
 #: sphinx/directives/other.py:159
 msgid "Module author: "
@@ -878,7 +878,7 @@ msgstr "格式转换程序退出并报错：\n[stderr]\n%s\n[stdout]\n%s"
 
 #: sphinx/ext/imgmath.py:338 sphinx/ext/jsmath.py:41 sphinx/ext/mathjax.py:42
 msgid "Permalink to this equation"
-msgstr "永久链接至公式"
+msgstr "公式的永久链接"
 
 #: sphinx/ext/intersphinx.py:339
 #, python-format
@@ -1047,7 +1047,7 @@ msgstr "搜索"
 
 #: sphinx/themes/agogo/layout.html:54 sphinx/themes/basic/searchbox.html:16
 msgid "Go"
-msgstr "转向"
+msgstr "搜"
 
 #: sphinx/themes/agogo/layout.html:81 sphinx/themes/basic/sourcelink.html:15
 msgid "Show Source"
@@ -1250,13 +1250,13 @@ msgstr "其他更改"
 #: sphinx/writers/html.py:410 sphinx/writers/html5.py:351
 #: sphinx/writers/html5.py:356
 msgid "Permalink to this headline"
-msgstr "永久链接至标题"
+msgstr "标题的永久链接"
 
 #: sphinx/themes/basic/static/doctools.js_t:199 sphinx/writers/html.py:126
 #: sphinx/writers/html.py:137 sphinx/writers/html5.py:95
 #: sphinx/writers/html5.py:106
 msgid "Permalink to this definition"
-msgstr "永久链接至目标"
+msgstr "定义的永久链接"
 
 #: sphinx/themes/basic/static/doctools.js_t:232
 msgid "Hide Search Matches"
@@ -1313,19 +1313,19 @@ msgstr "当添加指令类时，不应该给定额外参数"
 
 #: sphinx/writers/html.py:414 sphinx/writers/html5.py:360
 msgid "Permalink to this table"
-msgstr "永久链接至表格"
+msgstr "表格的永久链接"
 
 #: sphinx/writers/html.py:466 sphinx/writers/html5.py:412
 msgid "Permalink to this code"
-msgstr "永久链接至代码"
+msgstr "代码的永久链接"
 
 #: sphinx/writers/html.py:470 sphinx/writers/html5.py:416
 msgid "Permalink to this image"
-msgstr "永久链接至图片"
+msgstr "图片的永久链接"
 
 #: sphinx/writers/html.py:472 sphinx/writers/html5.py:418
 msgid "Permalink to this toctree"
-msgstr "永久链接至目录树"
+msgstr "目录的永久链接"
 
 #: sphinx/writers/latex.py:554
 msgid "Release"

sphinx/locale/zh_TW/LC_MESSAGES/sphinx.po

@@ -1046,7 +1046,7 @@ msgstr "搜尋"
 
 #: sphinx/themes/agogo/layout.html:54 sphinx/themes/basic/searchbox.html:16
 msgid "Go"
-msgstr "前往"
+msgstr "搜"
 
 #: sphinx/themes/agogo/layout.html:81 sphinx/themes/basic/sourcelink.html:15
 msgid "Show Source"

sphinx/registry.py

@@ -471,6 +471,7 @@ class SphinxComponentRegistry(object):
             logger.warning(__('extension %r returned an unsupported object from '
                               'its setup() function; it should return None or a '
                               'metadata dictionary'), extname)
+            metadata = {}
 
         app.extensions[extname] = Extension(extname, mod, **metadata)
         app._setting_up_extension.pop()

sphinx/search/de.py

@@ -318,4 +318,4 @@ class SearchGerman(SearchLanguage):
 
     def stem(self, word):
         # type: (unicode) -> unicode
-        return self.stemmer.stemWord(word)
+        return self.stemmer.stemWord(word.lower())

sphinx/search/ja.py

@@ -20,6 +20,7 @@
 import os
 import re
 import sys
+import warnings
 
 from six import iteritems, PY3
 
@@ -35,6 +36,7 @@ try:
 except ImportError:
     janome_module = False
 
+from sphinx.deprecation import RemovedInSphinx30Warning
 from sphinx.errors import SphinxError, ExtensionError
 from sphinx.search import SearchLanguage
 from sphinx.util import import_object
@@ -556,9 +558,12 @@ class SearchJapanese(SearchLanguage):
 
     def init(self, options):
         # type: (Dict) -> None
-        type = options.get('type', 'default')
+        type = options.get('type', 'sphinx.search.ja.DefaultSplitter')
         if type in self.splitters:
             dotted_path = self.splitters[type]
+            warnings.warn('html_search_options["type"]: %s is deprecated. '
+                          'Please give "%s" instead.' % (type, dotted_path),
+                          RemovedInSphinx30Warning)
         else:
             dotted_path = type
         try:

sphinx/templates/quickstart/conf.py_t

@@ -74,7 +74,7 @@ language = {{ language | repr }}
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
-# This pattern also affects html_static_path and html_extra_path .
+# This pattern also affects html_static_path and html_extra_path.
 exclude_patterns = [{{ exclude_patterns }}]
 
 # The name of the Pygments (syntax highlighting) style to use.

sphinx/texinputs/sphinx.sty

@@ -1578,6 +1578,7 @@
 \protected\def\sphinxtablecontinued#1{\textsf{#1}}
 \protected\def\sphinxtitleref#1{\emph{#1}}
 \protected\def\sphinxmenuselection#1{\emph{#1}}
+\protected\def\sphinxguilabel#1{\emph{#1}}
 \protected\def\sphinxaccelerator#1{\underline{#1}}
 \protected\def\sphinxcrossref#1{\emph{#1}}
 \protected\def\sphinxtermref#1{\emph{#1}}

sphinx/themes/basic/static/doctools.js_t

@@ -70,7 +70,9 @@ jQuery.fn.highlightText = function(text, className) {
     if (node.nodeType === 3) {
       var val = node.nodeValue;
       var pos = val.toLowerCase().indexOf(text);
-      if (pos >= 0 && !jQuery(node.parentNode).hasClass(className)) {
+      if (pos >= 0 &&
+          !jQuery(node.parentNode).hasClass(className) &&
+          !jQuery(node.parentNode).hasClass("nohighlight")) {
         var span;
         var isInSVG = jQuery(node).closest("body, svg, foreignObject").is("svg");
         if (isInSVG) {

sphinx/transforms/__init__.py

@@ -343,6 +343,8 @@ class SphinxSmartQuotes(SmartQuotes, SphinxTransform):
 
     refs: sphinx.parsers.RSTParser
     """
+    default_priority = 750
+
     def apply(self):
         # type: () -> None
         if not self.is_available():
@@ -397,6 +399,15 @@ class SphinxSmartQuotes(SmartQuotes, SphinxTransform):
             yield (texttype[notsmartquotable], txtnode.astext())
 
 
+class DoctreeReadEvent(SphinxTransform):
+    """Emit :event:`doctree-read` event."""
+    default_priority = 880
+
+    def apply(self):
+        # type: () -> None
+        self.app.emit('doctree-read', self.document)
+
+
 class ManpageLink(SphinxTransform):
     """Find manpage section numbers and names"""
     default_priority = 999

sphinx/transforms/post_transforms/images.py

@@ -165,19 +165,37 @@ def get_filename_for(filename, mimetype):
 
 
 class ImageConverter(BaseImageConverter):
-    """A base class images converter.
+    """A base class for image converters.
 
-    The concrete image converters should derive this class and
+    An image converter is kind of Docutils transform module.  It is used to
-    overrides the following methods and attributes:
+    convert image files which does not supported by builder to appropriate
+    format for that builder.
 
-    * default_priority (if needed)
+    For example, :py:class:`LaTeX builder <.LaTeXBuilder>` supports PDF,
-    * conversion_rules
+    PNG and JPEG as image formats.  However it does not support SVG images.
-    * is_available()
+    For such case, to use image converters allows to embed these
-    * convert()
+    unsupported images into the document.  One of image converters;
+    :ref:`sphinx.ext.imgconverter <sphinx.ext.imgconverter>` can convert
+    a SVG image to PNG format using Imagemagick internally.
+
+    There are three steps to make your custom image converter:
+
+    1. Make a subclass of ``ImageConverter`` class
+    2. Override ``conversion_rules``, ``is_available()`` and ``convert()``
+    3. Register your image converter to Sphinx using
+       :py:meth:`.Sphinx.add_post_transform`
     """
     default_priority = 200
 
-    #: A conversion rules between two mimetypes which this converters supports
+    #: A conversion rules the image converter supports.
+    #: It is represented as a list of pair of source image format (mimetype) and
+    #: destination one::
+    #:
+    #:     conversion_rules = [
+    #:         ('image/svg+xml', 'image/png'),
+    #:         ('image/gif', 'image/png'),
+    #:         ('application/pdf', 'image/png'),
+    #:     ]
     conversion_rules = []  # type: List[Tuple[unicode, unicode]]
 
     def __init__(self, *args, **kwargs):
@@ -216,7 +234,7 @@ class ImageConverter(BaseImageConverter):
 
     def is_available(self):
         # type: () -> bool
-        """Confirms the converter is available or not."""
+        """Return the image converter is available or not."""
         raise NotImplementedError()
 
     def guess_mimetypes(self, node):
@@ -255,7 +273,11 @@ class ImageConverter(BaseImageConverter):
 
     def convert(self, _from, _to):
         # type: (unicode, unicode) -> bool
-        """Converts the image to expected one."""
+        """Convert a image file to expected format.
+
+        *_from* is a path for source image file, and *_to* is a path for
+        destination file.
+        """
         raise NotImplementedError()
 
 

sphinx/transforms/references.py

@@ -11,15 +11,13 @@
 
 from docutils import nodes
 from docutils.transforms.references import Substitutions
+from six import itervalues
 
 from sphinx.transforms import SphinxTransform
 
 
 class SubstitutionDefinitionsRemover(SphinxTransform):
-    """Remove ``substitution_definition node from doctrees.
+    """Remove ``substitution_definition node from doctrees."""
-
-    .. note:: In Sphinx-1.7, this transform is only used in LaTeX builder.
-    """
 
     # should be invoked after Substitutions process
     default_priority = Substitutions.default_priority + 1
@@ -28,3 +26,13 @@ class SubstitutionDefinitionsRemover(SphinxTransform):
         # type: () -> None
         for node in self.document.traverse(nodes.substitution_definition):
             node.parent.remove(node)
+
+
+class SphinxDomains(SphinxTransform):
+    """Collect objects to Sphinx domains for cross references."""
+    default_priority = 850
+
+    def apply(self):
+        # type: () -> None
+        for domain in itervalues(self.env.domains):
+            domain.process_doc(self, self.env.docname, self.document)

sphinx/util/logging.py

@@ -62,13 +62,13 @@ def getLogger(name):
     # type: (str) -> SphinxLoggerAdapter
     """Get logger wrapped by :class:`sphinx.util.logging.SphinxLoggerAdapter`.
 
-    Sphinx logger always uses ``sphinx.*`` namesapce to be independent from
+    Sphinx logger always uses ``sphinx.*`` namespace to be independent from
-    settings of root logger.  It ensure logging is consistent even if a
+    settings of root logger.  It ensures logging is consistent even if a
     third-party extension or imported application resets logger settings.
 
     Example usage::
 
-        >>> from sphinx.utils import logging
+        >>> from sphinx.util import logging
         >>> logger = logging.getLogger(__name__)
         >>> logger.info('Hello, this is an extension!')
         Hello, this is an extension!

sphinx/versioning.py

@@ -9,6 +9,7 @@
     :copyright: Copyright 2007-2018 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
+import warnings
 from itertools import product
 from operator import itemgetter
 from uuid import uuid4
@@ -17,6 +18,7 @@ from six import iteritems
 from six.moves import cPickle as pickle
 from six.moves import range, zip_longest
 
+from sphinx.deprecation import RemovedInSphinx30Warning
 from sphinx.transforms import SphinxTransform
 
 if False:
@@ -155,12 +157,15 @@ def levenshtein_distance(a, b):
 
 class UIDTransform(SphinxTransform):
     """Add UIDs to doctree for versioning."""
-    default_priority = 100
+    default_priority = 880
 
     def apply(self):
         # type: () -> None
         env = self.env
         old_doctree = None
+        if not env.versioning_condition:
+            return
+
         if env.versioning_compare:
             # get old doctree
             try:
@@ -180,5 +185,7 @@ class UIDTransform(SphinxTransform):
 def prepare(document):
     # type: (nodes.Node) -> None
     """Simple wrapper for UIDTransform."""
+    warnings.warn('versioning.prepare() is deprecated. Use UIDTransform instead.',
+                  RemovedInSphinx30Warning)
     transform = UIDTransform(document)
     transform.apply()

sphinx/writers/latex.py

@@ -25,10 +25,9 @@ from sphinx import addnodes
 from sphinx import highlighting
 from sphinx.errors import SphinxError
 from sphinx.locale import admonitionlabels, _, __
-from sphinx.transforms import SphinxTransform
 from sphinx.util import split_into, logging
 from sphinx.util.i18n import format_date
-from sphinx.util.nodes import clean_astext, traverse_parent
+from sphinx.util.nodes import clean_astext
 from sphinx.util.template import LaTeXRenderer
 from sphinx.util.texescape import tex_escape_map, tex_replace_map
 
@@ -47,7 +46,6 @@ BEGIN_DOC = r'''
 '''
 
 
-URI_SCHEMES = ('mailto:', 'http:', 'https:', 'ftp:')
 LATEXSECTIONNAMES = ["part", "chapter", "section", "subsection",
                      "subsubsection", "paragraph", "subparagraph"]
 
@@ -190,10 +188,11 @@ class LaTeXWriter(writers.Writer):
 # Helper classes
 
 class ExtBabel(Babel):
-    def __init__(self, language_code):
+    def __init__(self, language_code, use_polyglossia=False):
-        # type: (unicode) -> None
+        # type: (unicode, bool) -> None
         super(ExtBabel, self).__init__(language_code or '')
         self.language_code = language_code
+        self.use_polyglossia = use_polyglossia
 
     def get_shorthandoff(self):
         # type: () -> unicode
@@ -221,116 +220,27 @@ class ExtBabel(Babel):
     def get_language(self):
         # type: () -> unicode
         language = super(ExtBabel, self).get_language()
-        if not language:
+        if language == 'ngerman' and self.use_polyglossia:
+            # polyglossia calls new orthography (Neue Rechtschreibung) as
+            # german (with new spelling option).
+            return 'german'
+        elif not language:
             return 'english'  # fallback to english
         else:
             return language
 
-
+    def get_mainlanguage_options(self):
-class ShowUrlsTransform(SphinxTransform, object):
+        # type: () -> unicode
-    def __init__(self, document, startnode=None):
+        """Return options for polyglossia's ``\setmainlanguage``."""
-        # type: (nodes.document, nodes.Node) -> None
+        language = super(ExtBabel, self).get_language()
-        super(ShowUrlsTransform, self).__init__(document, startnode)
+        if self.use_polyglossia is False:
-        self.expanded = False
+            return None
-
+        elif language == 'ngerman':
-    def apply(self):
+            return 'spelling=new'
-        # type: () -> None
+        elif language == 'german':
-        # replace id_prefix temporarily
+            return 'spelling=old'
-        id_prefix = self.document.settings.id_prefix
+        else:
-        self.document.settings.id_prefix = 'show_urls'
+            return None
-
-        self.expand_show_urls()
-        if self.expanded:
-            self.renumber_footnotes()
-
-        # restore id_prefix
-        self.document.settings.id_prefix = id_prefix
-
-    def expand_show_urls(self):
-        # type: () -> None
-        show_urls = self.document.settings.env.config.latex_show_urls
-        if show_urls is False or show_urls == 'no':
-            return
-
-        for node in self.document.traverse(nodes.reference):
-            uri = node.get('refuri', '')
-            if uri.startswith(URI_SCHEMES):
-                if uri.startswith('mailto:'):
-                    uri = uri[7:]
-                if node.astext() != uri:
-                    index = node.parent.index(node)
-                    if show_urls == 'footnote':
-                        footnote_nodes = self.create_footnote(uri)
-                        for i, fn in enumerate(footnote_nodes):
-                            node.parent.insert(index + i + 1, fn)
-
-                        self.expanded = True
-                    else:  # all other true values (b/w compat)
-                        textnode = nodes.Text(" (%s)" % uri)
-                        node.parent.insert(index + 1, textnode)
-
-    def create_footnote(self, uri):
-        # type: (unicode) -> List[Union[nodes.footnote, nodes.footnote_ref]]
-        label = nodes.label('', '#')
-        para = nodes.paragraph()
-        para.append(nodes.reference('', nodes.Text(uri), refuri=uri, nolinkurl=True))
-        footnote = nodes.footnote(uri, label, para, auto=1)
-        footnote['names'].append('#')
-        self.document.note_autofootnote(footnote)
-
-        label = nodes.Text('#')
-        footnote_ref = nodes.footnote_reference('[#]_', label, auto=1,
-                                                refid=footnote['ids'][0])
-        self.document.note_autofootnote_ref(footnote_ref)
-        footnote.add_backref(footnote_ref['ids'][0])
-
-        return [footnote, footnote_ref]
-
-    def renumber_footnotes(self):
-        # type: () -> None
-        def is_used_number(number):
-            # type: (unicode) -> bool
-            for node in self.document.traverse(nodes.footnote):
-                if not node.get('auto') and number in node['names']:
-                    return True
-
-            return False
-
-        def is_auto_footnote(node):
-            # type: (nodes.Node) -> bool
-            return isinstance(node, nodes.footnote) and node.get('auto')
-
-        def footnote_ref_by(node):
-            # type: (nodes.Node) -> Callable[[nodes.Node], bool]
-            ids = node['ids']
-            parent = list(traverse_parent(node, (nodes.document, addnodes.start_of_file)))[0]
-
-            def is_footnote_ref(node):
-                # type: (nodes.Node) -> bool
-                return (isinstance(node, nodes.footnote_reference) and
-                        ids[0] == node['refid'] and
-                        parent in list(traverse_parent(node)))
-
-            return is_footnote_ref
-
-        startnum = 1
-        for footnote in self.document.traverse(is_auto_footnote):
-            while True:
-                label = str(startnum)
-                startnum += 1
-                if not is_used_number(label):
-                    break
-
-            old_label = footnote[0].astext()
-            footnote.remove(footnote[0])
-            footnote.insert(0, nodes.label('', label))
-            if old_label in footnote['names']:
-                footnote['names'].remove(old_label)
-            footnote['names'].append(label)
-
-            for footnote_ref in self.document.traverse(footnote_ref_by(footnote)):
-                footnote_ref.remove(footnote_ref[0])
-                footnote_ref += nodes.Text(label)
 
 
 class Table(object):
@@ -618,9 +528,13 @@ class LaTeXTranslator(nodes.NodeVisitor):
         if builder.config.language \
            and 'fncychap' not in builder.config.latex_elements:
             # use Sonny style if any language specified
-            self.elements['fncychap'] = '\\usepackage[Sonny]{fncychap}'
+            self.elements['fncychap'] = ('\\usepackage[Sonny]{fncychap}\n'
+                                         '\\ChNameVar{\\Large\\normalfont'
+                                         '\\sffamily}\n\\ChTitleVar{\\Large'
+                                         '\\normalfont\\sffamily}')
 
-        self.babel = ExtBabel(builder.config.language)
+        self.babel = ExtBabel(builder.config.language,
+                              not self.elements['babel'])
         if builder.config.language and not self.babel.is_supported_language():
             # emit warning if specified language is invalid
             # (only emitting, nothing changed to processing)
@@ -654,8 +568,15 @@ class LaTeXTranslator(nodes.NodeVisitor):
                     # disable fncychap in Japanese documents
                     self.elements['fncychap'] = ''
         elif self.elements['polyglossia']:
-            self.elements['multilingual'] = '%s\n\\setmainlanguage{%s}' % \
+            options = self.babel.get_mainlanguage_options()
-                (self.elements['polyglossia'], self.babel.get_language())
+            if options:
+                mainlanguage = r'\setmainlanguage[%s]{%s}' % (options,
+                                                              self.babel.get_language())
+            else:
+                mainlanguage = r'\setmainlanguage{%s}' % self.babel.get_language()
+
+            self.elements['multilingual'] = '%s\n%s' % (self.elements['polyglossia'],
+                                                        mainlanguage)
 
         if getattr(builder, 'usepackages', None):
             def declare_package(packagename, options=None):
@@ -2502,20 +2423,15 @@ class LaTeXTranslator(nodes.NodeVisitor):
         # type: (nodes.Node) -> None
         self.body.append('}}$')
 
-    def visit_substitution_definition(self, node):
-        # type: (nodes.Node) -> None
-        raise nodes.SkipNode
-
-    def visit_substitution_reference(self, node):
-        # type: (nodes.Node) -> None
-        raise nodes.SkipNode
-
     def visit_inline(self, node):
         # type: (nodes.Node) -> None
         classes = node.get('classes', [])
-        if classes in [['menuselection'], ['guilabel']]:
+        if classes in [['menuselection']]:
             self.body.append(r'\sphinxmenuselection{')
             self.context.append('}')
+        elif classes in [['guilabel']]:
+            self.body.append(r'\sphinxguilabel{')
+            self.context.append('}')
         elif classes in [['accelerator']]:
             self.body.append(r'\sphinxaccelerator{')
             self.context.append('}')
@@ -2644,3 +2560,10 @@ class LaTeXTranslator(nodes.NodeVisitor):
     def unknown_visit(self, node):
         # type: (nodes.Node) -> None
         raise NotImplementedError('Unknown node: ' + node.__class__.__name__)
+
+
+# Import old modules here for compatibility
+# They should be imported after `LaTeXTranslator` to avoid recursive import.
+#
+# refs: https://github.com/sphinx-doc/sphinx/issues/4889
+from sphinx.builders.latex.transforms import URI_SCHEMES, ShowUrlsTransform  # NOQA

sphinx/writers/texinfo.py

@@ -1385,18 +1385,6 @@ class TexinfoTranslator(nodes.NodeVisitor):
         # type: (nodes.Node) -> None
         pass
 
-    def visit_substitution_reference(self, node):
-        # type: (nodes.Node) -> None
-        pass
-
-    def depart_substitution_reference(self, node):
-        # type: (nodes.Node) -> None
-        pass
-
-    def visit_substitution_definition(self, node):
-        # type: (nodes.Node) -> None
-        raise nodes.SkipNode
-
     def visit_system_message(self, node):
         # type: (nodes.Node) -> None
         self.body.append('\n@verbatim\n'

sphinx/writers/text.py

@@ -987,10 +987,6 @@ class TextTranslator(nodes.NodeVisitor):
         # type: (nodes.Node) -> None
         raise nodes.SkipNode
 
-    def visit_substitution_definition(self, node):
-        # type: (nodes.Node) -> None
-        raise nodes.SkipNode
-
     def visit_pending_xref(self, node):
         # type: (nodes.Node) -> None
         pass

tests/roots/test-config/conf.py

@@ -1,54 +1,3 @@
-from sphinx.config import string_classes, ENUM
+project = 'Sphinx <Tests>'
-
+release = '0.6alpha1'
-value1 = 123  # wrong type
+templates_path = ['_templates']
-value2 = 123  # lambda with wrong type
-value3 = []  # lambda with correct type
-value4 = True  # child type
-value5 = 3  # parent type
-value6 = ()  # other sequence type, also raises
-value7 = ['foo']  # explicitly permitted
-
-
-class A(object):
-    pass
-
-
-class B(A):
-    pass
-
-
-class C(A):
-    pass
-
-
-value8 = C()  # sibling type
-
-# both have no default or permissible types
-value9 = 'foo'
-value10 = 123
-value11 = u'bar'
-value12 = u'bar'
-value13 = 'bar'
-value14 = u'bar'
-value15 = 'bar'
-value16 = u'bar'
-
-
-def setup(app):
-    app.add_config_value('value1', 'string', False)
-    app.add_config_value('value2', lambda conf: [], False)
-    app.add_config_value('value3', [], False)
-    app.add_config_value('value4', 100, False)
-    app.add_config_value('value5', False, False)
-    app.add_config_value('value6', [], False)
-    app.add_config_value('value7', 'string', False, [list])
-    app.add_config_value('value8', B(), False)
-    app.add_config_value('value9', None, False)
-    app.add_config_value('value10', None, False)
-    app.add_config_value('value11', None, False, [str])
-    app.add_config_value('value12', 'string', False)
-    app.add_config_value('value13', None, False, string_classes)
-    app.add_config_value('value14', None, False, string_classes)
-    app.add_config_value('value15', u'unicode', False)
-    app.add_config_value('value16', u'unicode', False)
-    app.add_config_value('value17', 'default', False, ENUM('default', 'one', 'two'))

tests/roots/test-intl/refs.txt

@@ -4,7 +4,7 @@ References
 Translation Tips
 -----------------
 
-.. _download Sphinx: https://pypi.python.org/pypi/sphinx
+.. _download Sphinx: https://pypi.org/project/Sphinx/
 .. _Docutils site: http://docutils.sourceforge.net/
 .. _Sphinx site: http://sphinx-doc.org/
 

tests/roots/test-root/conf.py

@@ -12,7 +12,7 @@ from sphinx import addnodes
 sys.path.append(os.path.abspath('.'))
 
 extensions = ['sphinx.ext.autodoc', 'sphinx.ext.jsmath', 'sphinx.ext.todo',
-              'sphinx.ext.coverage', 'sphinx.ext.extlinks', 'ext']
+              'sphinx.ext.coverage', 'sphinx.ext.extlinks']
 
 jsmath_path = 'dummy.js'
 
@@ -65,8 +65,6 @@ man_pages = [
      'Georg Brandl and someone else', 1),
 ]
 
-value_from_conf_py = 84
-
 coverage_c_path = ['special/*.h']
 coverage_c_regexes = {'function': r'^PyAPI_FUNC\(.*\)\s+([^_][\w_]+)'}
 
@@ -104,7 +102,6 @@ class ClassDirective(Directive):
 def setup(app):
     import parsermod
 
-    app.add_config_value('value_from_conf_py', 42, False)
     app.add_directive('clsdir', ClassDirective)
     app.add_object_type('userdesc', 'userdescrole', '%s (userdesc)',
                         userdesc_parse, objname='user desc')

tests/roots/test-root/ext.py

@@ -1,5 +0,0 @@
-# Test extension module
-
-
-def setup(app):
-    app.add_config_value('value_from_ext', [], False)

tests/roots/test-toctree-glob/index.rst

@@ -15,7 +15,7 @@ normal order
    hyperref <https://sphinx-doc.org/?q=sphinx>
 
 reversed order
--------------
+--------------
 
 .. toctree::
    :glob:

tests/test_application.py

@@ -11,7 +11,7 @@
 import pytest
 from docutils import nodes
 
-from sphinx.application import ExtensionError
+from sphinx.errors import ExtensionError
 from sphinx.domains import Domain
 from sphinx.testing.util import strip_escseq
 from sphinx.util import logging

tests/test_build_latex.py

@@ -534,6 +534,50 @@ def test_babel_with_unknown_language(app, status, warning):
     assert "WARNING: no Babel option known for language 'unknown'" in warning.getvalue()
 
 
+@pytest.mark.sphinx(
+    'latex', testroot='latex-babel',
+    confoverrides={'language': 'de', 'latex_engine': 'lualatex'})
+def test_polyglossia_with_language_de(app, status, warning):
+    app.builder.build_all()
+    result = (app.outdir / 'Python.tex').text(encoding='utf8')
+    print(result)
+    print(status.getvalue())
+    print(warning.getvalue())
+    assert '\\documentclass[letterpaper,10pt,german]{sphinxmanual}' in result
+    assert '\\usepackage{polyglossia}' in result
+    assert '\\setmainlanguage[spelling=new]{german}' in result
+    assert '\\usepackage{times}' not in result
+    assert '\\usepackage[Sonny]{fncychap}' in result
+    assert ('\\addto\\captionsgerman{\\renewcommand{\\contentsname}{Table of content}}\n'
+            in result)
+    assert '\\addto\\captionsgerman{\\renewcommand{\\figurename}{Fig.}}\n' in result
+    assert '\\addto\\captionsgerman{\\renewcommand{\\tablename}{Table.}}\n' in result
+    assert '\\def\\pageautorefname{Seite}\n' in result
+    assert '\\shorthandoff' not in result
+
+
+@pytest.mark.sphinx(
+    'latex', testroot='latex-babel',
+    confoverrides={'language': 'de-1901', 'latex_engine': 'lualatex'})
+def test_polyglossia_with_language_de_1901(app, status, warning):
+    app.builder.build_all()
+    result = (app.outdir / 'Python.tex').text(encoding='utf8')
+    print(result)
+    print(status.getvalue())
+    print(warning.getvalue())
+    assert '\\documentclass[letterpaper,10pt,german]{sphinxmanual}' in result
+    assert '\\usepackage{polyglossia}' in result
+    assert '\\setmainlanguage[spelling=old]{german}' in result
+    assert '\\usepackage{times}' not in result
+    assert '\\usepackage[Sonny]{fncychap}' in result
+    assert ('\\addto\\captionsgerman{\\renewcommand{\\contentsname}{Table of content}}\n'
+            in result)
+    assert '\\addto\\captionsgerman{\\renewcommand{\\figurename}{Fig.}}\n' in result
+    assert '\\addto\\captionsgerman{\\renewcommand{\\tablename}{Table.}}\n' in result
+    assert '\\def\\pageautorefname{page}\n' in result
+    assert '\\shorthandoff' not in result
+
+
 @pytest.mark.sphinx('latex')
 def test_footnote(app, status, warning):
     app.builder.build_all()

tests/test_config.py

@@ -11,15 +11,15 @@
 """
 import mock
 import pytest
-from six import PY3, iteritems
+from six import PY3
 
 import sphinx
-from sphinx.config import Config
+from sphinx.config import Config, ENUM, string_classes, check_confval_types
 from sphinx.errors import ExtensionError, ConfigError, VersionRequirementError
 from sphinx.testing.path import path
 
 
-@pytest.mark.sphinx(confoverrides={
+@pytest.mark.sphinx(testroot='config', confoverrides={
     'master_doc': 'master',
     'nonexisting_value': 'True',
     'latex_elements.docclass': 'scrartcl',
@@ -74,36 +74,64 @@ def test_core_config(app, status, warning):
     assert cfg['project'] == cfg.project == 'Sphinx Tests'
 
 
-def test_extension_values(app, status, warning):
+def test_extension_values():
-    cfg = app.config
+    config = Config()
 
-    # default value
+    # check standard settings
-    assert cfg.value_from_ext == []
+    assert config.master_doc == 'contents'
-    # non-default value
-    assert cfg.value_from_conf_py == 84
 
-    # no duplicate values allowed
+    # can't override it by add_config_value()
     with pytest.raises(ExtensionError) as excinfo:
-        app.add_config_value('html_title', 'x', True)
+        config.add('master_doc', 'index', 'env', None)
     assert 'already present' in str(excinfo.value)
+
+    # add a new config value
+    config.add('value_from_ext', [], 'env', None)
+    assert config.value_from_ext == []
+
+    # can't override it by add_config_value()
     with pytest.raises(ExtensionError) as excinfo:
-        app.add_config_value('value_from_ext', 'x', True)
+        config.add('value_from_ext', [], 'env', None)
     assert 'already present' in str(excinfo.value)
 
 
+def test_overrides():
+    config = Config({'value1': '1', 'value2': 2, 'value6': {'default': 6}},
+                    {'value2': 999, 'value3': '999', 'value5.attr1': 999, 'value6.attr1': 999,
+                     'value7': 'abc,def,ghi', 'value8': 'abc,def,ghi'})
+    config.add('value1', None, 'env', ())
+    config.add('value2', None, 'env', ())
+    config.add('value3', 0, 'env', ())
+    config.add('value4', 0, 'env', ())
+    config.add('value5', {'default': 0}, 'env', ())
+    config.add('value6', {'default': 0}, 'env', ())
+    config.add('value7', None, 'env', ())
+    config.add('value8', [], 'env', ())
+    config.init_values()
+
+    assert config.value1 == '1'
+    assert config.value2 == 999
+    assert config.value3 == 999
+    assert config.value4 == 0
+    assert config.value5 == {'attr1': 999}
+    assert config.value6 == {'default': 6, 'attr1': 999}
+    assert config.value7 == 'abc,def,ghi'
+    assert config.value8 == ['abc', 'def', 'ghi']
+
+
 @mock.patch("sphinx.config.logger")
 def test_errors_warnings(logger, tempdir):
     # test the error for syntax errors in the config file
     (tempdir / 'conf.py').write_text(u'project = \n', encoding='ascii')
     with pytest.raises(ConfigError) as excinfo:
-        Config(tempdir, 'conf.py', {}, None)
+        Config.read(tempdir, {}, None)
     assert 'conf.py' in str(excinfo.value)
 
     # test the automatic conversion of 2.x only code in configs
     (tempdir / 'conf.py').write_text(
         u'# -*- coding: utf-8\n\nproject = u"Jägermeister"\n',
         encoding='utf-8')
-    cfg = Config(tempdir, 'conf.py', {}, None)
+    cfg = Config.read(tempdir, {}, None)
     cfg.init_values()
     assert cfg.project == u'Jägermeister'
     assert logger.called is False
@@ -115,7 +143,7 @@ def test_errors_warnings(logger, tempdir):
         return
     (tempdir / 'conf.py').write_text(
         u'# -*- coding: latin-1\nproject = "fooä"\n', encoding='latin-1')
-    cfg = Config(tempdir, 'conf.py', {}, None)
+    cfg = Config.read(tempdir, {}, None)
 
     assert logger.warning.called is False
     cfg.check_unicode()
@@ -174,7 +202,7 @@ def test_config_eol(logger, tempdir):
     configfile = tempdir / 'conf.py'
     for eol in (b'\n', b'\r\n'):
         configfile.write_bytes(b'project = "spam"' + eol)
-        cfg = Config(tempdir, 'conf.py', {}, None)
+        cfg = Config.read(tempdir, {}, None)
         cfg.init_values()
         assert cfg.project == u'spam'
         assert logger.called is False
@@ -195,60 +223,81 @@ def test_builtin_conf(app, status, warning):
         'warning')
 
 
-# See roots/test-config/conf.py.
+# example classes for type checking
-TYPECHECK_WARNINGS = {
+class A(object):
-    'value1': True,
+    pass
-    'value2': True,
-    'value3': False,
-    'value4': True,
-    'value5': False,
-    'value6': True,
-    'value7': False,
-    'value8': False,
-    'value9': False,
-    'value10': False,
-    'value11': False if PY3 else True,
-    'value12': False,
-    'value13': False,
-    'value14': False,
-    'value15': False,
-    'value16': False,
-}
 
 
-@pytest.mark.parametrize("key,should", iteritems(TYPECHECK_WARNINGS))
+class B(A):
-@pytest.mark.sphinx(testroot='config')
+    pass
-def test_check_types(warning, key, should):
-    warn = warning.getvalue()
-    if should:
-        assert key in warn, (
-            'override on "%s" should raise a type warning' % key
-        )
-    else:
-        assert key not in warn, (
-            'override on "%s" should NOT raise a type warning' % key
-        )
 
 
-@pytest.mark.sphinx(testroot='config')
+class C(A):
-def test_check_enum(app, status, warning):
+    pass
-    assert "The config value `value17` has to be a one of ('default', 'one', 'two'), " \
-           not in warning.getvalue()
 
 
-@pytest.mark.sphinx(testroot='config', confoverrides={'value17': 'invalid'})
+# name, default, annotation, actual, warned
-def test_check_enum_failed(app, status, warning):
+TYPECHECK_WARNINGS = [
-    assert "The config value `value17` has to be a one of ('default', 'one', 'two'), " \
+    ('value1', 'string', None, 123, True),                      # wrong type
-           "but `invalid` is given." in warning.getvalue()
+    ('value2', lambda _: [], None, 123, True),                  # lambda with wrong type
+    ('value3', lambda _: [], None, [], False),                  # lambda with correct type
+    ('value4', 100, None, True, True),                          # child type
+    ('value5', False, None, True, False),                       # parent type
+    ('value6', [], None, (), True),                             # other sequence type
+    ('value7', 'string', [list], ['foo'], False),               # explicit type annotation
+    ('value8', B(), None, C(), False),                          # sibling type
+    ('value9', None, None, 'foo', False),                       # no default or no annotations
+    ('value10', None, None, 123, False),                        # no default or no annotations
+    ('value11', None, [str], u'bar', False if PY3 else True),   # str vs unicode
+    ('value12', 'string', None, u'bar', False),                 # str vs unicode
+    ('value13', None, string_classes, 'bar', False),            # string_classes
+    ('value14', None, string_classes, u'bar', False),           # string_classes
+    ('value15', u'unicode', None, 'bar', False),                # str vs unicode
+    ('value16', u'unicode', None, u'bar', False),               # str vs unicode
+]
 
 
-@pytest.mark.sphinx(testroot='config', confoverrides={'value17': ['one', 'two']})
+@mock.patch("sphinx.config.logger")
-def test_check_enum_for_list(app, status, warning):
+@pytest.mark.parametrize("name,default,annotation,actual,warned", TYPECHECK_WARNINGS)
-    assert "The config value `value17` has to be a one of ('default', 'one', 'two'), " \
+def test_check_types(logger, name, default, annotation, actual, warned):
-           not in warning.getvalue()
+    config = Config({name: actual})
+    config.add(name, default, 'env', annotation or ())
+    config.init_values()
+    check_confval_types(None, config)
+    assert logger.warning.called == warned
 
 
-@pytest.mark.sphinx(testroot='config', confoverrides={'value17': ['one', 'two', 'invalid']})
+@mock.patch("sphinx.config.logger")
-def test_check_enum_for_list_failed(app, status, warning):
+def test_check_enum(logger):
-    assert "The config value `value17` has to be a one of ('default', 'one', 'two'), " \
+    config = Config()
-           "but `['one', 'two', 'invalid']` is given." in warning.getvalue()
+    config.add('value', 'default', False, ENUM('default', 'one', 'two'))
+    config.init_values()
+    check_confval_types(None, config)
+    logger.warning.assert_not_called()  # not warned
+
+
+@mock.patch("sphinx.config.logger")
+def test_check_enum_failed(logger):
+    config = Config({'value': 'invalid'})
+    config.add('value', 'default', False, ENUM('default', 'one', 'two'))
+    config.init_values()
+    check_confval_types(None, config)
+    logger.warning.assert_called()
+
+
+@mock.patch("sphinx.config.logger")
+def test_check_enum_for_list(logger):
+    config = Config({'value': ['one', 'two']})
+    config.add('value', 'default', False, ENUM('default', 'one', 'two'))
+    config.init_values()
+    check_confval_types(None, config)
+    logger.warning.assert_not_called()  # not warned
+
+
+@mock.patch("sphinx.config.logger")
+def test_check_enum_for_list_failed(logger):
+    config = Config({'value': ['one', 'two', 'invalid']})
+    config.add('value', 'default', False, ENUM('default', 'one', 'two'))
+    config.init_values()
+    check_confval_types(None, config)
+    logger.warning.assert_called()

tests/test_directive_code.py

@@ -17,7 +17,7 @@ from sphinx.config import Config
 from sphinx.directives.code import LiteralIncludeReader
 from sphinx.testing.util import etree_parse
 
-DUMMY_CONFIG = Config(None, None, {}, '')
+DUMMY_CONFIG = Config({}, {})
 
 
 @pytest.fixture(scope='module')

tests/test_directive_other.py

@@ -0,0 +1,153 @@
+# -*- coding: utf-8 -*-
+"""
+    test_directive_other
+    ~~~~~~~~~~~~~~~~~~~~
+
+    Test the other directives.
+
+    :copyright: Copyright 2007-2018 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+import pytest
+from docutils import nodes
+from docutils.core import publish_doctree
+
+from sphinx import addnodes
+from sphinx.io import SphinxStandaloneReader
+from sphinx.parsers import RSTParser
+from sphinx.testing.util import assert_node
+
+
+def parse(app, docname, text):
+    app.env.temp_data['docname'] = docname
+    return publish_doctree(text, app.srcdir / docname + '.rst',
+                           reader=SphinxStandaloneReader(app),
+                           parser=RSTParser(),
+                           settings_overrides={'env': app.env,
+                                               'gettext_compact': True})
+
+
+@pytest.mark.sphinx(testroot='toctree-glob')
+def test_toctree(app):
+    text = (".. toctree::\n"
+            "\n"
+            "   foo\n"
+            "   bar/index\n"
+            "   baz\n")
+
+    app.env.find_files(app.config, app.builder)
+    doctree = parse(app, 'index', text)
+    assert_node(doctree, [nodes.document, nodes.compound, addnodes.toctree])
+    assert_node(doctree[0][0],
+                entries=[(None, 'foo'), (None, 'bar/index'), (None, 'baz')],
+                includefiles=['foo', 'bar/index', 'baz'])
+
+
+@pytest.mark.sphinx(testroot='toctree-glob')
+def test_relative_toctree(app):
+    text = (".. toctree::\n"
+            "\n"
+            "   bar_1\n"
+            "   bar_2\n"
+            "   bar_3\n"
+            "   ../quux\n")
+
+    app.env.find_files(app.config, app.builder)
+    doctree = parse(app, 'bar/index', text)
+    assert_node(doctree, [nodes.document, nodes.compound, addnodes.toctree])
+    assert_node(doctree[0][0],
+                entries=[(None, 'bar/bar_1'), (None, 'bar/bar_2'), (None, 'bar/bar_3'),
+                         (None, 'quux')],
+                includefiles=['bar/bar_1', 'bar/bar_2', 'bar/bar_3', 'quux'])
+
+
+@pytest.mark.sphinx(testroot='toctree-glob')
+def test_toctree_urls_and_titles(app):
+    text = (".. toctree::\n"
+            "\n"
+            "   Sphinx <https://www.sphinx-doc.org/>\n"
+            "   https://readthedocs.org/\n"
+            "   The BAR <bar/index>\n")
+
+    app.env.find_files(app.config, app.builder)
+    doctree = parse(app, 'index', text)
+    assert_node(doctree, [nodes.document, nodes.compound, addnodes.toctree])
+    assert_node(doctree[0][0],
+                entries=[('Sphinx', 'https://www.sphinx-doc.org/'),
+                         (None, 'https://readthedocs.org/'),
+                         ('The BAR', 'bar/index')],
+                includefiles=['bar/index'])
+
+
+@pytest.mark.sphinx(testroot='toctree-glob')
+def test_toctree_glob(app):
+    text = (".. toctree::\n"
+            "   :glob:\n"
+            "\n"
+            "   *\n")
+
+    app.env.find_files(app.config, app.builder)
+    doctree = parse(app, 'index', text)
+    assert_node(doctree, [nodes.document, nodes.compound, addnodes.toctree])
+    assert_node(doctree[0][0],
+                entries=[(None, 'baz'), (None, 'foo'), (None, 'quux')],
+                includefiles=['baz', 'foo', 'quux'])
+
+    # give both docname and glob (case1)
+    text = (".. toctree::\n"
+            "   :glob:\n"
+            "\n"
+            "   foo\n"
+            "   *\n")
+
+    app.env.find_files(app.config, app.builder)
+    doctree = parse(app, 'index', text)
+    assert_node(doctree, [nodes.document, nodes.compound, addnodes.toctree])
+    assert_node(doctree[0][0],
+                entries=[(None, 'foo'), (None, 'baz'), (None, 'quux')],
+                includefiles=['foo', 'baz', 'quux'])
+
+    # give both docname and glob (case2)
+    text = (".. toctree::\n"
+            "   :glob:\n"
+            "\n"
+            "   *\n"
+            "   foo\n")
+
+    app.env.find_files(app.config, app.builder)
+    doctree = parse(app, 'index', text)
+    assert_node(doctree, [nodes.document, nodes.compound, addnodes.toctree])
+    assert_node(doctree[0][0],
+                entries=[(None, 'baz'), (None, 'foo'), (None, 'quux'), (None, 'foo')],
+                includefiles=['baz', 'foo', 'quux', 'foo'])
+
+
+@pytest.mark.sphinx(testroot='toctree-glob')
+def test_toctree_glob_and_url(app):
+    text = (".. toctree::\n"
+            "   :glob:\n"
+            "\n"
+            "   https://example.com/?q=sphinx\n")
+
+    app.env.find_files(app.config, app.builder)
+    doctree = parse(app, 'index', text)
+    assert_node(doctree, [nodes.document, nodes.compound, addnodes.toctree])
+    assert_node(doctree[0][0],
+                entries=[(None, 'https://example.com/?q=sphinx')],
+                includefiles=[])
+
+
+@pytest.mark.sphinx(testroot='toctree-glob')
+def test_toctree_twice(app):
+    text = (".. toctree::\n"
+            "\n"
+            "   foo\n"
+            "   foo\n")
+
+    app.env.find_files(app.config, app.builder)
+    doctree = parse(app, 'index', text)
+    assert_node(doctree, [nodes.document, nodes.compound, addnodes.toctree])
+    assert_node(doctree[0][0],
+                entries=[(None, 'foo'), (None, 'foo')],
+                includefiles=['foo', 'foo'])

tests/test_ext_math.py

@@ -35,19 +35,19 @@ def test_jsmath(app, status, warning):
     app.builder.build_all()
     content = (app.outdir / 'math.html').text()
 
-    assert '<div class="math notranslate">\na^2 + b^2 = c^2</div>' in content
+    assert '<div class="math notranslate nohighlight">\na^2 + b^2 = c^2</div>' in content
-    assert ('<div class="math notranslate">\n\\begin{split}a + 1 &lt; b\\end{split}</div>'
+    assert ('<div class="math notranslate nohighlight">\n\\begin{split}a + 1 &lt; '
-            in content)
+            'b\\end{split}</div>' in content)
     assert (u'<span class="eqno">(1)<a class="headerlink" href="#equation-foo" '
             u'title="Permalink to this equation">\xb6</a></span>'
-            u'<div class="math notranslate" id="equation-foo">\ne^{i\\pi} = 1</div>'
+            u'<div class="math notranslate nohighlight" id="equation-foo">'
-            in content)
+            '\ne^{i\\pi} = 1</div>' in content)
     assert (u'<span class="eqno">(2)<a class="headerlink" href="#equation-math-0" '
             u'title="Permalink to this equation">\xb6</a></span>'
-            u'<div class="math notranslate" id="equation-math-0">\n'
+            u'<div class="math notranslate nohighlight" id="equation-math-0">\n'
             u'e^{ix} = \\cos x + i\\sin x</div>' in content)
-    assert '<div class="math notranslate">\nn \\in \\mathbb N</div>' in content
+    assert '<div class="math notranslate nohighlight">\nn \\in \\mathbb N</div>' in content
-    assert '<div class="math notranslate">\na + 1 &lt; b</div>' in content
+    assert '<div class="math notranslate nohighlight">\na + 1 &lt; b</div>' in content
 
 
 @pytest.mark.skipif(not has_binary('dvipng'),
@@ -91,7 +91,7 @@ def test_mathjax_align(app, status, warning):
     app.builder.build_all()
 
     content = (app.outdir / 'index.html').text()
-    html = (r'<div class="math notranslate">\s*'
+    html = (r'<div class="math notranslate nohighlight">\s*'
             r'\\\[ \\begin\{align\}\\begin\{aligned\}S \&amp;= \\pi r\^2\\\\'
             r'V \&amp;= \\frac\{4\}\{3\} \\pi r\^3\\end\{aligned\}\\end\{align\} \\\]</div>')
     assert re.search(html, content, re.S)
@@ -104,7 +104,7 @@ def test_math_number_all_mathjax(app, status, warning):
     app.builder.build_all()
 
     content = (app.outdir / 'index.html').text()
-    html = (r'<div class="math notranslate" id="equation-index-0">\s*'
+    html = (r'<div class="math notranslate nohighlight" id="equation-index-0">\s*'
             r'<span class="eqno">\(1\)<a .*>\xb6</a></span>\\\[a\^2\+b\^2=c\^2\\\]</div>')
     assert re.search(html, content, re.S)
 
@@ -169,7 +169,7 @@ def test_mathjax_numfig_html(app, status, warning):
     app.builder.build_all()
 
     content = (app.outdir / 'math.html').text()
-    html = ('<div class="math notranslate" id="equation-math-0">\n'
+    html = ('<div class="math notranslate nohighlight" id="equation-math-0">\n'
             '<span class="eqno">(1.2)')
     assert html in content
     html = ('<p>Referencing equation <a class="reference internal" '

tests/test_markup.py

@@ -162,12 +162,20 @@ def get_verifier(verify, verify_re):
         '\\sphinxmenuselection{a \\(\\rightarrow\\) b}',
     ),
     (
-        # interpolation of ampersands in guilabel/menuselection
+        # interpolation of ampersands in menuselection
+        'verify',
+        ':menuselection:`&Foo -&&- &Bar`',
+        (u'<p><span class="menuselection"><span class="accelerator">F</span>oo '
+         '-&amp;- <span class="accelerator">B</span>ar</span></p>'),
+        r'\sphinxmenuselection{\sphinxaccelerator{F}oo -\&- \sphinxaccelerator{B}ar}',
+    ),
+    (
+        # interpolation of ampersands in guilabel
         'verify',
         ':guilabel:`&Foo -&&- &Bar`',
         (u'<p><span class="guilabel"><span class="accelerator">F</span>oo '
          '-&amp;- <span class="accelerator">B</span>ar</span></p>'),
-        r'\sphinxmenuselection{\sphinxaccelerator{F}oo -\&- \sphinxaccelerator{B}ar}',
+        r'\sphinxguilabel{\sphinxaccelerator{F}oo -\&- \sphinxaccelerator{B}ar}',
     ),
     (
         # non-interpolation of dashes in option role

utils/release-checklist

@@ -12,7 +12,7 @@ for stable releases
 * ``git commit -am 'Bump to X.Y.Z final'``
 * ``make clean``
 * ``python setup.py release bdist_wheel sdist upload --identity=[your key]``
-* open https://pypi.python.org/pypi/Sphinx and check there are no obvious errors
+* open https://pypi.org/project/Sphinx/ and check there are no obvious errors
 * ``git tag vX.Y.Z``
 * ``python utils/bump_version.py --in-develop X.Y.Zb0`` (ex. 1.5.3b0)
 * Check diff by ``git diff``
@@ -38,7 +38,7 @@ for first beta releases
 * ``git commit -am 'Bump to X.Y.0 beta1'``
 * ``make clean``
 * ``python setup.py release bdist_wheel sdist upload --identity=[your key]``
-* open https://pypi.python.org/pypi/Sphinx and check there are no obvious errors
+* open https://pypi.org/project/Sphinx/ and check there are no obvious errors
 * ``git tag vX.Y.0b1``
 * ``python utils/bump_version.py --in-develop X.Y.0b2`` (ex. 1.6.0b2)
 * Check diff by ``git diff``
@@ -67,7 +67,7 @@ for other beta releases
 * ``git commit -am 'Bump to X.Y.0 betaN'``
 * ``make clean``
 * ``python setup.py release bdist_wheel sdist upload --identity=[your key]``
-* open https://pypi.python.org/pypi/Sphinx and check there are no obvious errors
+* open https://pypi.org/project/Sphinx/ and check there are no obvious errors
 * ``git tag vX.Y.0bN``
 * ``python utils/bump_version.py --in-develop X.Y.0bM`` (ex. 1.6.0b3)
 * Check diff by `git diff``
@@ -95,7 +95,7 @@ for major releases
 * ``git commit -am 'Bump to X.Y.0 final'``
 * ``make clean``
 * ``python setup.py release bdist_wheel sdist upload --identity=[your key]``
-* open https://pypi.python.org/pypi/Sphinx and check there are no obvious errors
+* open https://pypi.org/project/Sphinx/ and check there are no obvious errors
 * ``git tag vX.Y.0``
 * ``python utils/bump_version.py --in-develop X.Y.1b0`` (ex. 1.6.1b0)
 * Check diff by ``git diff``