Merge branch 'master' into 1431_latex_enumlist

CHANGES

@@ -34,25 +34,28 @@ Incompatible changes
 * #1857: latex: :confval:`latex_show_pagerefs` does not add pagerefs for
   citations
 * #4648: latex: Now "rubric" elements are rendered as unnumbered section title
+* #4983: html: The anchor for productionlist tokens has been changed
+* Modifying a template variable ``script_files`` in templates is allowed now.
+  Please use ``app.add_js_file()`` instead.
+* #5072: Save environment object also with only new documents
+* #5035: qthelp builder allows dashes in :confval:`qthelp_namespace`
 
 Deprecated
 ----------
 
 * :confval:`source_parsers` is deprecated
-* ``Application.import_object()`` is deprecated
 * Drop function based directive support.  For now, Sphinx only supports class
   based directives.
-* ``Sphinx.add_source_parser()`` has changed;  the *suffix* argument has
-  been deprecated
 * ``sphinx.util.docutils.directive_helper()`` is deprecated
 * ``sphinx.cmdline`` is deprecated
-* All ``env.update()``, ``env._read_serial()`` and ``env._read_parallel()`` are
-  deprecated
 * ``sphinx.locale.l_()`` is deprecated
 * #2157: helper function ``warn()`` for HTML themes is deprecated
-* ``env._nitpick_ignore`` is deprecated
 * ``app.override_domain()`` is deprecated
 * ``app.add_stylesheet()`` is deprecated
+* ``app.add_javascript()`` is deprecated
+* ``app.import_object()`` is deprecated
+* ``app.add_source_parser()`` has changed;  the *suffix* argument has been
+  deprecated
 * ``sphinx.versioning.prepare()`` is deprecated
 * ``Config.__init__()`` has changed;  the *dirname*, *filename* and *tags*
   argument has been deprecated
@@ -60,19 +63,30 @@ Deprecated
 * ``Config.check_unicode()`` is deprecated
 * ``sphinx.application.CONFIG_FILENAME`` is deprecated
 * ``highlightlang`` directive is deprecated
-* ``env.read_doc()`` is deprecated
-* ``sphinx.writers.latex.Table.caption_footnotetexts`` is deprecated
-* ``sphinx.writers.latex.Table.header_footnotetexts`` is deprecated
-* ``sphinx.writers.latex.LaTeXWriter.footnotestack`` is deprecated
-* ``sphinx.writers.latex.LaTeXWriter.restrict_footnote()`` is deprecated
-* ``sphinx.writers.latex.LaTeXWriter.unrestrict_footnote()`` is deprecated
-* ``LaTeXWriter.bibitems`` is deprecated
 * ``BuildEnvironment.load()`` is deprecated
 * ``BuildEnvironment.loads()`` is deprecated
 * ``BuildEnvironment.frompickle()`` is deprecated
-* ``BuildEnvironment.dump()`` is deprecated
+* ``env.read_doc()`` is deprecated
-* ``BuildEnvironment.dumps()`` is deprecated
+* ``env.update()`` is deprecated
-* ``BuildEnvironment.topickle()`` is deprecated
+* ``env._read_serial()`` is deprecated
+* ``env._read_parallel()`` is deprecated
+* ``env.write_doctree()`` is deprecated
+* ``env._nitpick_ignore`` is deprecated
+* ``env.dump()`` is deprecated
+* ``env.dumps()`` is deprecated
+* ``env.topickle()`` is deprecated
+* ``sphinx.writers.latex.Table.caption_footnotetexts`` is deprecated
+* ``sphinx.writers.latex.Table.header_footnotetexts`` is deprecated
+* ``sphinx.writers.latex.LaTeXTranslator.footnotestack`` is deprecated
+* ``sphinx.writers.latex.LaTeXTranslator.in_container_literal_block`` is deprecated
+* ``sphinx.writers.latex.LaTeXTranslator.next_section_ids`` is deprecated
+* ``sphinx.writers.latex.LaTeXTranslator.next_hyperlink_ids`` is deprecated
+* ``sphinx.writers.latex.LaTeXTranslator.restrict_footnote()`` is deprecated
+* ``sphinx.writers.latex.LaTeXTranslator.unrestrict_footnote()`` is deprecated
+* ``sphinx.writers.latex.LaTeXTranslator.push_hyperlink_ids()`` is deprecated
+* ``sphinx.writers.latex.LaTeXTranslator.pop_hyperlink_ids()`` is deprecated
+* ``sphinx.writers.latex.LaTeXTranslator.bibitems`` is deprecated
+* ``sphinx.writers.latex.ExtBabel.get_shorthandoff()`` is deprecated
 * ``sphinx.ext.mathbase.math`` node is deprecated
 * ``sphinx.ext.mathbase.displaymath`` node is deprecated
 * ``sphinx.ext.mathbase.eqref`` node is deprecated
@@ -103,6 +117,7 @@ Features added
   fontsize in code-blocks (refs: #4793)
 * Add :confval:`html_css_files` and :confval:`epub_css_files` for adding CSS
   files from configuration
+* Add :confval:`html_js_files` for adding JS files from configuration
 * #4834: Ensure set object descriptions are reproducible.
 * #4828: Allow to override :confval:`numfig_format` partially.  Full definition
   is not needed.
@@ -116,6 +131,17 @@ Features added
 * #4785: napoleon: Add strings to translation file for localisation
 * #4927: Display a warning when invalid values are passed to linenothreshold
   option of highlight directive
+* C++, add a ``cpp:texpr`` role as a sibling to ``cpp:expr``.
+* C++, add support for unions.
+* C++, add support for anonymous entities using names staring with ``@``.
+  Fixes #3593 and #2683.
+* #3606: MathJax should be loaded with async attribute
+* html: Output ``canonical_url`` metadata if :confval:`html_baseurl` set (refs:
+  #4193)
+* #5029: autosummary: expose ``inherited_members`` to template
+* #3784: mathjax: Add :confval:`mathjax_options` to give options to script tag
+  for mathjax
+* #4362: latex: Don't overwrite .tex file if document not changed
 * #1431: latex: Add alphanumeric enumerated list support
 
 Bugs fixed
@@ -125,6 +151,7 @@ Bugs fixed
 * #4850: latex: footnote inside footnote was not rendered
 * #4945: i18n: fix lang_COUNTRY not fallback correctly for IndexBuilder. Thanks
   to Shengjing Zhu.
+* #4983: productionlist directive generates invalid IDs for the tokens
 
 Testing
 --------
@@ -134,7 +161,12 @@ Features removed
 
 * ``sphinx.ext.pngmath`` extension
 
-Release 1.7.5 (in development)
+Documentation
+-------------
+
+* #5083: Fix wrong make.bat option for internationalization.
+
+Release 1.7.6 (in development)
 ==============================
 
 Dependencies
@@ -152,6 +184,33 @@ Features added
 Bugs fixed
 ----------
 
+* #5037: LaTeX ``\sphinxupquote{}`` breaks in Russian
+* sphinx.testing uses deprecated pytest API; ``Node.get_marker(name)``
+* #5016: crashed when recommonmark.AutoStrictify is enabled
+* #5022: latex: crashed with docutils package provided by Debian/Ubuntu
+* #5009: latex: a label for table is vanished if table does not have a caption
+* #5048: crashed with numbered toctree
+* #2410: C, render empty argument lists for macros.
+* C++, fix lookup of full template specializations with no template arguments.
+* #4667: C++, fix assertion on missing references in global scope when using
+  intersphinx. Thanks to Alan M. Carroll.
+* #5019: autodoc: crashed by Form Feed Character
+* #5032: autodoc: loses the first staticmethod parameter for old styled classes
+* #5036: quickstart: Typing Ctrl-U clears the whole of line
+* #5066: html: "relations" sidebar is not shown by default
+* #5091: latex: curly braces in index entries are not handled correctly
+* #5070: epub: Wrong internal href fragment links
+* #5104: apidoc: Interface of ``sphinx.apidoc:main()`` has changed
+
+Testing
+--------
+
+Release 1.7.5 (released May 29, 2018)
+=====================================
+
+Bugs fixed
+----------
+
 * #4924: html search: Upper characters problem in any other languages
 * #4932: apidoc: some subpackage is ignored if sibling subpackage contains a
   module starting with underscore
@@ -162,6 +221,7 @@ Bugs fixed
 * #4825: C++, properly parse expr roles and give better error messages when
   (escaped) line breaks are present.
 * C++, properly use ``desc_addname`` nodes for prefixes of names.
+* C++, parse pack expansions in function calls.
 * #4915, #4916: links on search page are broken when using dirhtml builder
 * #4969: autodoc: constructor method should not have return annotation
 * latex: deeply nested enumerated list which is beginning with non-1 causes
@@ -174,9 +234,12 @@ Bugs fixed
   mocked module
 * #4973: latex: glossary directive adds whitespace to each item
 * #4980: latex: Explicit labels on code blocks are duplicated
-
+* #4919: node.asdom() crashes if toctree has :numbered: option
-Testing
+* #4914: autodoc: Parsing error when using dataclasses without default values
---------
+* #4931: autodoc: crashed when handler for autodoc-skip-member raises an error
+* #4931: autodoc: crashed when subclass of mocked class are processed by
+  napoleon module
+* #5007: sphinx-build crashes when error log contains a "%" character
 
 Release 1.7.4 (released Apr 25, 2018)
 =====================================

EXAMPLES

@@ -195,6 +195,7 @@ Documentation using sphinx_rtd_theme
 * Julia: https://julia.readthedocs.io/
 * Jupyter Notebook: https://jupyter-notebook.readthedocs.io/
 * Lasagne: https://lasagne.readthedocs.io/
+* latexindent.pl: https://latexindentpl.readthedocs.io/
 * Linguistica: https://linguistica-uchicago.github.io/lxa5/
 * Linux kernel: https://www.kernel.org/doc/html/latest/index.html
 * MathJax: https://docs.mathjax.org/
@@ -218,6 +219,7 @@ Documentation using sphinx_rtd_theme
 * peewee: http://docs.peewee-orm.com/
 * Phinx: http://docs.phinx.org/
 * phpMyAdmin: https://docs.phpmyadmin.net/
+* PROS: https://pros.cs.purdue.edu/v5/ (customized)
 * Pweave: http://mpastell.com/pweave/
 * PyPy: http://doc.pypy.org/
 * python-sqlparse: https://sqlparse.readthedocs.io/

doc/Makefile

@@ -1,9 +1,10 @@
 # Makefile for Sphinx documentation
 #
+PYTHON ?= python
 
 # You can set these variables from the command line.
 SPHINXOPTS   =
-SPHINXBUILD  = python ../sphinx/cmd/build.py
+SPHINXBUILD  = $(PYTHON) ../sphinx/cmd/build.py
 SPHINXPROJ   = sphinx
 SOURCEDIR    = .
 BUILDDIR     = _build

doc/_templates/index.html

@@ -73,11 +73,8 @@
   </table>
 
   <p>{%trans%}
-    You can also download PDF/EPUB versions of the Sphinx documentation:
+    You can also download PDF/EPUB versions of the Sphinx documentation
-    a <a href="https://media.readthedocs.org/pdf/sphinx/stable/sphinx.pdf">PDF version</a> generated from
+    from pop up menu on lower right corner.{%endtrans%}
-    the LaTeX Sphinx producer, and
-    an <a href="https://media.readthedocs.org/epub/sphinx/stable/sphinx.epub">EPUB version</a>.
-   {%endtrans%}
   </p>
 
   <h2>{%trans%}Examples{%endtrans%}</h2>

doc/_templates/indexsidebar.html

@@ -3,19 +3,9 @@
   {%trans%}project{%endtrans%}</p>
 
 <h3>Download</h3>
-{% if version.endswith('+') %}
+<p class="download">{%trans%}Current version: <a href="https://pypi.org/project/Sphinx/" alt="PyPI"><img src="https://img.shields.io/pypi/v/sphinx.svg"></a>{%endtrans%}</p>
-<p>{%trans%}This documentation is for version <b><a href="changes.html">{{ version }}</a></b>, which is
+<p>{%trans%}Install Sphinx with:{%endtrans%}</p>
-  not released yet.{%endtrans%}</p>
-<p>{%trans%}You can use it from the
-  <a href="https://github.com/sphinx-doc/sphinx/">Git repo</a> or look for
-  released versions in the <a href="https://pypi.org/project/Sphinx/">Python
-    Package Index</a>.{%endtrans%}</p>
-{% else %}
-<p>{%trans%}Current version: <b><a href="changes.html">{{ version }}</a></b>{%endtrans%}</p>
-<p>{%trans%}Get Sphinx from the <a href="https://pypi.org/project/Sphinx/">Python Package
-Index</a>, or install it with:{%endtrans%}</p>
 <pre>pip install -U Sphinx</pre>
-{% endif %}
 
 <h3>{%trans%}Questions? Suggestions?{%endtrans%}</h3>
 

doc/_themes/sphinx13/static/sphinx13.css

@@ -140,6 +140,10 @@ div.sphinxsidebar .logo img {
     vertical-align: middle;
 }
 
+div.sphinxsidebar .download a img {
+    vertical-align: middle;
+}
+
 div.subscribeformwrapper {
     display: block;
     overflow: auto;

doc/conf.py

@@ -103,7 +103,7 @@ texinfo_documents = [
 
 # We're not using intersphinx right now, but if we did, this would be part of
 # the mapping:
-intersphinx_mapping = {'python': ('https://docs.python.org/2/', None)}
+intersphinx_mapping = {'python': ('https://docs.python.org/3/', None)}
 
 # Sphinx document translation with sphinx gettext feature uses these settings:
 locale_dirs = ['locale/']

doc/ext/autodoc.rst

@@ -41,7 +41,7 @@ you can also enable the :mod:`napoleon <sphinx.ext.napoleon>` extension.
 docstrings to correct reStructuredText before :mod:`autodoc` processes them.
 
 .. _Google:
-   https://google.github.io/styleguide/pyguide.html#Comments
+   https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings
 .. _NumPy:
    https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt
 
@@ -374,7 +374,7 @@ There are also new config values that you can set:
    This value contains a list of modules to be mocked up. This is useful when
    some external dependencies are not met at build time and break the building
    process. You may only specify the root package of the dependencies
-   themselves and ommit the sub-modules:
+   themselves and omit the sub-modules:
 
    .. code-block:: python
 

doc/ext/autosummary.rst

@@ -205,6 +205,11 @@ The following variables available in the templates:
    List containing names of all members of the module or class.  Only available
    for modules and classes.
 
+.. data:: inherited_members
+
+   List containing names of all inherited members of class.  Only available for
+   classes.
+
 .. data:: functions
 
    List containing names of "public" functions in the module.  Here, "public"

doc/ext/math.rst

@@ -246,7 +246,7 @@ Sphinx.
    page`__ for details. If you want MathJax to be available offline, you have
    to download it and set this value to a different path.
 
-   __ https://cdjns.com
+   __ https://cdnjs.com
 
    __ https://docs.mathjax.org/en/latest/start.html
 
@@ -260,6 +260,16 @@ Sphinx.
 
    You can also give a full ``https://`` URL different from the CDN URL.
 
+.. confval:: mathjax_options
+
+   The options to script tag for mathjax.  For example, you can set integrity
+   option with following setting::
+
+       mathjax_options = {
+           'integrity': 'sha384-......',
+       }
+
+   The default is empty (``{}``).
 
 :mod:`sphinx.ext.jsmath` -- Render math via JavaScript
 ------------------------------------------------------

doc/extdev/appapi.rst

@@ -71,9 +71,9 @@ package.
 
 .. automethod:: Sphinx.add_post_transform(transform)
 
-.. automethod:: Sphinx.add_javascript(filename)
+.. automethod:: Sphinx.add_js_file(filename, **kwargs)
 
-.. automethod:: Sphinx.add_stylesheet(filename, alternate=None, title=None)
+.. automethod:: Sphinx.add_css_file(filename, **kwargs)
 
 .. automethod:: Sphinx.add_latex_package(packagename, options=None)
 
@@ -144,19 +144,19 @@ Sphinx runtime information
 
 The application object also provides runtime information as attributes.
 
-.. attribute:: srcdir
+.. attribute:: Sphinx.srcdir
 
    Source directory.
 
-.. attribute:: confdir
+.. attribute:: Sphinx.confdir
 
    Directory containing ``conf.py``.
 
-.. attribute:: doctreedir
+.. attribute:: Sphinx.doctreedir
 
    Directory for storing pickled doctrees.
 
-.. attribute:: outdir
+.. attribute:: Sphinx.outdir
 
    Directory for storing built document.
 
@@ -269,7 +269,7 @@ handlers to the events.  Example:
    Here is the place to replace custom nodes that don't have visitor methods in
    the writers, so that they don't cause errors when the writers encounter them.
 
-.. event:: env-merge-info (env, docnames, other)
+.. event:: env-merge-info (app, env, docnames, other)
 
    This event is only emitted when parallel reading of documents is enabled.  It
    is emitted once for every subprocess that has read some documents.
@@ -303,7 +303,7 @@ handlers to the events.  Example:
    .. versionchanged:: 1.3
       The handlers' return value is now used.
 
-.. event:: env-check-consistency (env)
+.. event:: env-check-consistency (app, env)
 
    Emitted when Consistency checks phase.  You can check consistency of
    metadata for whole of documents.

doc/extdev/index.rst

@@ -101,7 +101,7 @@ Deprecated APIs
 
 On developing Sphinx, we are always careful to the compatibility of our APIs.
 But, sometimes, the change of interface are needed for some reasons.  In such
-cases, we've marked thme as deprecated. And they are kept during the two
+cases, we've marked them as deprecated. And they are kept during the two
 major versions (for more details, please see :ref:`deprecation-policy`).
 
 The following is a list of deprecated interface.
@@ -126,6 +126,11 @@ The following is a list of deprecated interface.
      - 4.0
      - :meth:`~sphinx.application.Sphinx.add_css_file()`
 
+   * - :meth:`~sphinx.application.Sphinx.add_javascript()`
+     - 1.8
+     - 4.0
+     - :meth:`~sphinx.application.Sphinx.add_js_file()`
+
    * - ``sphinx.ext.mathbase.MathDomain``
      - 1.8
      - 3.0
@@ -166,22 +171,52 @@ The following is a list of deprecated interface.
      - 3.0
      - N/A
 
-   * - ``sphinx.writers.latex.LaTeXWriter.footnotestack``
+   * - ``sphinx.writers.latex.LaTeXTranslator.footnotestack``
      - 1.8
      - 3.0
      - N/A
 
-   * - ``sphinx.writers.latex.LaTeXWriter.restrict_footnote()``
+   * - ``sphinx.writers.latex.LaTeXTranslator.in_container_literal_block``
      - 1.8
      - 3.0
      - N/A
 
-   * - ``sphinx.writers.latex.LaTeXWriter.unrestrict_footnote()``
+   * - ``sphinx.writers.latex.LaTeXTranslator.next_section_ids``
      - 1.8
      - 3.0
      - N/A
 
-   * - ``sphinx.writers.latex.LaTeXWriter.bibitems``
+   * - ``sphinx.writers.latex.LaTeXTranslator.next_hyperlink_ids``
+     - 1.8
+     - 3.0
+     - N/A
+
+   * - ``sphinx.writers.latex.LaTeXTranslator.restrict_footnote()``
+     - 1.8
+     - 3.0
+     - N/A
+
+   * - ``sphinx.writers.latex.LaTeXTranslator.unrestrict_footnote()``
+     - 1.8
+     - 3.0
+     - N/A
+
+   * - ``sphinx.writers.latex.LaTeXTranslator.push_hyperlink_ids()``
+     - 1.8
+     - 3.0
+     - N/A
+
+   * - ``sphinx.writers.latex.LaTeXTranslator.pop_hyperlink_ids()``
+     - 1.8
+     - 3.0
+     - N/A
+
+   * - ``sphinx.writers.latex.LaTeXTranslator.bibitems``
+     - 1.8
+     - 3.0
+     - N/A
+
+   * - ``sphinx.writers.latex.ExtBabel.get_shorthandoff()``
      - 1.8
      - 3.0
      - N/A
@@ -217,11 +252,23 @@ The following is a list of deprecated interface.
      - 3.0
      - ``sphinx.versioning.UIDTransform``
 
-   * - ``sphinx.application.Sphinx.override_domain()``
+   * - ``Sphinx.override_domain()``
      - 1.8
      - 3.0
      - :meth:`~sphinx.application.Sphinx.add_domain()`
 
+   * - ``Sphinx.import_object()``
+     - 1.8
+     - 3.0
+     - ``sphinx.util.import_object()``
+
+   * - ``suffix`` argument of
+       :meth:`~sphinx.application.Sphinx.add_source_parser()`
+     - 1.8
+     - 3.0
+     - :meth:`~sphinx.application.Sphinx.add_source_suffix()`
+
+
    * - ``BuildEnvironment.load()``
      - 1.8
      - 3.0
@@ -257,37 +304,6 @@ The following is a list of deprecated interface.
      - 3.0
      - :confval:`nitpick_ignore`
 
-   * - ``warn()`` (template helper function)
-     - 1.8
-     - 3.0
-     - ``warning()``
-
-   * - :confval:`source_parsers`
-     - 1.8
-     - 3.0
-     - :meth:`~sphinx.application.Sphinx.add_source_parser()`
-
-   * - ``Sphinx.import_object()``
-     - 1.8
-     - 3.0
-     - ``sphinx.util.import_object()``
-
-   * - ``suffix`` argument of
-       :meth:`~sphinx.application.Sphinx.add_source_parser()`
-     - 1.8
-     - 3.0
-     - :meth:`~sphinx.application.Sphinx.add_source_suffix()`
-
-   * - ``sphinx.util.docutils.directive_helper()``
-     - 1.8
-     - 3.0
-     - ``Directive`` class of docutils
-
-   * - ``sphinx.cmdline``
-     - 1.8
-     - 3.0
-     - ``sphinx.cmd.build``
-
    * - ``BuildEnvironment.update()``
      - 1.8
      - 3.0
@@ -313,6 +329,26 @@ The following is a list of deprecated interface.
      - 3.0
      - ``Builder.write_doctree()``
 
+   * - ``warn()`` (template helper function)
+     - 1.8
+     - 3.0
+     - ``warning()``
+
+   * - :confval:`source_parsers`
+     - 1.8
+     - 3.0
+     - :meth:`~sphinx.application.Sphinx.add_source_parser()`
+
+   * - ``sphinx.util.docutils.directive_helper()``
+     - 1.8
+     - 3.0
+     - ``Directive`` class of docutils
+
+   * - ``sphinx.cmdline``
+     - 1.8
+     - 3.0
+     - ``sphinx.cmd.build``
+
    * - ``sphinx.locale.l_()``
      - 1.8
      - 3.0

doc/intl.rst

@@ -123,14 +123,14 @@ This section describe an easy way to translate with sphinx-intl.
 
    .. code-block:: console
 
-      > set SPHINXOPTS=-D language='de'
+      > set SPHINXOPTS=-D language=de
       > .\make.bat html
 
    command line (for PowerShell):
 
    .. code-block:: console
 
-      > Set-Item env:SPHINXOPTS "-D language='de'"
+      > Set-Item env:SPHINXOPTS "-D language=de"
       > .\make.bat html
 
 

doc/make.bat

@@ -3,7 +3,7 @@
 REM Command file for Sphinx documentation
 
 if "%SPHINXBUILD%" == "" (
-	set SPHINXBUILD=python ../sphinx-build.py
+	set SPHINXBUILD=python ../sphinx/cmd/build.py
 )
 set SOURCEDIR=.
 set BUILDDIR=_build

doc/usage/configuration.rst

@@ -827,6 +827,11 @@ that use Sphinx's HTMLWriter class.
 
    .. versionadded:: 0.4
 
+.. confval:: html_baseurl
+
+   The URL which points to the root of the HTML documentation.  It is used to
+   indicate the location of document like ``canonical_url``.
+
 .. confval:: html_context
 
    A dictionary of values to pass into the template engine's context for all
@@ -874,6 +879,22 @@ that use Sphinx's HTMLWriter class.
 
    .. versionadded:: 1.8
 
+.. confval:: html_js_files
+
+   A list of JavaScript *filename*.  The entry must be a *filename* string or a
+   tuple containing the *filename* string and the *attributes* dictionary.  The
+   *filename* must be relative to the :confval:`html_static_path`, or a full
+   URI with scheme like ``http://example.org/script.js``.  The *attributes* is
+   used for attributes of ``<script>`` tag.  It defaults to an empty list.
+
+   Example::
+
+       html_js_files = ['script.js',
+                        'https://example.com/scripts/custom.js',
+                        ('custom.js', {'async': 'async'})]
+
+   .. versionadded:: 1.8
+
 .. confval:: html_static_path
 
    A list of paths that contain custom static files (such as style
@@ -2114,7 +2135,6 @@ information.
      ``'author'``
      ``'logo'``
      ``'makeindex'``
-     ``'shorthandoff'``
 
 .. confval:: latex_docclass
 

doc/usage/restructuredtext/directives.rst

@@ -864,7 +864,7 @@ Including content based on tags
 
    All tags must follow the standard Python identifier syntax as set out in
    the `Identifiers and keywords
-   <https://docs.python.org/2/reference/lexical_analysis.html#identifiers>`_
+   <https://docs.python.org/3/reference/lexical_analysis.html#identifiers>`_
    documentation.  That is, a tag expression may only consist of tags that
    conform to the syntax of Python variables.  In ASCII, this consists of the
    uppercase and lowercase letters ``A`` through ``Z``, the underscore ``_``

doc/usage/restructuredtext/domains.rst

@@ -569,10 +569,10 @@ visibility statement (``public``, ``private`` or ``protected``).
    Full and partial template specialisations can be declared::
 
       .. cpp:class:: template<> \
-                      std::array<bool, 256>
+                     std::array<bool, 256>
 
       .. cpp:class:: template<typename T> \
-                      std::array<T, 42>
+                     std::array<T, 42>
 
 .. rst:directive:: .. cpp:function:: (member) function prototype
 
@@ -702,6 +702,10 @@ visibility statement (``public``, ``private`` or ``protected``).
 
       .. cpp:enumerator:: MyEnum::myOtherEnumerator = 42
 
+.. rst:directive:: .. cpp:union:: name
+
+   Describe a union.
+
 .. rst:directive:: .. cpp:concept:: template-parameter-list name
 
    .. warning:: The support for concepts is experimental. It is based on the
@@ -755,6 +759,41 @@ Some directives support options:
 - ``:tparam-line-spec:``, for templated declarations.
   If specified, each template parameter will be rendered on a separate line.
 
+Anonymous Entities
+~~~~~~~~~~~~~~~~~~
+
+C++ supposrts anonymous namespaces, classes, enums, and unions.
+For the sake of documentation they must be given some name that starts with ``@``,
+e.g., ``@42`` or ``@data``.
+These names can also be used in cross-references and (type) expressions,
+though nested symbols will be found even when omitted.
+The ``@...`` name will always be rendered as **[anonymous]** (possibly as a link).
+
+Example::
+
+   .. cpp:class:: Data
+
+      .. cpp:union:: @data
+
+         .. cpp:var:: int a
+
+         .. cpp:var:: double b
+
+   Explicit ref: :cpp:var:`Data::@data::a`. Short-hand ref: :cpp:var:`Data::a`.
+
+This will be rendered as:
+
+.. cpp:class:: Data
+
+   .. cpp:union:: @data
+
+      .. cpp:var:: int a
+
+      .. cpp:var:: double b
+
+Explicit ref: :cpp:var:`Data::@data::a`. Short-hand ref: :cpp:var:`Data::a`.
+
+
 Constrained Templates
 ~~~~~~~~~~~~~~~~~~~~~
 
@@ -815,24 +854,31 @@ Inline Expressions and Tpes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. rst:role:: cpp:expr
+              cpp:texpr
 
-   A role for inserting a C++ expression or type as inline text.  For example::
+   Insert a C++ expression or type either as inline code (``cpp:expr``)
+   or inline text (``cpp:texpr``). For example::
 
       .. cpp:var:: int a = 42
 
       .. cpp:function:: int f(int i)
 
-      An expression: :cpp:expr:`a * f(a)`.
+      An expression: :cpp:expr:`a * f(a)` (or as text: :cpp:texpr:`a * f(a)`).
-      A type: :cpp:expr:`const MySortedContainer<int>&`.
+
+      A type: :cpp:expr:`const MySortedContainer<int>&`
+      (or as text :cpp:texpr:`const MySortedContainer<int>&`).
 
    will be rendered as follows:
 
-  .. cpp:var:: int a = 42
+   .. cpp:var:: int a = 42
 
-  .. cpp:function:: int f(int i)
+   .. cpp:function:: int f(int i)
+
+   An expression: :cpp:expr:`a * f(a)` (or as text: :cpp:texpr:`a * f(a)`).
+
+   A type: :cpp:expr:`const MySortedContainer<int>&`
+   (or as text :cpp:texpr:`const MySortedContainer<int>&`).
 
-  An expression: :cpp:expr:`a * f(a)`.  A type: :cpp:expr:`const
-  MySortedContainer<int>&`.
 
 Namespacing
 ~~~~~~~~~~~
@@ -880,7 +926,7 @@ The ``cpp:namespace-pop`` directive undoes the most recent
 
          .. cpp:function:: std::size_t size() const
 
-   or:::
+   or::
 
       .. cpp:class:: template<typename T> \
                      std::vector
@@ -949,20 +995,23 @@ These roles link to the given declaration types:
 
 .. admonition:: Note on References with Templates Parameters/Arguments
 
-   Sphinx's syntax to give references a custom title can interfere with linking
+   These roles follow the Sphinx :ref:`xref-syntax` rules. This means care must be
-   to class templates, if nothing follows the closing angle bracket, i.e. if
+   taken when referencing a (partial) template specialization, e.g. if the link looks like
-   the link looks like this: ``:cpp:class:`MyClass<int>```.  This is
+   this: ``:cpp:class:`MyClass<int>```.
-   interpreted as a link to ``int`` with a title of ``MyClass``.  In this case,
+   This is interpreted as a link to ``int`` with a title of ``MyClass``.
-   please escape the opening angle bracket with a backslash, like this:
+   In this case, escape the opening angle bracket with a backslash,
-   ``:cpp:class:`MyClass\<int>```.
+   like this: ``:cpp:class:`MyClass\<int>```.
+
+   When a custom title is not needed it may be useful to use the roles for inline expressions,
+   :rst:role:`cpp:expr` and :rst:role:`cpp:texpr`, where angle brackets do not need escaping.
 
 .. admonition:: Note on References to Overloaded Functions
 
    It is currently impossible to link to a specific version of an overloaded
-   method.  Currently the C++ domain is the first domain that has basic support
+   function.  Currently the C++ domain is the first domain that has basic
-   for overloaded methods and until there is more data for comparison we don't
+   support for overloaded functions and until there is more data for comparison
-   want to select a bad syntax to reference a specific overload.  Currently
+   we don't want to select a bad syntax to reference a specific overload.
-   Sphinx will link to the first overloaded version of the method / function.
+   Currently Sphinx will link to the first overloaded version of the function.
 
 Declarations without template parameters and template arguments
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -983,13 +1032,13 @@ Assume the following declarations.
       .. cpp:class:: template<typename TInner> \
                      Inner
 
-In general the reference must include the template paraemter declarations,
+In general the reference must include the template parameter declarations,
 e.g., ``template\<typename TOuter> Wrapper::Outer``
 (:cpp:class:`template\<typename TOuter> Wrapper::Outer`).  Currently the lookup
 only succeed if the template parameter identifiers are equal strings. That is,
 ``template\<typename UOuter> Wrapper::Outer`` will not work.
 
-The inner class template can not be directly referenced, unless the current
+The inner class template cannot be directly referenced, unless the current
 namespace is changed or the following shorthand is used.  If a template
 parameter list is omitted, then the lookup will assume either a template or a
 non-template, but not a partial template specialisation.  This means the

sphinx/apidoc.py

@@ -9,6 +9,7 @@
     :license: BSD, see LICENSE for details.
 """
 
+import sys
 import warnings
 
 from sphinx.deprecation import RemovedInSphinx20Warning
@@ -16,19 +17,18 @@ from sphinx.ext.apidoc import main as _main
 
 if False:
     # For type annotation
-    from typing import Any  # NOQA
+    from typing import List  # NOQA
     from sphinx.application import Sphinx  # NOQA
 
 
-def main(*args, **kwargs):
+def main(argv=sys.argv):
-    # type: (Any, Any) -> None
+    # type: (List[str]) -> None
     warnings.warn(
         '`sphinx.apidoc.main()` has moved to `sphinx.ext.apidoc.main()`.',
         RemovedInSphinx20Warning,
         stacklevel=2,
     )
-    args = args[1:]  # skip first argument to adjust arguments (refs: #4615)
+    _main(argv[1:])  # skip first argument to adjust arguments (refs: #4615)
-    _main(*args, **kwargs)
 
 
 # So program can be started with "python -m sphinx.apidoc ..."

sphinx/application.py

@@ -13,7 +13,6 @@
 from __future__ import print_function
 
 import os
-import posixpath
 import sys
 import warnings
 from collections import deque
@@ -964,6 +963,7 @@ class Sphinx(object):
         document.
 
         .. list-table:: priority range categories for Sphinx transforms
+           :widths: 20,80
 
            * - Priority
              - Main purpose in Sphinx
@@ -998,25 +998,38 @@ class Sphinx(object):
         """
         self.registry.add_post_transform(transform)
 
-    def add_javascript(self, filename):
+    def add_javascript(self, filename, **kwargs):
-        # type: (unicode) -> None
+        # type: (unicode, **unicode) -> None
+        """An alias of :meth:`add_js_file`."""
+        warnings.warn('The app.add_javascript() is deprecated. '
+                      'Please use app.add_js_file() instead.',
+                      RemovedInSphinx40Warning)
+        self.add_js_file(filename, **kwargs)
+
+    def add_js_file(self, filename, **kwargs):
+        # type: (unicode, **unicode) -> None
         """Register a JavaScript file to include in the HTML output.
 
         Add *filename* to the list of JavaScript files that the default HTML
         template will include.  The filename must be relative to the HTML
-        static path, see :confval:`the docs for the config value
+        static path , or a full URI with scheme.  The keyword arguments are
-        <html_static_path>`.  A full URI with scheme, like
+        also accepted for attributes of ``<script>`` tag.
-        ``http://example.org/foo.js``, is also supported.
+
+        Example::
+
+            app.add_js_file('example.js')
+            # => <scrtipt src="_static/example.js"></script>
+
+            app.add_js_file('example.js', async="async")
+            # => <scrtipt src="_static/example.js" async="async"></script>
 
         .. versionadded:: 0.5
+
+        .. versionchanged:: 1.8
+           Renamed from ``app.add_javascript()``.
+           And it allows keyword arguments as attributes of script tag.
         """
-        logger.debug('[app] adding javascript: %r', filename)
+        self.registry.add_js_file(filename, **kwargs)
-        from sphinx.builders.html import StandaloneHTMLBuilder
-        if '://' in filename:
-            StandaloneHTMLBuilder.script_files.append(filename)
-        else:
-            StandaloneHTMLBuilder.script_files.append(
-                posixpath.join('_static', filename))
 
     def add_css_file(self, filename, **kwargs):
         # type: (unicode, **unicode) -> None

sphinx/builders/__init__.py

@@ -372,14 +372,14 @@ class Builder(object):
         else:
             logger.info(__('none found'))
 
-        if updated_docnames:
+        # save the environment
-            # save the environment
+        from sphinx.application import ENV_PICKLE_FILENAME
-            from sphinx.application import ENV_PICKLE_FILENAME
+        logger.info(bold(__('pickling environment... ')), nonl=True)
-            logger.info(bold(__('pickling environment... ')), nonl=True)
+        with open(path.join(self.doctreedir, ENV_PICKLE_FILENAME), 'wb') as f:
-            with open(path.join(self.doctreedir, ENV_PICKLE_FILENAME), 'wb') as f:
+            pickle.dump(self.env, f, pickle.HIGHEST_PROTOCOL)
-                pickle.dump(self.env, f, pickle.HIGHEST_PROTOCOL)
+        logger.info(__('done'))
-            logger.info(__('done'))
 
+        if updated_docnames:
             # global actions
             self.app.phase = BuildPhase.CONSISTENCY_CHECK
             logger.info(bold(__('checking consistency... ')), nonl=True)

sphinx/builders/_epub_base.py

@@ -272,6 +272,16 @@ class EpubBuilder(StandaloneHTMLBuilder):
                     node['refuri'] = self.fix_fragment(m.group(1), m.group(2))
             if 'refid' in node:
                 node['refid'] = self.fix_fragment('', node['refid'])
+        for node in tree.traverse(nodes.target):
+            for i, node_id in enumerate(node['ids']):
+                if ':' in node_id:
+                    node['ids'][i] = self.fix_fragment('', node_id)
+
+            next_node = node.next_node(siblings=True)
+            if next_node and isinstance(next_node, nodes.Element):
+                for i, node_id in enumerate(next_node['ids']):
+                    if ':' in node_id:
+                        next_node['ids'][i] = self.fix_fragment('', node_id)
         for node in tree.traverse(addnodes.desc_signature):
             ids = node.attributes['ids']
             newids = []

sphinx/builders/html.py

@@ -13,6 +13,7 @@ import codecs
 import posixpath
 import re
 import sys
+import types
 import warnings
 from hashlib import md5
 from os import path
@@ -92,7 +93,7 @@ def get_stable_hash(obj):
 
 
 class CSSContainer(list):
-    """The container of stylesheets.
+    """The container for stylesheets.
 
     To support the extensions which access the container directly, this wraps
     the entry with Stylesheet class.
@@ -139,7 +140,7 @@ class CSSContainer(list):
 
 
 class Stylesheet(text_type):
-    """The metadata of stylesheet.
+    """A metadata of stylesheet.
 
     To keep compatibility with old themes, an instance of stylesheet behaves as
     its filename (str).
@@ -162,6 +163,59 @@ class Stylesheet(text_type):
         return self
 
 
+class JSContainer(list):
+    """The container for JavaScript scripts."""
+    def insert(self, index, obj):
+        # type: (int, unicode) -> None
+        warnings.warn('builder.script_files is deprecated. '
+                      'Please use app.add_js_file() instead.',
+                      RemovedInSphinx30Warning)
+        super(JSContainer, self).insert(index, obj)
+
+    def extend(self, other):  # type: ignore
+        # type: (List[unicode]) -> None
+        warnings.warn('builder.script_files is deprecated. '
+                      'Please use app.add_js_file() instead.',
+                      RemovedInSphinx30Warning)
+        for item in other:
+            self.append(item)
+
+    def __iadd__(self, other):  # type: ignore
+        # type: (List[unicode]) -> JSContainer
+        warnings.warn('builder.script_files is deprecated. '
+                      'Please use app.add_js_file() instead.',
+                      RemovedInSphinx30Warning)
+        for item in other:
+            self.append(item)
+        return self
+
+    def __add__(self, other):
+        # type: (List[unicode]) -> JSContainer
+        ret = JSContainer(self)
+        ret += other
+        return ret
+
+
+class JavaScript(text_type):
+    """A metadata of javascript file.
+
+    To keep compatibility with old themes, an instance of javascript behaves as
+    its filename (str).
+    """
+
+    attributes = None   # type: Dict[unicode, unicode]
+    filename = None     # type: unicode
+
+    def __new__(cls, filename, **attributes):
+        # type: (unicode, **unicode) -> None
+        self = text_type.__new__(cls, filename)  # type: ignore
+        self.filename = filename
+        self.attributes = attributes
+        self.attributes.setdefault('type', 'text/javascript')
+
+        return self
+
+
 class BuildInfo(object):
     """buildinfo file manipulator.
 
@@ -246,10 +300,6 @@ class StandaloneHTMLBuilder(Builder):
     # use html5 translator by default
     default_html5_translator = False
 
-    # This is a class attribute because it is mutated by Sphinx.add_javascript.
-    script_files = ['_static/jquery.js', '_static/underscore.js',
-                    '_static/doctools.js']  # type: List[unicode]
-
     imgpath = None          # type: unicode
     domain_indices = []     # type: List[Tuple[unicode, Type[Index], List[Tuple[unicode, List[List[Union[unicode, int]]]]], bool]]  # NOQA
 
@@ -263,6 +313,9 @@ class StandaloneHTMLBuilder(Builder):
         # CSS files
         self.css_files = CSSContainer()  # type: List[Dict[unicode, unicode]]
 
+        # JS files
+        self.script_files = JSContainer()  # type: List[JavaScript]
+
     def init(self):
         # type: () -> None
         self.build_info = self.create_build_info()
@@ -276,6 +329,7 @@ class StandaloneHTMLBuilder(Builder):
         self.init_templates()
         self.init_highlighter()
         self.init_css_files()
+        self.init_js_files()
         if self.config.html_file_suffix is not None:
             self.out_suffix = self.config.html_file_suffix
 
@@ -284,9 +338,6 @@ class StandaloneHTMLBuilder(Builder):
         else:
             self.link_suffix = self.out_suffix
 
-        if self.config.language is not None:
-            if self._get_translations_js():
-                self.script_files.append('_static/translations.js')
         self.use_index = self.get_builder_config('use_index', 'html')
 
         if self.config.html_experimental_html5_writer and not html5_ready:
@@ -353,6 +404,28 @@ class StandaloneHTMLBuilder(Builder):
 
         self.css_files.append(Stylesheet(filename, **kwargs))  # type: ignore
 
+    def init_js_files(self):
+        # type: () -> None
+        self.add_js_file('jquery.js')
+        self.add_js_file('underscore.js')
+        self.add_js_file('doctools.js')
+
+        for filename, attrs in self.app.registry.js_files:
+            self.add_js_file(filename, **attrs)
+
+        for filename, attrs in self.get_builder_config('js_files', 'html'):
+            self.add_js_file(filename, **attrs)
+
+        if self.config.language and self._get_translations_js():
+            self.add_js_file('translations.js')
+
+    def add_js_file(self, filename, **kwargs):
+        # type: (unicode, **unicode) -> None
+        if '://' not in filename:
+            filename = posixpath.join('_static', filename)
+
+        self.script_files.append(JavaScript(filename, **kwargs))  # type: ignore
+
     @property
     def default_translator_class(self):
         # type: () -> nodes.NodeVisitor
@@ -941,19 +1014,27 @@ class StandaloneHTMLBuilder(Builder):
         def has_wildcard(pattern):
             # type: (unicode) -> bool
             return any(char in pattern for char in '*?[')
-        sidebars = self.theme.get_config('theme', 'sidebars', None)
+        sidebars = None
         matched = None
         customsidebar = None
 
         # default sidebars settings for selected theme
-        theme_default_sidebars = self.theme.get_config('theme', 'sidebars', None)
+        if self.theme.name == 'alabaster':
-        if theme_default_sidebars:
-            sidebars = [name.strip() for name in theme_default_sidebars.split(',')]
-        elif self.theme.name == 'alabaster':
             # provide default settings for alabaster (for compatibility)
             # Note: this will be removed before Sphinx-2.0
-            sidebars = ['about.html', 'navigation.html', 'relation.html',
+            try:
-                        'searchbox.html', 'donate.html']
+                # get default sidebars settings from alabaster (if defined)
+                theme_default_sidebars = self.theme.config.get('theme', 'sidebars')
+                if theme_default_sidebars:
+                    sidebars = [name.strip() for name in theme_default_sidebars.split(',')]
+            except Exception:
+                # fallback to better default settings
+                sidebars = ['about.html', 'navigation.html', 'relations.html',
+                            'searchbox.html', 'donate.html']
+        else:
+            theme_default_sidebars = self.theme.get_config('theme', 'sidebars', None)
+            if theme_default_sidebars:
+                sidebars = [name.strip() for name in theme_default_sidebars.split(',')]
 
         # user sidebar settings
         for pattern, patsidebars in iteritems(self.config.html_sidebars):
@@ -1004,6 +1085,12 @@ class StandaloneHTMLBuilder(Builder):
         # part, which relative_uri doesn't really like...
         default_baseuri = default_baseuri.rsplit('#', 1)[0]
 
+        if self.config.html_baseurl:
+            ctx['pageurl'] = posixpath.join(self.config.html_baseurl,
+                                            pagename + self.out_suffix)
+        else:
+            ctx['pageurl'] = None
+
         def pathto(otheruri, resource=False, baseuri=default_baseuri):
             # type: (unicode, bool, unicode) -> unicode
             if resource and '://' in otheruri:
@@ -1357,6 +1444,7 @@ class SerializingHTMLBuilder(StandaloneHTMLBuilder):
         self.init_templates()
         self.init_highlighter()
         self.init_css_files()
+        self.init_js_files()
         self.use_index = self.get_builder_config('use_index', 'html')
 
     def get_target_uri(self, docname, typ=None):
@@ -1390,6 +1478,11 @@ class SerializingHTMLBuilder(StandaloneHTMLBuilder):
         # actually rendered
         self.app.emit('html-page-context', pagename, templatename, ctx, event_arg)
 
+        # make context object serializable
+        for key in list(ctx):
+            if isinstance(ctx[key], types.FunctionType):
+                del ctx[key]
+
         ensuredir(path.dirname(outfilename))
         self.dump_context(ctx, outfilename)
 
@@ -1479,6 +1572,50 @@ def convert_html_css_files(app, config):
     config.html_css_files = html_css_files  # type: ignore
 
 
+def convert_html_js_files(app, config):
+    # type: (Sphinx, Config) -> None
+    """This converts string styled html_js_files to tuple styled one."""
+    html_js_files = []  # type: List[Tuple[unicode, Dict]]
+    for entry in config.html_js_files:
+        if isinstance(entry, string_types):
+            html_js_files.append((entry, {}))
+        else:
+            try:
+                filename, attrs = entry
+                html_js_files.append((filename, attrs))
+            except Exception:
+                logger.warning(__('invalid js_file: %r, ignored'), entry)
+                continue
+
+    config.html_js_files = html_js_files  # type: ignore
+
+
+def setup_js_tag_helper(app, pagename, templatexname, context, doctree):
+    # type: (Sphinx, unicode, unicode, Dict, nodes.Node) -> None
+    """Set up js_tag() template helper.
+
+    .. note:: This set up function is added to keep compatibility with webhelper.
+    """
+    pathto = context.get('pathto')
+
+    def js_tag(js):
+        # type: (JavaScript) -> unicode
+        attrs = []
+        if isinstance(js, JavaScript):
+            for key in sorted(js.attributes):
+                value = js.attributes[key]
+                if value is not None:
+                    attrs.append('%s="%s"' % (key, htmlescape(value, True)))
+            attrs.append('src="%s"' % pathto(js.filename, resource=True))
+        else:
+            # str value (old styled)
+            attrs.append('type="text/javascript"')
+            attrs.append('src="%s"' % pathto(js, resource=True))
+        return '<script %s></script>' % ' '.join(attrs)
+
+    context['js_tag'] = js_tag
+
+
 def setup(app):
     # type: (Sphinx) -> Dict[unicode, Any]
     # builders
@@ -1500,6 +1637,7 @@ def setup(app):
     app.add_config_value('html_logo', None, 'html', string_classes)
     app.add_config_value('html_favicon', None, 'html', string_classes)
     app.add_config_value('html_css_files', [], 'html')
+    app.add_config_value('html_js_files', [], 'html')
     app.add_config_value('html_static_path', [], 'html')
     app.add_config_value('html_extra_path', [], 'html')
     app.add_config_value('html_last_updated_fmt', None, 'html', string_classes)
@@ -1526,9 +1664,12 @@ def setup(app):
     app.add_config_value('html_search_scorer', '', None)
     app.add_config_value('html_scaled_image_link', True, 'html')
     app.add_config_value('html_experimental_html5_writer', None, 'html')
+    app.add_config_value('html_baseurl', '', 'html')
 
     # event handlers
     app.connect('config-inited', convert_html_css_files)
+    app.connect('config-inited', convert_html_js_files)
+    app.connect('html-page-context', setup_js_tag_helper)
 
     return {
         'version': 'builtin',

sphinx/builders/latex/__init__.py

@@ -13,14 +13,14 @@ import os
 from os import path
 
 from docutils.frontend import OptionParser
-from docutils.io import FileOutput
 from six import text_type
 
 from sphinx import package_dir, addnodes, highlighting
 from sphinx.builders import Builder
 from sphinx.builders.latex.transforms import (
     BibliographyTransform, CitationReferenceTransform, MathReferenceTransform,
-    FootnoteDocnameUpdater, LaTeXFootnoteTransform, ShowUrlsTransform
+    FootnoteDocnameUpdater, LaTeXFootnoteTransform, LiteralBlockTransform,
+    ShowUrlsTransform, DocumentTargetTransform,
 )
 from sphinx.config import string_classes, ENUM
 from sphinx.environment import NoUri
@@ -30,7 +30,7 @@ from sphinx.locale import _, __
 from sphinx.transforms import SphinxTransformer
 from sphinx.util import texescape, logging, status_iterator
 from sphinx.util.console import bold, darkgreen  # type: ignore
-from sphinx.util.docutils import new_document
+from sphinx.util.docutils import SphinxFileOutput, new_document
 from sphinx.util.fileutil import copy_asset_file
 from sphinx.util.nodes import inline_all_toctrees
 from sphinx.util.osutil import SEP, make_filename
@@ -133,9 +133,8 @@ class LaTeXBuilder(Builder):
             toctree_only = False
             if len(entry) > 5:
                 toctree_only = entry[5]
-            destination = FileOutput(
+            destination = SphinxFileOutput(destination_path=path.join(self.outdir, targetname),
-                destination_path=path.join(self.outdir, targetname),
+                                           encoding='utf-8', overwrite_if_changed=True)
-                encoding='utf-8')
             logger.info(__("processing %s..."), targetname, nonl=1)
             toctrees = self.env.get_doctree(docname).traverse(addnodes.toctree)
             if toctrees:
@@ -223,7 +222,9 @@ class LaTeXBuilder(Builder):
         transformer.set_environment(self.env)
         transformer.add_transforms([BibliographyTransform,
                                     ShowUrlsTransform,
-                                    LaTeXFootnoteTransform])
+                                    LaTeXFootnoteTransform,
+                                    LiteralBlockTransform,
+                                    DocumentTargetTransform])
         transformer.apply_transforms()
 
     def finish(self):

sphinx/builders/latex/nodes.py

@@ -12,6 +12,11 @@
 from docutils import nodes
 
 
+class captioned_literal_block(nodes.container):
+    """A node for a container of literal_block having a caption."""
+    pass
+
+
 class footnotemark(nodes.Inline, nodes.Referential, nodes.TextElement):
     """A node represents ``\footnotemark``."""
     pass

sphinx/builders/latex/transforms.py

@@ -13,7 +13,7 @@ from docutils import nodes
 
 from sphinx import addnodes
 from sphinx.builders.latex.nodes import (
-    footnotemark, footnotetext, math_reference, thebibliography
+    captioned_literal_block, footnotemark, footnotetext, math_reference, thebibliography
 )
 from sphinx.transforms import SphinxTransform
 
@@ -566,3 +566,33 @@ class MathReferenceTransform(SphinxTransform):
                 if docname:
                     refnode = math_reference('', docname=docname, target=node['reftarget'])
                     node.replace_self(refnode)
+
+
+class LiteralBlockTransform(SphinxTransform):
+    """Replace container nodes for literal_block by captioned_literal_block."""
+    default_priority = 400
+
+    def apply(self):
+        # type: () -> None
+        if self.app.builder.name != 'latex':
+            return
+
+        for node in self.document.traverse(nodes.container):
+            if node.get('literal_block') is True:
+                newnode = captioned_literal_block('', *node.children, **node.attributes)
+                node.replace_self(newnode)
+
+
+class DocumentTargetTransform(SphinxTransform):
+    """Add :doc label to the first section of each document."""
+    default_priority = 400
+
+    def apply(self):
+        # type: () -> None
+        if self.app.builder.name != 'latex':
+            return
+
+        for node in self.document.traverse(addnodes.start_of_file):
+            section = node.next_node(nodes.section)
+            if section:
+                section['ids'].append(':doc')  # special label for :doc:

sphinx/builders/qthelp.py

@@ -140,7 +140,7 @@ class QtHelpBuilder(StandaloneHTMLBuilder):
         else:
             nspace = 'org.sphinx.%s.%s' % (outname, self.config.version)
 
-        nspace = re.sub('[^a-zA-Z0-9.]', '', nspace)
+        nspace = re.sub('[^a-zA-Z0-9.\-]', '', nspace)
         nspace = re.sub(r'\.+', '.', nspace).strip('.')
         nspace = nspace.lower()
 

sphinx/cmd/quickstart.py

@@ -41,7 +41,7 @@ from sphinx import __display_version__, package_dir
 from sphinx.locale import __
 from sphinx.util import texescape
 from sphinx.util.console import (  # type: ignore
-    purple, bold, red, turquoise, nocolor, color_terminal
+    colorize, bold, red, turquoise, nocolor, color_terminal
 )
 from sphinx.util.osutil import ensuredir, make_filename
 from sphinx.util.template import SphinxRenderer
@@ -85,8 +85,14 @@ PROMPT_PREFIX = '> '
 # function to get input from terminal -- overridden by the test suite
 def term_input(prompt):
     # type: (unicode) -> unicode
-    print(prompt, end='')
+    if sys.platform == 'win32':
-    return input('')
+        # Important: On windows, readline is not enabled by default.  In these
+        #            environment, escape sequences have been broken.  To avoid the
+        #            problem, quickstart uses ``print()`` to show prompt.
+        print(prompt, end='')
+        return input('')
+    else:
+        return input(prompt)
 
 
 class ValidationError(Exception):
@@ -186,7 +192,7 @@ def do_prompt(text, default=None, validator=nonempty):
                         prompt = prompt.encode('utf-8')
                     except UnicodeEncodeError:
                         prompt = prompt.encode('latin1')
-        prompt = purple(prompt)
+        prompt = colorize('purple', prompt, input_mode=True)
         x = term_input(prompt).strip()
         if default and not x:
             x = default

sphinx/config.py

@@ -365,7 +365,7 @@ def eval_config_file(filename, tags):
                      "called sys.exit()")
             raise ConfigError(msg)
         except Exception:
-            msg = __("There is a programable error in your configuration file:\n\n%s")
+            msg = __("There is a programmable error in your configuration file:\n\n%s")
             raise ConfigError(msg % traceback.format_exc())
 
     return namespace

sphinx/directives/code.py

@@ -249,7 +249,7 @@ class LiteralIncludeReader(object):
         new_lines = self.read_file(self.filename)
         old_filename = self.options.get('diff')
         old_lines = self.read_file(old_filename)
-        diff = unified_diff(old_lines, new_lines, old_filename, self.filename)  # type: ignore
+        diff = unified_diff(old_lines, new_lines, old_filename, self.filename)
         return list(diff)
 
     def pyobject_filter(self, lines, location=None):

sphinx/domains/c.py

@@ -147,7 +147,8 @@ class CObject(ObjectDescription):
             fullname = name
 
         if not arglist:
-            if self.objtype == 'function':
+            if self.objtype == 'function' or \
+                    self.objtype == 'macro' and sig.rstrip().endswith('()'):
                 # for functions, add an empty parameter list
                 signode += addnodes.desc_parameterlist()
             if const:

sphinx/domains/std.py

@@ -432,7 +432,7 @@ class ProductionList(SphinxDirective):
             subnode = addnodes.production()
             subnode['tokenname'] = name.strip()
             if subnode['tokenname']:
-                idname = 'grammar-token-%s' % subnode['tokenname']
+                idname = nodes.make_id('grammar-token-%s' % subnode['tokenname'])
                 if idname not in self.state.document.ids:
                     subnode['ids'].append(idname)
                 self.state.document.note_implicit_target(subnode, subnode)

sphinx/environment/__init__.py

@@ -313,19 +313,6 @@ class BuildEnvironment(object):
         """Like :meth:`warn`, but with source information taken from *node*."""
         self._warnfunc(msg, '%s:%s' % get_source_line(node), **kwargs)
 
-    def need_refresh(self, app):
-        # type: (Sphinx) -> Tuple[bool, unicode]
-        """Check refresh environment is needed.
-
-        If needed, this method returns the reason for refresh.
-        """
-        if self.version != app.registry.get_envversion(app):
-            return True, __('build environment version not current')
-        elif self.srcdir != app.srcdir:
-            return True, __('source directory has changed')
-        else:
-            return False, None
-
     def clear_doc(self, docname):
         # type: (unicode) -> None
         """Remove all traces of a source file in the inventory."""

sphinx/environment/collectors/toctree.py

@@ -169,11 +169,12 @@ class TocTreeCollector(EnvironmentCollector):
                 elif isinstance(subnode, addnodes.compact_paragraph):
                     numstack[-1] += 1
                     if depth > 0:
-                        number = tuple(numstack)
+                        number = list(numstack)
+                        secnums[subnode[0]['anchorname']] = tuple(numstack)
                     else:
                         number = None
-                    secnums[subnode[0]['anchorname']] = \
+                        secnums[subnode[0]['anchorname']] = None
-                        subnode[0]['secnumber'] = number
+                    subnode[0]['secnumber'] = number
                     if titlenode:
                         titlenode['secnumber'] = number
                         titlenode = None

sphinx/ext/autodoc/__init__.py

@@ -643,11 +643,17 @@ class Documenter(object):
             # should be skipped
             if self.env.app:
                 # let extensions preprocess docstrings
-                skip_user = self.env.app.emit_firstresult(
+                try:
-                    'autodoc-skip-member', self.objtype, membername, member,
+                    skip_user = self.env.app.emit_firstresult(
-                    not keep, self.options)
+                        'autodoc-skip-member', self.objtype, membername, member,
-                if skip_user is not None:
+                        not keep, self.options)
-                    keep = not skip_user
+                    if skip_user is not None:
+                        keep = not skip_user
+                except Exception as exc:
+                    logger.warning(__('autodoc: failed to determine %r to be documented.'
+                                      'the following exception was raised:\n%s'),
+                                   member, exc)
+                    keep = False
 
             if keep:
                 ret.append((membername, member, isattr))

sphinx/ext/autodoc/importer.py

@@ -23,7 +23,7 @@ from sphinx.util.inspect import isenumclass, safe_getattr
 
 if False:
     # For type annotation
-    from typing import Any, Callable, Dict, Generator, List, Optional, Tuple  # NOQA
+    from typing import Any, Callable, Dict, Generator, Iterator, List, Optional, Tuple  # NOQA
 
 logger = logging.getLogger(__name__)
 
@@ -41,7 +41,7 @@ class _MockObject(object):
 
     def __init__(self, *args, **kwargs):
         # type: (Any, Any) -> None
-        pass
+        self.__qualname__ = ''
 
     def __len__(self):
         # type: () -> int
@@ -52,8 +52,8 @@ class _MockObject(object):
         return False
 
     def __iter__(self):
-        # type: () -> None
+        # type: () -> Iterator
-        pass
+        return iter([])
 
     def __mro_entries__(self, bases):
         # type: (Tuple) -> Tuple

sphinx/ext/autosummary/generate.py

@@ -203,6 +203,8 @@ def generate_autosummary_docs(sources, output_dir=None, suffix='.rst',
                     get_members(obj, 'exception', imported=imported_members)
             elif doc.objtype == 'class':
                 ns['members'] = dir(obj)
+                ns['inherited_members'] = \
+                    set(dir(obj)) - set(obj.__dict__.keys())
                 ns['methods'], ns['all_methods'] = \
                     get_members(obj, 'method', ['__init__'])
                 ns['attributes'], ns['all_attributes'] = \

sphinx/ext/coverage.py

@@ -56,7 +56,7 @@ class CoverageBuilder(Builder):
     """
     name = 'coverage'
     epilog = __('Testing of coverage in the sources finished, look at the '
-                'results in %(outdir)s/python.txt.')
+                'results in %(outdir)s' + path.sep + 'python.txt.')
 
     def init(self):
         # type: () -> None

sphinx/ext/jsmath.py

@@ -64,7 +64,8 @@ def builder_inited(app):
     if not app.config.jsmath_path:
         raise ExtensionError('jsmath_path config value must be set for the '
                              'jsmath extension to work')
-    app.add_javascript(app.config.jsmath_path)
+    if app.builder.format == 'html':
+        app.builder.add_js_file(app.config.jsmath_path)  # type: ignore
 
 
 def setup(app):

sphinx/ext/mathjax.py

@@ -72,7 +72,11 @@ def builder_inited(app):
     if not app.config.mathjax_path:
         raise ExtensionError('mathjax_path config value must be set for the '
                              'mathjax extension to work')
-    app.add_javascript(app.config.mathjax_path)
+    if app.builder.format == 'html':
+        options = {'async': 'async'}
+        if app.config.mathjax_options:
+            options.update(app.config.mathjax_options)
+        app.builder.add_js_file(app.config.mathjax_path, **options)  # type: ignore
 
 
 def setup(app):
@@ -86,7 +90,8 @@ def setup(app):
     # https://docs.mathjax.org/en/latest/start.html#secure-access-to-the-cdn
     app.add_config_value('mathjax_path',
                          'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?'
-                         'config=TeX-AMS-MML_HTMLorMML', False)
+                         'config=TeX-AMS-MML_HTMLorMML', 'html')
+    app.add_config_value('mathjax_options', {}, 'html')
     app.add_config_value('mathjax_inline', [r'\(', r'\)'], 'html')
     app.add_config_value('mathjax_display', [r'\[', r'\]'], 'html')
     app.connect('builder-inited', builder_inited)

sphinx/ext/napoleon/iterators.py

@@ -41,7 +41,7 @@ class peek_iter(object):
     See Also
     --------
     `peek_iter` can operate as a drop in replacement for the built-in
-    `iter <https://docs.python.org/2/library/functions.html#iter>`_ function.
+    `iter <https://docs.python.org/3/library/functions.html#iter>`_ function.
 
     Attributes
     ----------

sphinx/io.py

@@ -13,6 +13,7 @@ import re
 
 from docutils.core import Publisher
 from docutils.io import FileInput, NullOutput
+from docutils.parsers.rst import Parser as RSTParser
 from docutils.readers import standalone
 from docutils.statemachine import StringList, string2lines
 from docutils.writers import UnfilteredWriter
@@ -304,6 +305,13 @@ def read_doc(app, env, filename):
     source = input_class(app, env, source=None, source_path=filename,
                          encoding=env.config.source_encoding)
     parser = app.registry.create_source_parser(app, filetype)
+    if parser.__class__.__name__ == 'CommonMarkParser' and parser.settings_spec == ():
+        # a workaround for recommonmark
+        #   If recommonmark.AutoStrictify is enabled, the parser invokes reST parser
+        #   internally.  But recommonmark-0.4.0 does not provide settings_spec for reST
+        #   parser.  As a workaround, this copies settings_spec for RSTParser to the
+        #   CommonMarkParser.
+        parser.settings_spec = RSTParser.settings_spec
 
     pub = Publisher(reader=reader,
                     parser=parser,

sphinx/parsers.py

@@ -41,6 +41,9 @@ class Parser(docutils.parsers.Parser):
         Emit a warning. (Same as :meth:`sphinx.application.Sphinx.warn()`)
     self.info()
         Emit a informational message. (Same as :meth:`sphinx.application.Sphinx.info()`)
+
+    .. deprecated:: 1.6
+       ``warn()`` and ``info()`` is deprecated.  Use :mod:`sphinx.util.logging` instead.
     """
 
     def set_application(self, app):

sphinx/pycode/parser.py

@@ -34,6 +34,11 @@ else:
     ASSIGN_NODES = (ast.Assign)
 
 
+def filter_whitespace(code):
+    # type: (unicode) -> unicode
+    return code.replace('\f', ' ')  # replace FF (form feed) with whitespace
+
+
 def get_assign_targets(node):
     # type: (ast.AST) -> List[ast.expr]
     """Get list of targets from Assign and AnnAssign node."""
@@ -225,12 +230,13 @@ class AfterCommentParser(TokenProcessor):
     def parse(self):
         # type: () -> None
         """Parse the code and obtain comment after assignment."""
-        # skip lvalue (until '=' operator)
+        # skip lvalue (or whole of AnnAssign)
-        while self.fetch_token() != [OP, '=']:
+        while not self.fetch_token().match([OP, '='], NEWLINE, COMMENT):
             assert self.current
 
-        # skip rvalue
+        # skip rvalue (if exists)
-        self.fetch_rvalue()
+        if self.current == [OP, '=']:
+            self.fetch_rvalue()
 
         if self.current == COMMENT:
             self.comment = self.current.value
@@ -466,7 +472,7 @@ class Parser(object):
 
     def __init__(self, code, encoding='utf-8'):
         # type: (unicode, unicode) -> None
-        self.code = code
+        self.code = filter_whitespace(code)
         self.encoding = encoding
         self.comments = {}          # type: Dict[Tuple[unicode, unicode], unicode]
         self.deforders = {}         # type: Dict[unicode, int]

sphinx/registry.py

@@ -91,6 +91,9 @@ class SphinxComponentRegistry(object):
         #: a dict of node class -> tuple of figtype and title_getter function
         self.enumerable_nodes = {}      # type: Dict[nodes.Node, Tuple[unicode, TitleGetter]]
 
+        #: js_files; list of JS paths or URLs
+        self.js_files = []              # type: List[Tuple[unicode, Dict[unicode, unicode]]]
+
         #: LaTeX packages; list of package names and its options
         self.latex_packages = []        # type: List[Tuple[unicode, unicode]]
 
@@ -418,6 +421,11 @@ class SphinxComponentRegistry(object):
     def add_css_files(self, filename, **attributes):
         self.css_files.append((filename, attributes))
 
+    def add_js_file(self, filename, **attributes):
+        # type: (unicode, **unicode) -> None
+        logger.debug('[app] adding js_file: %r, %r', filename, attributes)
+        self.js_files.append((filename, attributes))  # type: ignore
+
     def add_latex_package(self, name, options):
         # type: (unicode, unicode) -> None
         logger.debug('[app] adding latex package: %r', name)

sphinx/templates/latex/longtable.tex_t

@@ -9,8 +9,12 @@
 <%= table.get_colspec() %>
 <%- if table.caption -%>
 \caption{<%= ''.join(table.caption) %>\strut}<%= labels %>\\*[\sphinxlongtablecapskipadjust]
-<% endif -%>
 \hline
+<% elif labels -%>
+\hline\noalign{\phantomsection<%= labels %>}%
+<% else -%>
+\hline
+<% endif -%>
 <%= ''.join(table.header) %>
 \endfirsthead
 

sphinx/templates/latex/tabular.tex_t

@@ -14,6 +14,8 @@
 \sphinxcapstartof{table}
 \sphinxcaption{<%= ''.join(table.caption) %>}<%= labels %>
 \sphinxaftercaption
+<% elif labels -%>
+\phantomsection<%= labels %>\nobreak
 <% endif -%>
 \begin{tabular}[t]<%= table.get_colspec() -%>
 \hline

sphinx/templates/latex/tabulary.tex_t

@@ -14,6 +14,8 @@
 \sphinxcapstartof{table}
 \sphinxcaption{<%= ''.join(table.caption) %>}<%= labels %>
 \sphinxaftercaption
+<% elif labels -%>
+\phantomsection<%= labels %>\nobreak
 <% endif -%>
 \begin{tabulary}{\linewidth}[t]<%= table.get_colspec() -%>
 \hline

sphinx/testing/fixtures.py

@@ -42,7 +42,10 @@ def app_params(request, test_params, shared_result, sphinx_test_tempdir, rootdir
 
     # ##### process pytest.mark.sphinx
 
-    markers = request.node.get_marker("sphinx")
+    if hasattr(request.node, 'iter_markers'):  # pytest-3.6.0 or newer
+        markers = request.node.iter_markers("sphinx")
+    else:
+        markers = request.node.get_marker("sphinx")
     pargs = {}
     kwargs = {}  # type: Dict[str, str]
 
@@ -89,7 +92,10 @@ def test_params(request):
        have same 'shared_result' value.
        **NOTE**: You can not specify shared_result and srcdir in same time.
     """
-    env = request.node.get_marker('test_params')
+    if hasattr(request.node, 'get_closest_marker'):  # pytest-3.6.0 or newer
+        env = request.node.get_closest_marker('test_params')
+    else:
+        env = request.node.get_marker('test_params')
     kwargs = env.kwargs if env else {}
     result = {
         'shared_result': None,

sphinx/themes/basic/layout.html

@@ -88,8 +88,8 @@
 
 {%- macro script() %}
     <script type="text/javascript" id="documentation_options" data-url_root="{{ pathto('', 1) }}" src="{{ pathto('_static/documentation_options.js', 1) }}"></script>
-    {%- for scriptfile in script_files %}
+    {%- for js in script_files %}
-    <script type="text/javascript" src="{{ pathto(scriptfile, 1) }}"></script>
+    {{ js_tag(js) }}
     {%- endfor %}
 {%- endmacro %}
 
@@ -130,6 +130,9 @@
     {%- block scripts %}
     {{- script() }}
     {%- endblock %}
+    {%- if pageurl %}
+    <link rel="canonical" href="{{ pageurl }}" />
+    {%- endif %}
     {%- if use_opensearch %}
     <link rel="search" type="application/opensearchdescription+xml"
           title="{% trans docstitle=docstitle|e %}Search within {{ docstitle }}{% endtrans %}"

sphinx/themes/basic/search.html

@@ -9,7 +9,10 @@
 #}
 {%- extends "layout.html" %}
 {% set title = _('Search') %}
-{% set script_files = script_files + ['_static/searchtools.js'] %}
+{%- block scripts %}
+    {{ super() }}
+    <script type="text/javascript" src="{{ pathto('_static/searchtools.js', 1) }}"></script>
+{%- endblock %}
 {% block extrahead %}
   <script type="text/javascript">
     jQuery(function() { Search.loadIndex("{{ pathto('searchindex.js', 1) }}"); });

sphinx/themes/bizstyle/layout.html

@@ -9,7 +9,10 @@
 #}
 {% extends "basic/layout.html" %}
 
-{% set script_files = script_files + ["_static/bizstyle.js"] %}
+{%- block scripts %}
+    {{ super() }}
+    <script type="text/javascript" src="{{ pathto('_static/bizstyle.js', 1) }}"></script>
+{%- endblock %}
 
 {# put the sidebar before the body #}
 {% block sidebar1 %}{{ sidebar() }}{% endblock %}

sphinx/themes/classic/layout.html

@@ -10,5 +10,8 @@
 {%- extends "basic/layout.html" %}
 
 {% if theme_collapsiblesidebar|tobool %}
-{% set script_files = script_files + ['_static/sidebar.js'] %}
+{%- block scripts %}
+    {{ super() }}
+    <script type="text/javascript" src="{{ pathto('_static/sidebar.js', 1) }}"></script>
+{%- endblock %}
 {% endif %}

sphinx/themes/scrolls/layout.html

@@ -9,11 +9,14 @@
     :license: BSD, see LICENSE for details.
 #}
 {%- extends "basic/layout.html" %}
-{% set script_files = script_files + ['_static/theme_extras.js'] %}
 {%- block css %}
-  {{ super() }}
+    {{ super() }}
     <link rel="stylesheet" href="_static/print.css" type="text/css" />
 {%- endblock %}
+{%- block scripts %}
+    {{ super() }}
+    <script type="text/javascript" src="{{ pathto('_static/theme_extras.js', 1) }}"></script>
+{%- endblock %}
 {# do not display relbars #}
 {% block relbar1 %}{% endblock %}
 {% block relbar2 %}{% endblock %}

sphinx/transforms/post_transforms/compat.py

@@ -71,8 +71,12 @@ class MathNodeMigrator(SphinxTransform):
                     warnings.warn("math node for Sphinx was replaced by docutils'. "
                                   "Please use ``docutils.nodes.math_block`` instead.",
                                   RemovedInSphinx30Warning)
-                    latex = node['latex']
+                    if isinstance(node, displaymath):
-                    node += nodes.Text(latex, latex)
+                        newnode = nodes.math_block('', node['latex'], **node.attributes)
+                        node.replace_self(newnode)
+                    else:
+                        latex = node['latex']
+                        node += nodes.Text(latex, latex)
 
 
 def setup(app):

sphinx/util/console.py

@@ -87,9 +87,21 @@ def coloron():
     codes.update(_orig_codes)
 
 
-def colorize(name, text):
+def colorize(name, text, input_mode=False):
-    # type: (str, unicode) -> unicode
+    # type: (str, unicode, bool) -> unicode
-    return codes.get(name, '') + text + codes.get('reset', '')
+    def escseq(name):
+        # Wrap escape sequence with ``\1`` and ``\2`` to let readline know
+        # it is non-printable characters
+        # ref: https://tiswww.case.edu/php/chet/readline/readline.html
+        #
+        # Note: This hack does not work well in Windows (see #5059)
+        escape = codes.get(name, '')
+        if input_mode and escape and sys.platform != 'win32':
+            return '\1' + escape + '\2'
+        else:
+            return escape
+
+    return escseq(name) + text + escseq('reset')
 
 
 def strip_colors(s):

sphinx/util/docutils.py

@@ -10,6 +10,7 @@
 """
 from __future__ import absolute_import
 
+import codecs
 import os
 import re
 import types
@@ -21,6 +22,7 @@ from os import path
 
 import docutils
 from docutils import nodes
+from docutils.io import FileOutput
 from docutils.parsers.rst import Directive, directives, roles, convert_directive_function
 from docutils.statemachine import StateMachine
 from docutils.utils import Reporter
@@ -300,11 +302,33 @@ def switch_source_input(state, content):
         state.memo.reporter.get_source_and_line = get_source_and_line
 
 
-class SphinxDirective(Directive):
+class SphinxFileOutput(FileOutput):
-    """A base class for Directives.
+    """Better FileOutput class for Sphinx."""
 
-    Compared with ``docutils.parsers.rst.Directive``, this class improves
+    def __init__(self, **kwargs):
-    accessibility to Sphinx APIs.
+        # type: (Any) -> None
+        self.overwrite_if_changed = kwargs.pop('overwrite_if_changed', False)
+        FileOutput.__init__(self, **kwargs)
+
+    def write(self, data):
+        # type: (unicode) -> unicode
+        if (self.destination_path and self.autoclose and 'b' not in self.mode and
+                self.overwrite_if_changed and os.path.exists(self.destination_path)):
+            with codecs.open(self.destination_path, encoding=self.encoding) as f:
+                # skip writing: content not changed
+                if f.read() == data:
+                    return data
+
+        return FileOutput.write(self, data)
+
+
+class SphinxDirective(Directive):
+    """A base class for Sphinx directives.
+
+    This class provides helper methods for Sphinx directives.
+
+    .. note:: The subclasses of this class might not work with docutils.
+              This class is strongly coupled with Sphinx.
     """
 
     @property

sphinx/util/inspect.py

@@ -176,8 +176,8 @@ def isstaticmethod(obj, cls=None, name=None):
     elif cls and name:
         # trace __mro__ if the method is defined in parent class
         #
-        # .. note:: This only works with new style classes.
+        # .. note:: This only works well with new style classes.
-        for basecls in getattr(cls, '__mro__', []):
+        for basecls in getattr(cls, '__mro__', [cls]):
             meth = basecls.__dict__.get(name)
             if meth:
                 if isinstance(meth, staticmethod):

sphinx/util/logging.py

@@ -392,7 +392,7 @@ class WarningIsErrorFilter(logging.Filter):
             location = getattr(record, 'location', '')
             try:
                 message = record.msg % record.args
-            except TypeError:
+            except (TypeError, ValueError):
                 message = record.msg  # use record.msg itself
 
             if location:

sphinx/util/nodes.py

@@ -56,8 +56,7 @@ def repr_domxml(node, length=80):
        returns full of DOM XML representation.
     :return: DOM XML representation
     """
-    # text = node.asdom().toxml()  # #4919 crush if node has secnumber with tuple value
+    text = node.asdom().toxml()
-    text = text_type(node)  # workaround for #4919
     if length and len(text) > length:
         text = text[:length] + '...'
     return text
@@ -82,9 +81,8 @@ def apply_source_workaround(node):
                      get_full_module_name(node), repr_domxml(node))
         node.source, node.line = node.parent.source, node.parent.line
     if isinstance(node, nodes.title) and node.source is None:
-        # Uncomment these lines after merging into master(1.8)
+        logger.debug('[i18n] PATCH: %r to have source: %s',
-        # logger.debug('[i18n] PATCH: %r to have source: %s',
+                     get_full_module_name(node), repr_domxml(node))
-        #              get_full_module_name(node), repr_domxml(node))
         node.source, node.line = node.parent.source, node.parent.line
     if isinstance(node, nodes.term):
         logger.debug('[i18n] PATCH: %r to have rawsource: %s',

sphinx/writers/latex.py

@@ -19,13 +19,12 @@ from collections import defaultdict
 from os import path
 
 from docutils import nodes, writers
-from docutils.utils.roman import toRoman
 from docutils.writers.latex2e import Babel
 from six import itervalues, text_type
 
 from sphinx import addnodes
 from sphinx import highlighting
-from sphinx.builders.latex.nodes import footnotetext
+from sphinx.builders.latex.nodes import captioned_literal_block, footnotetext
 from sphinx.deprecation import RemovedInSphinx30Warning
 from sphinx.errors import SphinxError
 from sphinx.locale import admonitionlabels, _, __
@@ -35,6 +34,12 @@ from sphinx.util.nodes import clean_astext
 from sphinx.util.template import LaTeXRenderer
 from sphinx.util.texescape import tex_escape_map, tex_replace_map
 
+try:
+    from docutils.utils.roman import toRoman
+except ImportError:
+    # In Debain/Ubuntu, roman package is provided as roman, not as docutils.utils.roman
+    from roman import toRoman
+
 if False:
     # For type annotation
     from typing import Any, Callable, Dict, Iterator, List, Pattern, Tuple, Set, Union  # NOQA
@@ -49,6 +54,12 @@ BEGIN_DOC = r'''
 %(tableofcontents)s
 '''
 
+SHORTHANDOFF = r'''
+\ifdefined\shorthandoff
+  \ifnum\catcode`\=\string=\active\shorthandoff{=}\fi
+  \ifnum\catcode`\"=\active\shorthandoff{"}\fi
+\fi
+'''
 
 MAX_CITATION_LABEL_LENGTH = 8
 LATEXSECTIONNAMES = ["part", "chapter", "section", "subsection",
@@ -58,6 +69,7 @@ HYPERLINK_SUPPORT_NODES = (
     nodes.literal_block,
     nodes.table,
     nodes.section,
+    captioned_literal_block,
 )
 ENUMERATE_LIST_STYLE = defaultdict(lambda: r'\arabic',
                                    {
@@ -206,16 +218,6 @@ class LaTeXWriter(writers.Writer):
 
 class ExtBabel(Babel):
     cyrillic_languages = ('bulgarian', 'kazakh', 'mongolian', 'russian', 'ukrainian')
-    shorthands = {
-        'ngerman': '"',
-        'slovene': '"',
-        'portuges': '"',
-        'brazil': '"',
-        'spanish': '"',
-        'dutch': '"',
-        'polish': '"',
-        'italian': '"',
-    }
 
     def __init__(self, language_code, use_polyglossia=False):
         # type: (unicode, bool) -> None
@@ -226,13 +228,9 @@ class ExtBabel(Babel):
 
     def get_shorthandoff(self):
         # type: () -> unicode
-        shorthand = self.shorthands.get(self.language)
+        warnings.warn('ExtBabel.get_shorthandoff() is deprecated.',
-        if shorthand:
+                      RemovedInSphinx30Warning)
-            return r'\ifnum\catcode`\%s=\active\shorthandoff{%s}\fi' % (shorthand, shorthand)
+        return SHORTHANDOFF
-        elif self.language == 'turkish':
-            # memo: if ever Sphinx starts supporting 'Latin', do as for Turkish
-            return r'\ifnum\catcode`\=\string=\active\shorthandoff{=}\fi'
-        return ''
 
     def uses_cyrillic(self):
         # type: () -> bool
@@ -473,7 +471,6 @@ class LaTeXTranslator(nodes.NodeVisitor):
         self.in_production_list = 0
         self.in_footnote = 0
         self.in_caption = 0
-        self.in_container_literal_block = 0
         self.in_term = 0
         self.needs_linetrimming = 0
         self.in_minipage = 0
@@ -590,7 +587,7 @@ class LaTeXTranslator(nodes.NodeVisitor):
             # this branch is not taken for xelatex/lualatex if default settings
             self.elements['multilingual'] = self.elements['babel']
             if builder.config.language:
-                self.elements['shorthandoff'] = self.babel.get_shorthandoff()
+                self.elements['shorthandoff'] = SHORTHANDOFF
 
                 # Times fonts don't work with Cyrillic languages
                 if self.babel.uses_cyrillic() \
@@ -603,6 +600,7 @@ class LaTeXTranslator(nodes.NodeVisitor):
                     self.elements['classoptions'] = ',dvipdfmx'
                     # disable babel which has not publishing quality in Japanese
                     self.elements['babel'] = ''
+                    self.elements['shorthandoff'] = ''
                     self.elements['multilingual'] = ''
                     # disable fncychap in Japanese documents
                     self.elements['fncychap'] = ''
@@ -699,8 +697,6 @@ class LaTeXTranslator(nodes.NodeVisitor):
         self.pending_footnotes = []     # type: List[nodes.footnote_reference]
         self.curfilestack = []          # type: List[unicode]
         self.handled_abbrs = set()      # type: Set[unicode]
-        self.next_hyperlink_ids = {}    # type: Dict[unicode, Set[unicode]]
-        self.next_section_ids = set()   # type: Set[unicode]
 
     def pushbody(self, newbody):
         # type: (List[unicode]) -> None
@@ -713,15 +709,6 @@ class LaTeXTranslator(nodes.NodeVisitor):
         self.body = self.bodystack.pop()
         return body
 
-    def push_hyperlink_ids(self, figtype, ids):
-        # type: (unicode, Set[unicode]) -> None
-        hyperlink_ids = self.next_hyperlink_ids.setdefault(figtype, set())
-        hyperlink_ids.update(ids)
-
-    def pop_hyperlink_ids(self, figtype):
-        # type: (unicode) -> Set[unicode]
-        return self.next_hyperlink_ids.pop(figtype, set())
-
     def check_latex_elements(self):
         # type: () -> None
         for key in self.builder.config.latex_elements:
@@ -939,8 +926,6 @@ class LaTeXTranslator(nodes.NodeVisitor):
 
     def visit_start_of_file(self, node):
         # type: (nodes.Node) -> None
-        # also add a document target
-        self.next_section_ids.add(':doc')
         self.curfilestack.append(node['docname'])
         # use default highlight settings for new file
         self.hlsettingstack.append(self.hlsettingstack[0])
@@ -1075,11 +1060,6 @@ class LaTeXTranslator(nodes.NodeVisitor):
                     # just use "subparagraph", it's not numbered anyway
                     self.body.append(r'\%s%s{' % (self.sectionnames[-1], short))
                 self.context.append('}\n' + self.hypertarget_to(node.parent))
-
-                if self.next_section_ids:
-                    for id in self.next_section_ids:
-                        self.context[-1] += self.hypertarget(id, anchor=False)
-                    self.next_section_ids.clear()
         elif isinstance(parent, nodes.topic):
             self.body.append(r'\sphinxstyletopictitle{')
             self.context.append('}\n')
@@ -1815,7 +1795,7 @@ class LaTeXTranslator(nodes.NodeVisitor):
     def visit_caption(self, node):
         # type: (nodes.Node) -> None
         self.in_caption += 1
-        if self.in_container_literal_block:
+        if isinstance(node.parent, captioned_literal_block):
             self.body.append('\\sphinxSetupCaptionForVerbatim{')
         elif self.in_minipage and isinstance(node.parent, nodes.figure):
             self.body.append('\\captionof{figure}{')
@@ -1908,35 +1888,13 @@ class LaTeXTranslator(nodes.NodeVisitor):
             self.body.append(self.hypertarget(id, anchor=anchor))
 
         # skip if visitor for next node supports hyperlink
+        domain = self.builder.env.get_domain('std')
         next_node = node.next_node(ascend=True)
         if isinstance(next_node, HYPERLINK_SUPPORT_NODES):
             return
+        elif domain.get_enumerable_node_type(next_node) and domain.get_numfig_title(next_node):
+            return
 
-        # postpone the labels until after the sectioning command
-        parindex = node.parent.index(node)
-        try:
-            try:
-                next = node.parent[parindex + 1]
-            except IndexError:
-                # last node in parent, look at next after parent
-                # (for section of equal level) if it exists
-                if node.parent.parent is not None:
-                    next = node.parent.parent[
-                        node.parent.parent.index(node.parent)]
-                else:
-                    raise
-            domain = self.builder.env.get_domain('std')
-            figtype = domain.get_enumerable_node_type(next)
-            if figtype and domain.get_numfig_title(next):
-                ids = set()
-                # labels for figures go in the figure body, not before
-                if node.get('refid'):
-                    ids.add(node['refid'])
-                ids.update(node['ids'])
-                self.push_hyperlink_ids(figtype, ids)
-                return
-        except IndexError:
-            pass
         if 'refuri' in node:
             return
         if node.get('refid'):
@@ -1961,8 +1919,8 @@ class LaTeXTranslator(nodes.NodeVisitor):
         # type: (nodes.Node, Pattern) -> None
         def escape(value):
             value = self.encode(value)
-            value = value.replace(r'\{', r'\sphinxleftcurlybrace')
+            value = value.replace(r'\{', r'{\sphinxleftcurlybrace}')
-            value = value.replace(r'\}', r'\sphinxrightcurlybrace')
+            value = value.replace(r'\}', r'{\sphinxrightcurlybrace}')
             return value
 
         if not node.get('inline', True):
@@ -2246,6 +2204,14 @@ class LaTeXTranslator(nodes.NodeVisitor):
         # the \ignorespaces in particular for after table header use
         self.body.append('%\n\\end{footnotetext}\\ignorespaces ')
 
+    def visit_captioned_literal_block(self, node):
+        # type: (nodes.Node) -> None
+        pass
+
+    def depart_captioned_literal_block(self, node):
+        # type: (nodes.Node) -> None
+        pass
+
     def visit_literal_block(self, node):
         # type: (nodes.Node) -> None
         if node.rawsource != node.astext():
@@ -2254,9 +2220,11 @@ class LaTeXTranslator(nodes.NodeVisitor):
             self.body.append('\\begin{sphinxalltt}\n')
         else:
             labels = self.hypertarget_to(node)
-            # LaTeX code will insert \phantomsection prior to \label
+            if isinstance(node.parent, captioned_literal_block):
+                labels += self.hypertarget_to(node.parent)
             if labels and not self.in_footnote:
                 self.body.append('\n\\def\\sphinxLiteralBlockLabel{' + labels + '}')
+
             code = node.astext()
             lang = self.hlsettingstack[-1][0]
             linenos = code.count('\n') >= self.hlsettingstack[-1][1] - 1
@@ -2483,22 +2451,11 @@ class LaTeXTranslator(nodes.NodeVisitor):
 
     def visit_container(self, node):
         # type: (nodes.Node) -> None
-        if node.get('literal_block'):
+        pass
-            self.in_container_literal_block += 1
-            ids = ''  # type: unicode
-            for id in self.pop_hyperlink_ids('code-block'):
-                ids += self.hypertarget(id, anchor=False)
-            if node['ids']:
-                # suppress with anchor=False \phantomsection insertion
-                ids += self.hypertarget(node['ids'][0], anchor=False)
-            # define label for use in caption.
-            if ids:
-                self.body.append('\n\\def\\sphinxLiteralBlockLabel{' + ids + '}\n')
 
     def depart_container(self, node):
         # type: (nodes.Node) -> None
-        if node.get('literal_block'):
+        pass
-            self.in_container_literal_block -= 1
 
     def visit_decoration(self, node):
         # type: (nodes.Node) -> None
@@ -2628,6 +2585,39 @@ class LaTeXTranslator(nodes.NodeVisitor):
                       RemovedInSphinx30Warning)
         return []
 
+    @property
+    def in_container_literal_block(self):
+        # type: () -> int
+        warnings.warn('LaTeXTranslator.in_container_literal_block is deprecated.',
+                      RemovedInSphinx30Warning)
+        return 0
+
+    @property
+    def next_section_ids(self):
+        # type: () -> Set[unicode]
+        warnings.warn('LaTeXTranslator.next_section_ids is deprecated.',
+                      RemovedInSphinx30Warning)
+        return set()
+
+    @property
+    def next_hyperlink_ids(self):
+        # type: () -> Dict
+        warnings.warn('LaTeXTranslator.next_hyperlink_ids is deprecated.',
+                      RemovedInSphinx30Warning)
+        return {}
+
+    def push_hyperlink_ids(self, figtype, ids):
+        # type: (unicode, Set[unicode]) -> None
+        warnings.warn('LaTeXTranslator.push_hyperlink_ids() is deprecated.',
+                      RemovedInSphinx30Warning)
+        pass
+
+    def pop_hyperlink_ids(self, figtype):
+        # type: (unicode) -> Set[unicode]
+        warnings.warn('LaTeXTranslator.pop_hyperlink_ids() is deprecated.',
+                      RemovedInSphinx30Warning)
+        return set()
+
 
 # Import old modules here for compatibility
 # They should be imported after `LaTeXTranslator` to avoid recursive import.

tests/roots/test-domain-cpp/xref_consistency.rst

@@ -0,0 +1,12 @@
+xref consistency
+----------------
+
+.. cpp:namespace:: xref_consistency
+
+.. cpp:class:: item
+
+code-role:      :code:`item`
+any-role:       :any:`item`
+cpp-any-role:   :cpp:any:`item`
+cpp-expr-role:  :cpp:expr:`item`
+cpp-texpr-role: :cpp:texpr:`item`

tests/roots/test-epub-anchor-id/conf.py

@@ -0,0 +1,11 @@
+# -*- coding: utf-8 -*-
+
+master_doc = 'index'
+
+latex_documents = [
+    (master_doc, 'test.tex', 'The basic Sphinx documentation for testing', 'Sphinx', 'report')
+]
+
+
+def setup(app):
+    app.add_crossref_type(directivename="setting", rolename="setting")

tests/roots/test-epub-anchor-id/index.rst

@@ -0,0 +1,8 @@
+test-epub-anchor-id
+===================
+
+.. setting:: STATICFILES_FINDERS
+
+blah blah blah
+
+see :setting:`STATICFILES_FINDERS`

tests/roots/test-html_assets/conf.py

@@ -8,4 +8,6 @@ html_static_path = ['static', 'subdir']
 html_extra_path = ['extra', 'subdir']
 html_css_files = ['css/style.css',
                   ('https://example.com/custom.css', {'title': 'title', 'media': 'print'})]
+html_js_files = ['js/custom.js',
+                 ('https://example.com/script.js', {'async': 'async'})]
 exclude_patterns = ['**/_build', '**/.htpasswd']

tests/roots/test-latex-table/expects/longtable_having_widths.tex

@@ -1,7 +1,7 @@
 \label{\detokenize{longtable:longtable-having-widths-option}}
 
 \begin{savenotes}\sphinxatlongtablestart\begin{longtable}{|\X{30}{100}|\X{70}{100}|}
-\hline
+\hline\noalign{\phantomsection\label{\detokenize{longtable:namedlongtable}}\label{\detokenize{longtable:mylongtable}}}%
 \sphinxstyletheadfamily 
 header1
 &\sphinxstyletheadfamily 
@@ -43,3 +43,5 @@ cell3-2
 \\
 \hline
 \end{longtable}\sphinxatlongtableend\end{savenotes}
+
+See {\hyperref[\detokenize{longtable:mylongtable}]{\sphinxcrossref{mylongtable}}}, same as {\hyperref[\detokenize{longtable:namedlongtable}]{\sphinxcrossref{\DUrole{std,std-ref}{this one}}}}.

tests/roots/test-latex-table/expects/table_having_widths.tex

@@ -2,6 +2,7 @@
 
 \begin{savenotes}\sphinxattablestart
 \centering
+\phantomsection\label{\detokenize{tabular:namedtabular}}\label{\detokenize{tabular:mytabular}}\nobreak
 \begin{tabular}[t]{|\X{30}{100}|\X{70}{100}|}
 \hline
 \sphinxstyletheadfamily 
@@ -28,3 +29,5 @@ cell3-2
 \end{tabular}
 \par
 \sphinxattableend\end{savenotes}
+
+See {\hyperref[\detokenize{tabular:mytabular}]{\sphinxcrossref{\DUrole{std,std-ref}{this}}}}, same as {\hyperref[\detokenize{tabular:namedtabular}]{\sphinxcrossref{namedtabular}}}.

tests/roots/test-latex-table/longtable.rst

@@ -18,9 +18,12 @@ longtable
 longtable having :widths: option
 --------------------------------
 
+.. _mylongtable:
+
 .. table::
    :class: longtable
    :widths: 30,70
+   :name: namedlongtable
 
    ======= =======
    header1 header2
@@ -30,6 +33,8 @@ longtable having :widths: option
    cell3-1 cell3-2
    ======= =======
 
+See mylongtable_, same as :ref:`this one <namedlongtable>`.
+
 longtable having :align: option
 -------------------------------
 

tests/roots/test-latex-table/tabular.rst

@@ -1,4 +1,4 @@
-taburar and taburary
+tabular and tabulary
 ====================
 
 simple table
@@ -15,8 +15,11 @@ cell3-1 cell3-2
 table having :widths: option
 ----------------------------
 
+.. _mytabular:
+
 .. table::
    :widths: 30,70
+   :name: namedtabular
 
    ======= =======
    header1 header2
@@ -26,6 +29,8 @@ table having :widths: option
    cell3-1 cell3-2
    ======= =======
 
+See :ref:`this <mytabular>`, same as namedtabular_.
+
 table having :align: option (tabulary)
 --------------------------------------
 

tests/roots/test-root/conf.py

@@ -105,6 +105,6 @@ def setup(app):
     app.add_directive('clsdir', ClassDirective)
     app.add_object_type('userdesc', 'userdescrole', '%s (userdesc)',
                         userdesc_parse, objname='user desc')
-    app.add_javascript('file://moo.js')
+    app.add_js_file('file://moo.js')
     app.add_source_suffix('.foo', 'foo')
     app.add_source_parser(parsermod.Parser)

tests/test_build_epub.py

@@ -317,6 +317,15 @@ def test_epub_writing_mode(app):
     assert 'writing-mode: vertical-rl;' in css
 
 
+@pytest.mark.sphinx('epub', testroot='epub-anchor-id')
+def test_epub_anchor_id(app):
+    app.build()
+
+    html = (app.outdir / 'index.xhtml').text()
+    assert '<p id="std-setting-STATICFILES_FINDERS">blah blah blah</p>' in html
+    assert 'see <a class="reference internal" href="#std-setting-STATICFILES_FINDERS">' in html
+
+
 @pytest.mark.sphinx('epub', testroot='html_assets')
 def test_epub_assets(app):
     app.builder.build_all()

tests/test_build_html.py

@@ -227,7 +227,7 @@ def test_html_warnings(app, warning):
          "[@class='reference internal']/code/span[@class='pre']", 'HOME'),
         (".//a[@href='#with']"
          "[@class='reference internal']/code/span[@class='pre']", '^with$'),
-        (".//a[@href='#grammar-token-try_stmt']"
+        (".//a[@href='#grammar-token-try-stmt']"
          "[@class='reference internal']/code/span", '^statement$'),
         (".//a[@href='#some-label'][@class='reference internal']/span", '^here$'),
         (".//a[@href='#some-label'][@class='reference internal']/span", '^there$'),
@@ -259,7 +259,7 @@ def test_html_warnings(app, warning):
         (".//dl/dt[@id='term-boson']", 'boson'),
         # a production list
         (".//pre/strong", 'try_stmt'),
-        (".//pre/a[@href='#grammar-token-try1_stmt']/code/span", 'try1_stmt'),
+        (".//pre/a[@href='#grammar-token-try1-stmt']/code/span", 'try1_stmt'),
         # tests for ``only`` directive
         (".//p", 'A global substitution.'),
         (".//p", 'In HTML.'),
@@ -1123,6 +1123,11 @@ def test_html_assets(app):
     assert ('<link media="print" rel="stylesheet" title="title" type="text/css" '
             'href="https://example.com/custom.css" />' in content)
 
+    # html_js_files
+    assert '<script type="text/javascript" src="_static/js/custom.js"></script>' in content
+    assert ('<script async="async" type="text/javascript" src="https://example.com/script.js">'
+            '</script>' in content)
+
 
 @pytest.mark.sphinx('html', testroot='basic', confoverrides={'html_copy_source': False})
 def test_html_copy_source(app):
@@ -1240,21 +1245,50 @@ def test_html_remote_images(app, status, warning):
 
 @pytest.mark.sphinx('html', testroot='basic')
 def test_html_sidebar(app, status, warning):
+    ctx = {}
+
+    # default for alabaster
     app.builder.build_all()
     result = (app.outdir / 'index.html').text(encoding='utf8')
-    assert '<h3><a href="#">Table Of Contents</a></h3>' in result
+    assert ('<div class="sphinxsidebar" role="navigation" '
+            'aria-label="main navigation">' in result)
+    assert '<h1 class="logo"><a href="#">Python</a></h1>' in result
+    assert '<h3>Navigation</h3>' in result
     assert '<h3>Related Topics</h3>' in result
-    assert '<h3>This Page</h3>' in result
     assert '<h3>Quick search</h3>' in result
 
+    app.builder.add_sidebars('index', ctx)
+    assert ctx['sidebars'] == ['about.html', 'navigation.html', 'relations.html',
+                               'searchbox.html', 'donate.html']
+
+    # only relations.html
+    app.config.html_sidebars = {'**': ['relations.html']}
+    app.builder.build_all()
+    result = (app.outdir / 'index.html').text(encoding='utf8')
+    assert ('<div class="sphinxsidebar" role="navigation" '
+            'aria-label="main navigation">' in result)
+    assert '<h1 class="logo"><a href="#">Python</a></h1>' not in result
+    assert '<h3>Navigation</h3>' not in result
+    assert '<h3>Related Topics</h3>' in result
+    assert '<h3>Quick search</h3>' not in result
+
+    app.builder.add_sidebars('index', ctx)
+    assert ctx['sidebars'] == ['relations.html']
+
+    # no sidebars
     app.config.html_sidebars = {'**': []}
     app.builder.build_all()
     result = (app.outdir / 'index.html').text(encoding='utf8')
-    assert '<h3><a href="#">Table Of Contents</a></h3>' not in result
+    assert ('<div class="sphinxsidebar" role="navigation" '
+            'aria-label="main navigation">' not in result)
+    assert '<h1 class="logo"><a href="#">Python</a></h1>' not in result
+    assert '<h3>Navigation</h3>' not in result
     assert '<h3>Related Topics</h3>' not in result
-    assert '<h3>This Page</h3>' not in result
     assert '<h3>Quick search</h3>' not in result
 
+    app.builder.add_sidebars('index', ctx)
+    assert ctx['sidebars'] == []
+
 
 @pytest.mark.parametrize('fname,expect', flat_dict({
     'index.html': [(".//em/a[@href='https://example.com/man.1']", "", True),
@@ -1268,3 +1302,28 @@ def test_html_sidebar(app, status, warning):
 def test_html_manpage(app, cached_etree_parse, fname, expect):
     app.build()
     check_xpath(cached_etree_parse(app.outdir / fname), fname, *expect)
+
+
+@pytest.mark.sphinx('html', testroot='toctree-glob',
+                    confoverrides={'html_baseurl': 'https://example.com/'})
+def test_html_baseurl(app, status, warning):
+    app.build()
+
+    result = (app.outdir / 'index.html').text(encoding='utf8')
+    assert '<link rel="canonical" href="https://example.com/index.html" />' in result
+
+    result = (app.outdir / 'qux' / 'index.html').text(encoding='utf8')
+    assert '<link rel="canonical" href="https://example.com/qux/index.html" />' in result
+
+
+@pytest.mark.sphinx('html', testroot='toctree-glob',
+                    confoverrides={'html_baseurl': 'https://example.com/subdir',
+                                   'html_file_suffix': '.htm'})
+def test_html_baseurl_and_html_file_suffix(app, status, warning):
+    app.build()
+
+    result = (app.outdir / 'index.htm').text(encoding='utf8')
+    assert '<link rel="canonical" href="https://example.com/subdir/index.htm" />' in result
+
+    result = (app.outdir / 'qux' / 'index.htm').text(encoding='utf8')
+    assert '<link rel="canonical" href="https://example.com/subdir/qux/index.htm" />' in result

tests/test_build_html5.py

@@ -135,7 +135,7 @@ def cached_etree_parse():
          "[@class='reference internal']/code/span[@class='pre']", 'HOME'),
         (".//a[@href='#with']"
          "[@class='reference internal']/code/span[@class='pre']", '^with$'),
-        (".//a[@href='#grammar-token-try_stmt']"
+        (".//a[@href='#grammar-token-try-stmt']"
          "[@class='reference internal']/code/span", '^statement$'),
         (".//a[@href='#some-label'][@class='reference internal']/span", '^here$'),
         (".//a[@href='#some-label'][@class='reference internal']/span", '^there$'),
@@ -167,7 +167,7 @@ def cached_etree_parse():
         (".//dl/dt[@id='term-boson']", 'boson'),
         # a production list
         (".//pre/strong", 'try_stmt'),
-        (".//pre/a[@href='#grammar-token-try1_stmt']/code/span", 'try1_stmt'),
+        (".//pre/a[@href='#grammar-token-try1-stmt']/code/span", 'try1_stmt'),
         # tests for ``only`` directive
         (".//p", 'A global substitution.'),
         (".//p", 'In HTML.'),

tests/test_build_latex.py

@@ -467,7 +467,7 @@ def test_babel_with_language_ru(app, status, warning):
     assert '\\addto\\captionsrussian{\\renewcommand{\\tablename}{Table.}}\n' in result
     assert (u'\\addto\\extrasrussian{\\def\\pageautorefname'
             u'{\u0441\u0442\u0440\u0430\u043d\u0438\u0446\u0430}}\n' in result)
-    assert '\\shorthandoff' not in result
+    assert '\\shorthandoff{"}' in result
 
 
 @pytest.mark.sphinx(
@@ -529,7 +529,7 @@ def test_babel_with_unknown_language(app, status, warning):
     assert '\\addto\\captionsenglish{\\renewcommand{\\figurename}{Fig.}}\n' in result
     assert '\\addto\\captionsenglish{\\renewcommand{\\tablename}{Table.}}\n' in result
     assert '\\addto\\extrasenglish{\\def\\pageautorefname{page}}\n' in result
-    assert '\\shorthandoff' not in result
+    assert '\\shorthandoff' in result
 
     assert "WARNING: no Babel option known for language 'unknown'" in warning.getvalue()
 
@@ -1201,7 +1201,7 @@ def test_latex_index(app, status, warning):
     result = (app.outdir / 'Python.tex').text(encoding='utf8')
     assert 'A \\index{famous}famous \\index{equation}equation:\n' in result
     assert '\n\\index{Einstein}\\index{relativity}\\ignorespaces \nand' in result
-    assert '\n\\index{main \\sphinxleftcurlybrace}\\ignorespaces ' in result
+    assert '\n\\index{main {\\sphinxleftcurlybrace}}\\ignorespaces ' in result
 
 
 @pytest.mark.sphinx('latex', testroot='latex-equations')

tests/test_build_qthelp.py

@@ -88,11 +88,11 @@ def test_qthelp_namespace(app, status, warning):
     app.builder.build_all()
 
     qhp = (app.outdir / 'Python.qhp').text()
-    assert '<namespace>org.sphinxdoc.sphinx</namespace>' in qhp
+    assert '<namespace>org.sphinx-doc.sphinx</namespace>' in qhp
 
     qhcp = (app.outdir / 'Python.qhcp').text()
-    assert '<homePage>qthelp://org.sphinxdoc.sphinx/doc/index.html</homePage>' in qhcp
+    assert '<homePage>qthelp://org.sphinx-doc.sphinx/doc/index.html</homePage>' in qhcp
-    assert '<startPage>qthelp://org.sphinxdoc.sphinx/doc/index.html</startPage>' in qhcp
+    assert '<startPage>qthelp://org.sphinx-doc.sphinx/doc/index.html</startPage>' in qhcp
 
 
 @pytest.mark.sphinx('qthelp', testroot='basic')

tests/test_domain_cpp.py

@@ -214,6 +214,9 @@ def test_expressions():
     exprCheck('operator()()', 'clclE')
     exprCheck('operator()<int>()', 'clclIiEE')
 
+    # pack expansion
+    exprCheck('a(b(c, 1 + d...)..., e(f..., g))', 'cl1aspcl1b1cspplL1E1dEcl1esp1f1gEE')
+
 
 def test_type_definitions():
     check("type", "public bool b", {1: "b", 2: "1b"}, "bool b")
@@ -411,7 +414,7 @@ def test_function_definitions():
     # TODO: make tests for functions in a template, e.g., Test<int&&()>
     # such that the id generation for function type types is correct.
 
-    check('function', 'friend std::ostream &f(std::ostream&, int)',
+    check('function', 'friend std::ostream &f(std::ostream &s, int i)',
           {1: 'f__osR.i', 2: '1fRNSt7ostreamEi'})
 
     # from breathe#223
@@ -491,6 +494,10 @@ def test_class_definitions():
           {2: 'I0E7has_varI1TNSt6void_tIDTadN1T3varEEEEE'})
 
 
+def test_union_definitions():
+    check('union', 'A', {2: "1A"})
+
+
 def test_enum_definitions():
     check('enum', 'A', {2: "1A"})
     check('enum', 'A : std::underlying_type<B>::type', {2: "1A"})
@@ -502,6 +509,13 @@ def test_enum_definitions():
     check('enumerator', 'A = std::numeric_limits<unsigned long>::max()', {2: "1A"})
 
 
+def test_anon_definitions():
+    check('class', '@a', {3: "Ut1_a"})
+    check('union', '@a', {3: "Ut1_a"})
+    check('enum', '@a', {3: "Ut1_a"})
+    check('class', '@1', {3: "Ut1_1"})
+
+
 def test_templates():
     check('class', "A<T>", {2: "IE1AI1TE"}, output="template<> A<T>")
     # first just check which objects support templating
@@ -735,3 +749,68 @@ def test_build_domain_cpp_with_add_function_parentheses_is_False(app, status, wa
     t = (app.outdir / f).text()
     for s in parenPatterns:
         check(s, t, f)
+
+
+@pytest.mark.sphinx(testroot='domain-cpp')
+def test_xref_consistency(app, status, warning):
+    app.builder.build_all()
+
+    test = 'xref_consistency.html'
+    output = (app.outdir / test).text()
+
+    def classes(role, tag):
+        pattern = (r'{role}-role:.*?'
+                    '<(?P<tag>{tag}) .*?class=["\'](?P<classes>.*?)["\'].*?>'
+                    '.*'
+                    '</(?P=tag)>').format(role=role, tag=tag)
+        result = re.search(pattern, output)
+        expect = '''\
+Pattern for role `{role}` with tag `{tag}`
+\t{pattern}
+not found in `{test}`
+'''.format(role=role, tag=tag, pattern=pattern, test=test)
+        assert result, expect
+        return set(result.group('classes').split())
+
+    class RoleClasses(object):
+        """Collect the classes from the layout that was generated for a given role."""
+
+        def __init__(self, role, root, contents):
+            self.name = role
+            self.classes = classes(role, root)
+            self.content_classes = dict()
+            for tag in contents:
+                self.content_classes[tag] = classes(role, tag)
+
+    # not actually used as a reference point
+    #code_role = RoleClasses('code', 'code', [])
+    any_role = RoleClasses('any', 'a', ['code'])
+    cpp_any_role = RoleClasses('cpp-any', 'a', ['code'])
+    # NYI: consistent looks
+    #texpr_role = RoleClasses('cpp-texpr', 'span', ['a', 'code'])
+    expr_role = RoleClasses('cpp-expr', 'code', ['a'])
+    texpr_role = RoleClasses('cpp-texpr', 'span', ['a', 'span'])
+
+    # XRefRole-style classes
+
+    ## any and cpp:any do not put these classes at the root
+
+    # n.b. the generic any machinery finds the specific 'cpp-class' object type
+    expect = 'any uses XRefRole classes'
+    assert {'xref', 'any', 'cpp', 'cpp-class'} <= any_role.content_classes['code'], expect
+
+    expect = 'cpp:any uses XRefRole classes'
+    assert {'xref', 'cpp-any', 'cpp'} <= cpp_any_role.content_classes['code'], expect
+
+    for role in (expr_role, texpr_role):
+        name = role.name
+        expect = '`{name}` puts the domain and role classes at its root'.format(name=name)
+        # NYI: xref should go in the references
+        assert {'xref', 'cpp', name} <= role.classes, expect
+
+    # reference classes
+
+    expect = 'the xref roles use the same reference classes'
+    assert any_role.classes == cpp_any_role.classes, expect
+    assert any_role.classes == expr_role.content_classes['a'], expect
+    assert any_role.classes == texpr_role.content_classes['a'], expect

tests/test_environment_toctree.py

@@ -227,11 +227,11 @@ def test_get_toctree_for(app):
                  [list_item, compact_paragraph, reference, "foo.1"],
                  [list_item, compact_paragraph, reference, "foo.2"]))
 
-    assert_node(toctree[1][0][0][0], reference, refuri="foo", secnumber=(1,))
+    assert_node(toctree[1][0][0][0], reference, refuri="foo", secnumber=[1])
-    assert_node(toctree[1][0][1][0][0][0], reference, refuri="quux", secnumber=(1, 1))
+    assert_node(toctree[1][0][1][0][0][0], reference, refuri="quux", secnumber=[1, 1])
-    assert_node(toctree[1][0][1][1][0][0], reference, refuri="foo#foo-1", secnumber=(1, 2))
+    assert_node(toctree[1][0][1][1][0][0], reference, refuri="foo#foo-1", secnumber=[1, 2])
-    assert_node(toctree[1][0][1][2][0][0], reference, refuri="foo#foo-2", secnumber=(1, 3))
+    assert_node(toctree[1][0][1][2][0][0], reference, refuri="foo#foo-2", secnumber=[1, 3])
-    assert_node(toctree[1][1][0][0], reference, refuri="bar", secnumber=(2,))
+    assert_node(toctree[1][1][0][0], reference, refuri="bar", secnumber=[2])
     assert_node(toctree[1][2][0][0], reference, refuri="http://sphinx-doc.org/")
 
     assert_node(toctree[2],
@@ -258,8 +258,8 @@ def test_get_toctree_for_collapse(app):
                 ([list_item, compact_paragraph, reference, "foo"],
                  [list_item, compact_paragraph, reference, "bar"],
                  [list_item, compact_paragraph, reference, "http://sphinx-doc.org/"]))
-    assert_node(toctree[1][0][0][0], reference, refuri="foo", secnumber=(1,))
+    assert_node(toctree[1][0][0][0], reference, refuri="foo", secnumber=[1])
-    assert_node(toctree[1][1][0][0], reference, refuri="bar", secnumber=(2,))
+    assert_node(toctree[1][1][0][0], reference, refuri="bar", secnumber=[2])
     assert_node(toctree[1][2][0][0], reference, refuri="http://sphinx-doc.org/")
 
     assert_node(toctree[2],
@@ -296,13 +296,13 @@ def test_get_toctree_for_maxdepth(app):
     assert_node(toctree[1][0][1][1][1],
                 [bullet_list, list_item, compact_paragraph, reference, "foo.1-1"])
 
-    assert_node(toctree[1][0][0][0], reference, refuri="foo", secnumber=(1,))
+    assert_node(toctree[1][0][0][0], reference, refuri="foo", secnumber=[1])
-    assert_node(toctree[1][0][1][0][0][0], reference, refuri="quux", secnumber=(1, 1))
+    assert_node(toctree[1][0][1][0][0][0], reference, refuri="quux", secnumber=[1, 1])
-    assert_node(toctree[1][0][1][1][0][0], reference, refuri="foo#foo-1", secnumber=(1, 2))
+    assert_node(toctree[1][0][1][1][0][0], reference, refuri="foo#foo-1", secnumber=[1, 2])
     assert_node(toctree[1][0][1][1][1][0][0][0],
-                reference, refuri="foo#foo-1-1", secnumber=(1, 2, 1))
+                reference, refuri="foo#foo-1-1", secnumber=[1, 2, 1])
-    assert_node(toctree[1][0][1][2][0][0], reference, refuri="foo#foo-2", secnumber=(1, 3))
+    assert_node(toctree[1][0][1][2][0][0], reference, refuri="foo#foo-2", secnumber=[1, 3])
-    assert_node(toctree[1][1][0][0], reference, refuri="bar", secnumber=(2,))
+    assert_node(toctree[1][1][0][0], reference, refuri="bar", secnumber=[2])
     assert_node(toctree[1][2][0][0], reference, refuri="http://sphinx-doc.org/")
 
     assert_node(toctree[2],
@@ -335,11 +335,11 @@ def test_get_toctree_for_includehidden(app):
                  [list_item, compact_paragraph, reference, "foo.1"],
                  [list_item, compact_paragraph, reference, "foo.2"]))
 
-    assert_node(toctree[1][0][0][0], reference, refuri="foo", secnumber=(1,))
+    assert_node(toctree[1][0][0][0], reference, refuri="foo", secnumber=[1])
-    assert_node(toctree[1][0][1][0][0][0], reference, refuri="quux", secnumber=(1, 1))
+    assert_node(toctree[1][0][1][0][0][0], reference, refuri="quux", secnumber=[1, 1])
-    assert_node(toctree[1][0][1][1][0][0], reference, refuri="foo#foo-1", secnumber=(1, 2))
+    assert_node(toctree[1][0][1][1][0][0], reference, refuri="foo#foo-1", secnumber=[1, 2])
-    assert_node(toctree[1][0][1][2][0][0], reference, refuri="foo#foo-2", secnumber=(1, 3))
+    assert_node(toctree[1][0][1][2][0][0], reference, refuri="foo#foo-2", secnumber=[1, 3])
-    assert_node(toctree[1][1][0][0], reference, refuri="bar", secnumber=(2,))
+    assert_node(toctree[1][1][0][0], reference, refuri="bar", secnumber=[2])
     assert_node(toctree[1][2][0][0], reference, refuri="http://sphinx-doc.org/")
 
     assert_node(toctree[2],

tests/test_ext_apidoc.py

@@ -32,7 +32,10 @@ def apidoc(rootdir, tempdir, apidoc_params):
 
 @pytest.fixture
 def apidoc_params(request):
-    markers = request.node.get_marker("apidoc")
+    if hasattr(request.node, 'iter_markers'):  # pytest-3.6.0 or newer
+        markers = request.node.iter_markers("apidoc")
+    else:
+        markers = request.node.get_marker("apidoc")
     pargs = {}
     kwargs = {}
 

tests/test_ext_math.py

@@ -89,6 +89,18 @@ def test_imgmath_svg(app, status, warning):
     assert re.search(html, content, re.S)
 
 
+@pytest.mark.sphinx('html', testroot='ext-math',
+                    confoverrides={'extensions': ['sphinx.ext.mathjax'],
+                                   'mathjax_options': {'integrity': 'sha384-0123456789'}})
+def test_mathjax_options(app, status, warning):
+    app.builder.build_all()
+
+    content = (app.outdir / 'index.html').text()
+    assert ('<script async="async" integrity="sha384-0123456789" type="text/javascript" '
+            'src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?'
+            'config=TeX-AMS-MML_HTMLorMML"></script>' in content)
+
+
 @pytest.mark.sphinx('html', testroot='ext-math',
                     confoverrides={'extensions': ['sphinx.ext.mathjax']})
 def test_mathjax_align(app, status, warning):

tests/test_pycode_parser.py

@@ -100,11 +100,13 @@ def test_comment_picker_location():
 def test_annotated_assignment_py36():
     source = ('a: str = "Sphinx"  #: comment\n'
               'b: int = 1\n'
-              '"""string on next line"""')
+              '"""string on next line"""\n'
+              'c: int  #: comment')
     parser = Parser(source)
     parser.parse()
     assert parser.comments == {('', 'a'): 'comment',
-                               ('', 'b'): 'string on next line'}
+                               ('', 'b'): 'string on next line',
+                               ('', 'c'): 'comment'}
     assert parser.definitions == {}
 
 
@@ -313,3 +315,12 @@ def test_decorators():
                                   'func3': ('def', 7, 9),
                                   'Foo': ('class', 11, 15),
                                   'Foo.method': ('def', 13, 15)}
+
+
+def test_formfeed_char():
+    source = ('class Foo:\n'
+              '\f\n'
+              '    attr = 1234  #: comment\n')
+    parser = Parser(source)
+    parser.parse()
+    assert parser.comments == {('Foo', 'attr'): 'comment'}

tests/test_setup_command.py

@@ -27,7 +27,10 @@ def setup_command(request, tempdir, rootdir):
     Run `setup.py build_sphinx` with args and kwargs,
     pass it to the test and clean up properly.
     """
-    marker = request.node.get_marker('setup_command')
+    if hasattr(request.node, 'get_closest_marker'):  # pytest-3.6.0 or newer
+        marker = request.node.get_closest_marker('setup_command')
+    else:
+        marker = request.node.get_marker('setup_command')
     args = marker.args if marker else []
 
     pkgrootdir = tempdir / 'test-setup'

tests/test_theming.py

@@ -11,6 +11,7 @@
 
 import os
 
+import alabaster
 import pytest
 
 from sphinx.theming import ThemeError
@@ -23,11 +24,14 @@ from sphinx.theming import ThemeError
 def test_theme_api(app, status, warning):
     cfg = app.config
 
+    themes = ['basic', 'default', 'scrolls', 'agogo', 'sphinxdoc', 'haiku',
+              'traditional', 'epub', 'nature', 'pyramid', 'bizstyle', 'classic', 'nonav',
+              'test-theme', 'ziptheme', 'staticfiles', 'parent', 'child']
+    if alabaster.version.__version_info__ >= (0, 7, 11):
+        themes.append('alabaster')
+
     # test Theme class API
-    assert set(app.html_themes.keys()) == \
+    assert set(app.html_themes.keys()) == set(themes)
-        set(['basic', 'default', 'scrolls', 'agogo', 'sphinxdoc', 'haiku',
-             'traditional', 'epub', 'nature', 'pyramid', 'bizstyle', 'classic', 'nonav',
-             'test-theme', 'ziptheme', 'staticfiles', 'parent', 'child'])
     assert app.html_themes['test-theme'] == app.srcdir / 'test_theme' / 'test-theme'
     assert app.html_themes['ziptheme'] == app.srcdir / 'ziptheme.zip'
     assert app.html_themes['staticfiles'] == app.srcdir / 'test_theme' / 'staticfiles'

tests/test_toctree.py

@@ -8,6 +8,8 @@
     :copyright: Copyright 2007-2018 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
+import re
+
 import pytest
 
 
@@ -35,3 +37,12 @@ def test_singlehtml_toctree(app, status, warning):
         app.builder._get_local_toctree('index')
     except AttributeError:
         pytest.fail('Unexpected AttributeError in app.builder.fix_refuris')
+
+
+@pytest.mark.sphinx(testroot='toctree', srcdir="numbered-toctree")
+def test_numbered_toctree(app, status, warning):
+    # give argument to :numbered: option
+    index = (app.srcdir / 'index.rst').text()
+    index = re.sub(':numbered:.*', ':numbered: 1', index)
+    (app.srcdir / 'index.rst').write_text(index, encoding='utf-8')
+    app.builder.build_all()

tests/test_util_docutils.py

@@ -9,9 +9,11 @@
     :license: BSD, see LICENSE for details.
 """
 
+import os
+
 from docutils import nodes
 
-from sphinx.util.docutils import docutils_namespace, register_node
+from sphinx.util.docutils import SphinxFileOutput, docutils_namespace, register_node
 
 
 def test_register_node():
@@ -32,3 +34,31 @@ def test_register_node():
     assert not hasattr(nodes.GenericNodeVisitor, 'depart_custom_node')
     assert not hasattr(nodes.SparseNodeVisitor, 'visit_custom_node')
     assert not hasattr(nodes.SparseNodeVisitor, 'depart_custom_node')
+
+
+def test_SphinxFileOutput(tmpdir):
+    content = 'Hello Sphinx World'
+
+    # write test.txt at first
+    filename = str(tmpdir / 'test.txt')
+    output = SphinxFileOutput(destination_path=filename)
+    output.write(content)
+    os.utime(filename, (0, 0))
+
+    # overrite it again
+    output.write(content)
+    assert os.stat(filename).st_mtime != 0  # updated
+
+    # write test2.txt at first
+    filename = str(tmpdir / 'test2.txt')
+    output = SphinxFileOutput(destination_path=filename, overwrite_if_changed=True)
+    output.write(content)
+    os.utime(filename, (0, 0))
+
+    # overrite it again
+    output.write(content)
+    assert os.stat(filename).st_mtime == 0  # not updated
+
+    # overrite it again (content changed)
+    output.write(content + "; content change")
+    assert os.stat(filename).st_mtime != 0  # updated

tests/test_util_inspect.py

@@ -380,3 +380,26 @@ def test_dict_customtype():
     description = inspect.object_description(dictionary)
     # Type is unsortable, just check that it does not crash
     assert "<CustomType(2)>: 2" in description
+
+
+def test_isstaticmethod():
+    class Foo():
+        @staticmethod
+        def method1():
+            pass
+
+        def method2(self):
+            pass
+
+    class Bar(Foo):
+        pass
+
+    assert inspect.isstaticmethod(Foo.method1, Foo, 'method1') is True
+    assert inspect.isstaticmethod(Foo.method2, Foo, 'method2') is False
+
+    if sys.version_info < (3, 0):
+        assert inspect.isstaticmethod(Bar.method1, Bar, 'method1') is False
+        assert inspect.isstaticmethod(Bar.method2, Bar, 'method2') is False
+    else:
+        assert inspect.isstaticmethod(Bar.method1, Bar, 'method1') is True
+        assert inspect.isstaticmethod(Bar.method2, Bar, 'method2') is False