Merge branch '3.x' into 8446_escape_spaces_inside_desc_signatures

.github/PULL_REQUEST_TEMPLATE.md

@@ -7,7 +7,7 @@ Subject: <short purpose of this pull request>
   - Critical or severe bugs: X.Y.Z
   - Others: X.Y
 
-  For more details, see https://www.sphinx-doc.org/en/master/devguide.html#branch-model
+  For more details, see https://www.sphinx-doc.org/en/master/internals/release-process.html#branch-model
 -->
 
 ### Feature or Bugfix

.readthedocs.yml

@@ -0,0 +1,8 @@
+version: 2
+python:
+  version: 3
+  install:
+    - method: pip
+      path: .
+      extra_requirements:
+        - docs

CHANGES

@@ -1,9 +1,193 @@
-Release 3.4.0 (in development)
+Release 3.5.0 (in development)
 ==============================
 
 Dependencies
 ------------
 
+* LaTeX: ``multicol`` (it is anyhow a required part of the official latex2e
+  base distribution)
+
+Incompatible changes
+--------------------
+
+* Update Underscore.js to 1.12.0
+* #6550: html: The config variable ``html_add_permalinks`` is replaced by
+  :confval:`html_permalinks` and :confval:`html_permalinks_icon`
+
+Deprecated
+----------
+
+* pending_xref node for viewcode extension
+* ``sphinx.builders.linkcheck.CheckExternalLinksBuilder.broken``
+* ``sphinx.builders.linkcheck.CheckExternalLinksBuilder.good``
+* ``sphinx.builders.linkcheck.CheckExternalLinksBuilder.redirected``
+* ``sphinx.builders.linkcheck.node_line_or_0()``
+* ``sphinx.ext.autodoc.AttributeDocumenter.isinstanceattribute()``
+* ``sphinx.ext.autodoc.directive.DocumenterBridge.reporter``
+* ``sphinx.ext.autodoc.importer.get_module_members()``
+* ``sphinx.ext.autosummary.generate._simple_info()``
+* ``sphinx.ext.autosummary.generate._simple_warn()``
+* ``sphinx.writers.html.HTMLTranslator.permalink_text``
+* ``sphinx.writers.html5.HTML5Translator.permalink_text``
+
+Features added
+--------------
+
+* #8022: autodoc: autodata and autoattribute directives does not show right-hand
+  value of the variable if docstring contains ``:meta hide-value:`` in
+  info-field-list
+* #8514: autodoc: Default values of overloaded functions are taken from actual
+  implementation if they're ellipsis
+* #8619: html: kbd role generates customizable HTML tags for compound keys
+* #8634: html: Allow to change the order of JS/CSS via ``priority`` parameter
+  for :meth:`Sphinx.add_js_file()` and :meth:`Sphinx.add_css_file()`
+* #6241: html: Allow to add JS/CSS files to the specific page when an extension
+  calls ``app.add_js_file()`` or ``app.add_css_file()`` on
+  :event:`html-page-context` event
+* #6550: html: Allow to use HTML permalink texts via
+  :confval:`html_permalinks_icon`
+* #1638: html: Add permalink icons to glossary terms
+* #8649: imgconverter: Skip availability check if builder supports the image
+  type
+* #8573: napoleon: Allow to change the style of custom sections using
+  :confval:`napoleon_custom_styles`
+* #8004: napoleon: Type definitions in Google style docstrings are rendered as
+  references when :confval:`napoleon_preprocess_types` enabled
+* #6241: mathjax: Include mathjax.js only on the document using equations
+* #8651: std domain: cross-reference for a rubric having inline item is broken
+* #7642: std domain: Optimize case-insensitive match of term
+* #8681: viewcode: Support incremental build
+* #8132: Add :confval:`project_copyright` as an alias of :confval:`copyright`
+* #207: Now :confval:`highlight_language` supports multiple languages
+* #2030: :rst:dir:`code-block` and :rst:dir:`literalinclude` supports automatic
+  dedent via no-argument ``:dedent:`` option
+* C++, also hyperlink operator overloads in expressions and alias declarations.
+* #8247: Allow production lists to refer to tokens from other production groups
+
+Bugs fixed
+----------
+
+* #8727: apidoc: namespace module file is not generated if no submodules there
+* #741: autodoc: inherited-members doesn't work for instance attributes on super
+  class
+* #8592: autodoc: ``:meta public:`` does not effect to variables
+* #8594: autodoc: empty __all__ attribute is ignored
+* #8315: autodoc: Failed to resolve struct.Struct type annotation
+* #8652: autodoc: All variable comments in the module are ignored if the module
+  contains invalid type comments
+* #8693: autodoc: Default values for overloaded functions are rendered as string
+* #8134: autodoc: crashes when mocked decorator takes arguments
+* #8306: autosummary: mocked modules are documented as empty page when using
+  :recursive: option
+* #8232: graphviz: Image node is not rendered if graph file is in subdirectory
+* #8618: html: kbd role produces incorrect HTML when compound-key separators (-,
+  + or ^) are used as keystrokes
+* #8629: html: A type warning for html_use_opensearch is shown twice
+* #8714: html: kbd role with "Caps Lock" rendered incorrectly
+* #8123: html search: fix searching for terms containing + (Requires a custom
+  search language that does not split on +)
+* #8665: html theme: Could not override globaltoc_maxdepth in theme.conf
+* #8446: html: consecutive spaces are displayed as single space
+* #8745: i18n: crashes with KeyError when translation message adds a new auto
+  footnote reference
+* #4304: linkcheck: Fix race condition that could lead to checking the
+  availability of the same URL twice
+* #7118: sphinx-quickstart: questionare got Mojibake if libreadline unavailable
+* #8094: texinfo: image files on the different directory with document are not
+  copied
+* #8720: viewcode: module pages are generated for epub on incremental build
+* #8704: viewcode: anchors are generated in incremental build after singlehtml
+* #8756: viewcode: highlighted code is generated even if not referenced
+* #8671: :confval:`highlight_options` is not working
+* #8341: C, fix intersphinx lookup types for names in declarations.
+* C, C++: in general fix intersphinx and role lookup types.
+* #8683: :confval:`html_last_updated_fmt` does not support UTC offset (%z)
+* #8683: :confval:`html_last_updated_fmt` generates wrong time zone for %Z
+* #1112: ``download`` role creates duplicated copies when relative path is
+  specified
+* #7576: LaTeX with French babel and memoir crash: "Illegal parameter number
+  in definition of ``\FNH@prefntext``"
+* #8072: LaTeX: Directive :rst:dir:`hlist` not implemented in LaTeX
+* #8214: LaTeX: The :rst:role:`index` role and the glossary generate duplicate
+  entries in the LaTeX index (if both used for same term)
+* #8735: LaTeX: wrong internal links in pdf to captioned code-blocks when
+  :confval:`numfig` is not True
+* #8442: LaTeX: some indexed terms are ignored when using xelatex engine
+  (or pdflatex and :confval:`latex_use_xindy` set to True) with memoir class
+* #8780: LaTeX: long words in narrow columns may not be hyphenated
+* #8788: LaTeX: ``\titleformat`` last argument in sphinx.sty should be
+  bracketed, not braced (and is anyhow not needed) 
+
+Testing
+--------
+
+Release 3.4.4 (in development)
+==============================
+
+Dependencies
+------------
+
+Incompatible changes
+--------------------
+
+Deprecated
+----------
+
+Features added
+--------------
+
+Bugs fixed
+----------
+
+* #8655: autodoc: Failed to generate document if target module contains an
+  object that raises an exception on ``hasattr()``
+* C, ``expr`` role should start symbol lookup in the current scope.
+* #8796: LaTeX: potentially critical low level TeX coding mistake has gone
+  unnoticed so far
+
+Testing
+--------
+
+Release 3.4.3 (released Jan 08, 2021)
+=====================================
+
+Bugs fixed
+----------
+
+* #8655: autodoc: Failed to generate document if target module contains an
+  object that raises an exception on ``hasattr()``
+
+Release 3.4.2 (released Jan 04, 2021)
+=====================================
+
+Bugs fixed
+----------
+
+* #8164: autodoc: Classes that inherit mocked class are not documented
+* #8602: autodoc: The ``autodoc-process-docstring`` event is emitted to the
+  non-datadescriptors unexpectedly
+* #8616: autodoc: AttributeError is raised on non-class object is passed to
+  autoclass directive
+
+Release 3.4.1 (released Dec 25, 2020)
+=====================================
+
+Bugs fixed
+----------
+
+* #8559: autodoc: AttributeError is raised when using forward-reference type
+  annotations
+* #8568: autodoc: TypeError is raised on checking slots attribute
+* #8567: autodoc: Instance attributes are incorrectly added to Parent class
+* #8566: autodoc: The ``autodoc-process-docstring`` event is emitted to the
+  alias classes unexpectedly
+* #8583: autodoc: Unnecessary object comparision via ``__eq__`` method
+* #8565: linkcheck: Fix PriorityQueue crash when link tuples are not
+  comparable
+
+Release 3.4.0 (released Dec 20, 2020)
+=====================================
+
 Incompatible changes
 --------------------
 
@@ -14,8 +198,18 @@ Deprecated
 ----------
 
 * The ``follow_wrapped`` argument of ``sphinx.util.inspect.signature()``
+* The ``no_docstring`` argument of
+  ``sphinx.ext.autodoc.Documenter.add_content()``
+* ``sphinx.ext.autodoc.Documenter.get_object_members()``
 * ``sphinx.ext.autodoc.DataDeclarationDocumenter``
+* ``sphinx.ext.autodoc.GenericAliasDocumenter``
+* ``sphinx.ext.autodoc.InstanceAttributeDocumenter``
+* ``sphinx.ext.autodoc.SlotsAttributeDocumenter``
+* ``sphinx.ext.autodoc.TypeVarDocumenter``
+* ``sphinx.ext.autodoc.importer._getannotations()``
+* ``sphinx.ext.autodoc.importer._getmro()``
 * ``sphinx.pycode.ModuleAnalyzer.parse()``
+* ``sphinx.util.osutil.movefile()``
 * ``sphinx.util.requests.is_ssl_error()``
 
 Features added
@@ -32,9 +226,14 @@ Features added
 * #8209: autodoc: Add ``:no-value:`` option to :rst:dir:`autoattribute` and
   :rst:dir:`autodata` directive to suppress the default value of the variable
 * #8460: autodoc: Support custom types defined by typing.NewType
+* #8285: napoleon: Add :confval:`napoleon_attr_annotations` to merge type hints
+  on source code automatically if any type is specified in docstring
+* #8236: napoleon: Support numpydoc's "Receives" section
 * #6914: Add a new event :event:`warn-missing-reference` to custom warning
   messages when failed to resolve a cross-reference
 * #6914: Emit a detailed warning when failed to resolve a ``:ref:`` reference
+* #6629: linkcheck: The builder now handles rate limits. See
+  :confval:`linkcheck_retry_on_rate_limit` for details.
 
 Bugs fixed
 ----------
@@ -49,36 +248,40 @@ Bugs fixed
   type annotated variables
 * #8443: autodoc: autoattribute directive can't create document for PEP-526
   based uninitalized variables
+* #8480: autodoc: autoattribute could not create document for __slots__
+  attributes
+* #8503: autodoc: autoattribute could not create document for a GenericAlias as
+  class attributes correctly
+* #8534: autodoc: autoattribute could not create document for a commented
+  attribute in alias class
 * #8452: autodoc: autodoc_type_aliases doesn't work when autodoc_typehints is
   set to "description"
-* #8446: html: consecutive spaces are displayed as single space
+* #8541: autodoc: autodoc_type_aliases doesn't work for the type annotation to
+  instance attributes
+* #8460: autodoc: autodata and autoattribute directives do not display type
+  information of TypeVars
+* #8493: autodoc: references to builtins not working in class aliases
+* #8522: autodoc:  ``__bool__`` method could be called
+* #8067: autodoc: A typehint for the instance variable having type_comment on
+  super class is not displayed
+* #8545: autodoc: a __slots__ attribute is not documented even having docstring
+* #741: autodoc: inherited-members doesn't work for instance attributes on super
+  class
+* #8477: autosummary: non utf-8 reST files are generated when template contains
+  multibyte characters
+* #8501: autosummary: summary extraction splits text after "el at." unexpectedly
+* #8524: html: Wrong url_root has been generated on a document named "index"
 * #8419: html search: Do not load ``language_data.js`` in non-search pages
+* #8549: i18n: ``-D gettext_compact=0`` is no longer working
 * #8454: graphviz: The layout option for graph and digraph directives don't work
+* #8131: linkcheck: Use GET when HEAD requests cause Too Many Redirects, to
+  accommodate infinite redirect loops on HEAD
 * #8437: Makefile: ``make clean`` with empty BUILDDIR is dangerous
-
+* #8365: py domain: ``:type:`` and ``:rtype:`` gives false ambiguous class
-Testing
+  lookup warnings
---------
+* #8352: std domain: Failed to parse an option that starts with bracket
-
+* #8519: LaTeX: Prevent page brake in the middle of a seealso
-Release 3.3.2 (in development)
+* #8520: C, fix copying of AliasNode.
-==============================
-
-Dependencies
-------------
-
-Incompatible changes
---------------------
-
-Deprecated
-----------
-
-Features added
---------------
-
-Bugs fixed
-----------
-
-Testing
---------
 
 Release 3.3.1 (released Nov 12, 2020)
 =====================================

CONTRIBUTING.rst

@@ -14,4 +14,5 @@ You can also browse it from this repository from
 ``doc/internals/contributing.rst``
 
 Sphinx uses GitHub to host source code, track patches and bugs, and more.
-Please make an effort to provide as much possible when filing bugs.
+Please make an effort to provide as much detail as possible when filing
+bugs.

EXAMPLES

@@ -318,6 +318,7 @@ Documentation using a custom theme or integrated in a website
 * `Django <https://docs.djangoproject.com/>`__
 * `Doctrine <https://www.doctrine-project.org/>`__
 * `Enterprise Toolkit for Acrobat products <https://www.adobe.com/devnet-docs/acrobatetk/>`__
+* `FreeFEM <https://doc.freefem.org/introduction/>`__
 * `Gameduino <http://excamera.com/sphinx/gameduino/>`__
 * `gensim <https://radimrehurek.com/gensim/>`__
 * `GeoServer <http://docs.geoserver.org/>`__

LICENSE

@@ -1,7 +1,7 @@
 License for Sphinx
 ==================
 
-Copyright (c) 2007-2020 by the Sphinx team (see AUTHORS file).
+Copyright (c) 2007-2021 by the Sphinx team (see AUTHORS file).
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without

doc/_templates/index.html

@@ -33,9 +33,9 @@
     <li>{%trans path=pathto('ext/builtins')%}<b>Extensions:</b> automatic testing of code snippets, inclusion of
       docstrings from Python modules (API docs), and
       <a href="{{ path }}#builtin-sphinx-extensions">more</a>{%endtrans%}</li>
-    <li>{%trans path=pathto('develop')%}<b>Contributed extensions:</b> more than
+    <li>{%trans path=pathto("usage/extensions")%}<b>Contributed extensions:</b> dozens of
-      50 extensions <a href="{{ path }}#extensions">contributed by users</a>
+      extensions <a href="{{ path }}#third-party-extensions">contributed by users</a>;
-      in a second repository; most of them installable from PyPI{%endtrans%}</li>
+      most of them installable from PyPI{%endtrans%}</li>
   </ul>
   <p>{%trans%}
     Sphinx uses <a href="http://docutils.sourceforge.net/rst.html">reStructuredText</a>

doc/_themes/sphinx13/layout.html

@@ -67,7 +67,7 @@
     <li><a href="{{ pathto('index') }}">Home</a></li>
     <li><a href="{{ pathto('usage/installation') }}">Get it</a></li>
     <li><a href="{{ pathto('contents') }}">Docs</a></li>
-    <li><a href="{{ pathto('develop') }}">Extend/Develop</a></li>
+    <li><a href="{{ pathto('development') }}">Extend</a></li>
   </ul>
   <div>
     <a href="{{ pathto('index') }}">

doc/changes.rst

@@ -8,4 +8,9 @@
 Changelog
 =========
 
+.. raw:: latex
+
+   \hypersetup{bookmarksdepth=1}% pdf bookmarks
+   \addtocontents{toc}{\protect\setcounter{tocdepth}{1}}%
+
 .. include:: ../CHANGES

doc/conf.py

@@ -14,7 +14,7 @@ templates_path = ['_templates']
 exclude_patterns = ['_build']
 
 project = 'Sphinx'
-copyright = '2007-2020, Georg Brandl and the Sphinx team'
+copyright = '2007-2021, Georg Brandl and the Sphinx team'
 version = sphinx.__display_version__
 release = version
 show_authors = True
@@ -69,8 +69,15 @@ latex_elements = {
 \substitutefont{X2}{\sfdefault}{cmss}
 \substitutefont{X2}{\ttdefault}{cmtt}
 ''',
-    'passoptionstopackages': '\\PassOptionsToPackage{svgnames}{xcolor}',
+    'passoptionstopackages': r'''
-    'preamble': '\\DeclareUnicodeCharacter{229E}{\\ensuremath{\\boxplus}}',
+\PassOptionsToPackage{svgnames}{xcolor}
+\PassOptionsToPackage{bookmarksdepth=3}{hyperref}% depth of pdf bookmarks
+''',
+    'preamble': r'''
+\DeclareUnicodeCharacter{229E}{\ensuremath{\boxplus}}
+\setcounter{tocdepth}{3}%    depth of what is kept from toc file
+\setcounter{secnumdepth}{1}% depth of section numbering
+''',
     'fvset': '\\fvset{fontsize=auto}',
     # fix missing index entry due to RTD doing only once pdflatex after makeindex
     'printindex': r'''
@@ -110,7 +117,10 @@ texinfo_documents = [
      1),
 ]
 
-intersphinx_mapping = {'python': ('https://docs.python.org/3/', None)}
+intersphinx_mapping = {
+    'python': ('https://docs.python.org/3/', None),
+    'requests': ('https://requests.readthedocs.io/en/master', None),
+}
 
 # Sphinx document translation with sphinx gettext feature uses these settings:
 locale_dirs = ['locale/']

doc/develop.rst

@@ -1,153 +0,0 @@
-:orphan:
-
-Sphinx development
-==================
-
-Sphinx is a maintained by a group of volunteers.  We value every contribution!
-
-* The code can be found in a Git repository, at
-  https://github.com/sphinx-doc/sphinx/.
-* Issues and feature requests should be raised in the `tracker
-  <https://github.com/sphinx-doc/sphinx/issues>`_.
-* The mailing list for development is at `Google Groups
-  <https://groups.google.com/forum/#!forum/sphinx-dev>`_.
-* There is also the #sphinx-doc IRC channel on `freenode
-  <https://freenode.net/>`_.
-
-For more about our development process and methods, refer to
-:doc:`/internals/index`.
-
-Extensions
-==========
-
-To learn how to write your own extension, see :ref:`dev-extensions`.
-
-The `sphinx-contrib <https://github.com/sphinx-contrib>`_ repository contains many
-contributed extensions.  Some of them have their own releases on PyPI, others you
-can install from a checkout.
-
-This is the current list of contributed extensions in that repository:
-
-- aafig: render embedded ASCII art as nice images using aafigure_
-- actdiag: embed activity diagrams by using actdiag_
-- adadomain: an extension for Ada support (Sphinx 1.0 needed)
-- ansi: parse ANSI color sequences inside documents
-- argdoc: automatically generate documentation for command-line arguments,
-  descriptions and help text
-- astah: embed diagram by using astah
-- autoanysrc: Gather reST documentation from any source files
-- autorun: Execute code in a ``runblock`` directive
-- beamer_: A builder for Beamer (LaTeX) output.
-- blockdiag: embed block diagrams by using blockdiag_
-- cacoo: embed diagram from Cacoo
-- cf3domain: a domain for CFEngine 3 policies
-- cheader: The missing c:header directive for Sphinx's built-in C domain
-- cheeseshop: easily link to PyPI packages
-- clearquest: create tables from ClearQuest_ queries
-- cmakedomain_: a domain for CMake_
-- coffeedomain: a domain for (auto)documenting CoffeeScript source code
-- context: a builder for ConTeXt
-- disqus: embed Disqus comments in documents
-- documentedlist: converts a Python list to a table in the generated
-  documentation
-- doxylink: Link to external Doxygen-generated HTML documentation
-- domaintools_: A tool for easy domain creation
-- email: obfuscate email addresses
-- erlangdomain: an extension for Erlang support (Sphinx 1.0 needed)
-- exceltable: embed Excel spreadsheets into documents using exceltable_
-- feed: an extension for creating syndication feeds and time-based overviews
-  from your site content
-- findanything_: an extension to add Sublime Text 2-like findanything panels
-  to your documentation to find pages, sections and index entries while typing
-- gnuplot: produces images using gnuplot_ language
-- googleanalytics: track web visitor statistics by using `Google Analytics`_
-- googlechart: embed charts by using `Google Chart`_
-- googlemaps: embed maps by using `Google Maps`_
-- httpdomain: a domain for documenting RESTful HTTP APIs
-- hyphenator: client-side hyphenation of HTML using hyphenator_
-- imgur: embed Imgur images, albums, and metadata in documents
-- inlinesyntaxhighlight_: inline syntax highlighting
-- lassodomain: a domain for documenting Lasso_ source code
-- libreoffice: an extension to include any drawing supported by LibreOffice
-  (e.g. odg, vsd, ...)
-- lilypond: an extension inserting music scripts from Lilypond_ in PNG format
-- makedomain_: a domain for `GNU Make`_
-- matlabdomain: document MATLAB_ code
-- mockautodoc: mock imports
-- mscgen: embed mscgen-formatted MSC (Message Sequence Chart)s
-- napoleon: supports `Google style`_ and `NumPy style`_ docstrings
-- nicovideo: embed videos from nicovideo
-- nwdiag: embed network diagrams by using nwdiag_
-- omegat: support tools to collaborate with OmegaT_ (Sphinx 1.1 needed)
-- osaka: convert standard Japanese doc to Osaka dialect (this is a joke
-  extension)
-- paverutils: an alternate integration of Sphinx with Paver_
-- phpdomain: an extension for PHP support
-- plantuml: embed UML diagram by using PlantUML_
-- py_directive: Execute python code in a ``py`` directive and return a math
-  node
-- rawfiles: copy raw files, like a CNAME
-- requirements: declare requirements wherever you need (e.g. in test
-  docstrings), mark statuses and collect them in a single list
-- restbuilder: a builder for reST (reStructuredText) files
-- rubydomain: an extension for Ruby support (Sphinx 1.0 needed)
-- sadisplay: display SqlAlchemy model sadisplay_
-- sdedit: an extension inserting sequence diagram by using Quick Sequence
-  Diagram Editor (sdedit_)
-- seqdiag: embed sequence diagrams by using seqdiag_
-- slide: embed presentation slides on slideshare_ and other sites
-- swf_: embed flash files
-- sword: an extension inserting Bible verses from Sword_
-- tikz: draw pictures with the `TikZ/PGF LaTeX package`_
-- traclinks: create TracLinks_ to a Trac_ instance from within Sphinx
-- versioning: Sphinx extension that allows building versioned docs for
-  self-hosting
-- whooshindex: whoosh indexer extension
-- youtube: embed videos from YouTube_
-- zopeext: provide an ``autointerface`` directive for using `Zope interfaces`_
-
-
-See the :doc:`extension tutorials <../development/tutorials/index>` on getting
-started with writing your own extensions.
-
-
-.. _aafigure: https://launchpad.net/aafigure
-.. _gnuplot: http://www.gnuplot.info/
-.. _paver: https://paver.readthedocs.io/en/latest/
-.. _Sword: https://www.crosswire.org/sword/
-.. _Lilypond: http://lilypond.org/
-.. _sdedit: http://sdedit.sourceforge.net/
-.. _Trac: https://trac.edgewall.org/
-.. _TracLinks: https://trac.edgewall.org/wiki/TracLinks
-.. _OmegaT: https://omegat.org/
-.. _PlantUML: http://plantuml.com/
-.. _PyEnchant: https://pythonhosted.org/pyenchant/
-.. _sadisplay: https://bitbucket.org/estin/sadisplay/wiki/Home
-.. _blockdiag: http://blockdiag.com/en/
-.. _seqdiag: http://blockdiag.com/en/
-.. _actdiag: http://blockdiag.com/en/
-.. _nwdiag: http://blockdiag.com/en/
-.. _Google Analytics: https://www.google.com/analytics/
-.. _Google Chart: https://developers.google.com/chart/
-.. _Google Maps: https://www.google.com/maps
-.. _Google style: https://google.github.io/styleguide/pyguide.html
-.. _NumPy style: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt
-.. _hyphenator: https://github.com/mnater/hyphenator
-.. _exceltable: https://pythonhosted.org/sphinxcontrib-exceltable/
-.. _YouTube: https://www.youtube.com/
-.. _ClearQuest: https://www.ibm.com/us-en/marketplace/rational-clearquest
-.. _Zope interfaces: https://zopeinterface.readthedocs.io/en/latest/README.html
-.. _slideshare: https://www.slideshare.net/
-.. _TikZ/PGF LaTeX package: https://sourceforge.net/projects/pgf/
-.. _MATLAB: https://www.mathworks.com/products/matlab.html
-.. _swf: https://github.com/sphinx-contrib/swf
-.. _findanything: https://github.com/sphinx-contrib/findanything
-.. _cmakedomain: https://github.com/sphinx-contrib/cmakedomain
-.. _GNU Make: https://www.gnu.org/software/make/
-.. _makedomain: https://github.com/sphinx-contrib/makedomain
-.. _inlinesyntaxhighlight: https://sphinxcontrib-inlinesyntaxhighlight.readthedocs.io/
-.. _CMake: https://cmake.org
-.. _domaintools: https://github.com/sphinx-contrib/domaintools
-.. _restbuilder: https://pypi.org/project/sphinxcontrib-restbuilder/
-.. _Lasso: http://www.lassosoft.com/
-.. _beamer: https://pypi.org/project/sphinxcontrib-beamer/

doc/development/index.rst

@@ -2,10 +2,12 @@
 Extending Sphinx
 ================
 
-This guide is aimed at those wishing to develop their own extensions for
+This guide is aimed at giving a quick introduction for those wishing to 
-Sphinx. Sphinx possesses significant extensibility capabilities including the
+develop their own extensions for Sphinx. Sphinx possesses significant 
-ability to hook into almost every point of the build process.  If you simply
+extensibility capabilities including the ability to hook into almost every
-wish to use Sphinx with existing extensions, refer to :doc:`/usage/index`.
+point of the build process.  If you simply wish to use Sphinx with existing 
+extensions, refer to :doc:`/usage/index`. For a more detailed discussion of
+the extension interface see :doc:`/extdev/index`.
 
 .. toctree::
    :maxdepth: 2

doc/extdev/appapi.rst

@@ -25,75 +25,75 @@ package.
 
 .. currentmodule:: sphinx.application
 
-.. automethod:: Sphinx.setup_extension(name)
+.. automethod:: Sphinx.setup_extension
 
-.. automethod:: Sphinx.require_sphinx(version)
+.. automethod:: Sphinx.require_sphinx
 
-.. automethod:: Sphinx.connect(event, callback)
+.. automethod:: Sphinx.connect
 
-.. automethod:: Sphinx.disconnect(listener_id)
+.. automethod:: Sphinx.disconnect
 
-.. automethod:: Sphinx.add_builder(builder)
+.. automethod:: Sphinx.add_builder
 
-.. automethod:: Sphinx.add_config_value(name, default, rebuild)
+.. automethod:: Sphinx.add_config_value
 
-.. automethod:: Sphinx.add_event(name)
+.. automethod:: Sphinx.add_event
 
-.. automethod:: Sphinx.set_translator(name, translator_class)
+.. automethod:: Sphinx.set_translator
 
-.. automethod:: Sphinx.add_node(node, \*\*kwds)
+.. automethod:: Sphinx.add_node
 
-.. automethod:: Sphinx.add_enumerable_node(node, figtype, title_getter=None, \*\*kwds)
+.. automethod:: Sphinx.add_enumerable_node
 
-.. automethod:: Sphinx.add_directive(name, directiveclass)
+.. automethod:: Sphinx.add_directive
 
-.. automethod:: Sphinx.add_role(name, role)
+.. automethod:: Sphinx.add_role
 
-.. automethod:: Sphinx.add_generic_role(name, nodeclass)
+.. automethod:: Sphinx.add_generic_role
 
-.. automethod:: Sphinx.add_domain(domain)
+.. automethod:: Sphinx.add_domain
 
-.. automethod:: Sphinx.add_directive_to_domain(domain, name, directiveclass)
+.. automethod:: Sphinx.add_directive_to_domain
 
-.. automethod:: Sphinx.add_role_to_domain(domain, name, role)
+.. automethod:: Sphinx.add_role_to_domain
 
-.. automethod:: Sphinx.add_index_to_domain(domain, index)
+.. automethod:: Sphinx.add_index_to_domain
 
-.. automethod:: Sphinx.add_object_type(directivename, rolename, indextemplate='', parse_node=None, ref_nodeclass=None, objname='', doc_field_types=[])
+.. automethod:: Sphinx.add_object_type
 
-.. automethod:: Sphinx.add_crossref_type(directivename, rolename, indextemplate='', ref_nodeclass=None, objname='')
+.. automethod:: Sphinx.add_crossref_type
 
-.. automethod:: Sphinx.add_transform(transform)
+.. automethod:: Sphinx.add_transform
 
-.. automethod:: Sphinx.add_post_transform(transform)
+.. automethod:: Sphinx.add_post_transform
 
-.. automethod:: Sphinx.add_js_file(filename, **kwargs)
+.. automethod:: Sphinx.add_js_file
 
-.. automethod:: Sphinx.add_css_file(filename, **kwargs)
+.. automethod:: Sphinx.add_css_file
 
-.. automethod:: Sphinx.add_latex_package(packagename, options=None)
+.. automethod:: Sphinx.add_latex_package
 
-.. automethod:: Sphinx.add_lexer(alias, lexer)
+.. automethod:: Sphinx.add_lexer
 
-.. automethod:: Sphinx.add_autodocumenter(cls)
+.. automethod:: Sphinx.add_autodocumenter
 
-.. automethod:: Sphinx.add_autodoc_attrgetter(type, getter)
+.. automethod:: Sphinx.add_autodoc_attrgetter
 
-.. automethod:: Sphinx.add_search_language(cls)
+.. automethod:: Sphinx.add_search_language
 
-.. automethod:: Sphinx.add_source_suffix(suffix, filetype)
+.. automethod:: Sphinx.add_source_suffix
 
-.. automethod:: Sphinx.add_source_parser(parser)
+.. automethod:: Sphinx.add_source_parser
 
-.. automethod:: Sphinx.add_env_collector(collector)
+.. automethod:: Sphinx.add_env_collector
 
-.. automethod:: Sphinx.add_html_theme(name, theme_path)
+.. automethod:: Sphinx.add_html_theme
 
-.. automethod:: Sphinx.add_html_math_renderer(name, inline_renderers, block_renderers)
+.. automethod:: Sphinx.add_html_math_renderer
 
-.. automethod:: Sphinx.add_message_catalog(catalog, locale_dir)
+.. automethod:: Sphinx.add_message_catalog
 
-.. automethod:: Sphinx.is_parallel_allowed(typ)
+.. automethod:: Sphinx.is_parallel_allowed
 
 .. exception:: ExtensionError
 
@@ -107,9 +107,9 @@ Emitting events
 .. class:: Sphinx
    :noindex:
 
-   .. automethod:: emit(event, \*arguments)
+   .. automethod:: emit
 
-   .. automethod:: emit_firstresult(event, \*arguments)
+   .. automethod:: emit_firstresult
 
 
 Sphinx runtime information
@@ -369,6 +369,9 @@ Here is a more detailed list of these events.
    You can return a string from the handler, it will then replace
    ``'page.html'`` as the HTML template for this page.
 
+   .. note:: You can install JS/CSS files for the specific page via
+             :meth:`Sphinx.add_js_file` and :meth:`Sphinx.add_css_file` since v3.5.0.
+
    .. versionadded:: 0.4
 
    .. versionchanged:: 1.3

doc/extdev/deprecated.rst

@@ -12,35 +12,137 @@ The following is a list of deprecated interfaces.
 
 .. tabularcolumns:: |>{\raggedright}\Y{.4}|>{\centering}\Y{.1}|>{\centering}\Y{.12}|>{\raggedright\arraybackslash}\Y{.38}|
 
-.. |LaTeXHyphenate| raw:: latex
-
-                    \hspace{0pt}
-
 .. list-table:: deprecated APIs
    :header-rows: 1
    :class: deprecated
    :widths: 40, 10, 10, 40
 
    * - Target
-     - |LaTeXHyphenate|\ Deprecated
+     - Deprecated
      - (will be) Removed
      - Alternatives
 
+   * - pending_xref node for viewcode extension
+     - 3.5
+     - 5.0
+     - ``sphinx.ext.viewcode.viewcode_anchor``
+
+   * - ``sphinx.builders.linkcheck.CheckExternalLinksBuilder.broken``
+     - 3.5
+     - 5.0
+     - N/A
+
+   * - ``sphinx.builders.linkcheck.CheckExternalLinksBuilder.good``
+     - 3.5
+     - 5.0
+     - N/A
+
+   * - ``sphinx.builders.linkcheck.CheckExternalLinksBuilder.redirected``
+     - 3.5
+     - 5.0
+     - N/A
+
+   * - ``sphinx.builders.linkcheck.node_line_or_0()``
+     - 3.5
+     - 5.0
+     - ``sphinx.util.nodes.get_node_line()``
+
+   * - ``sphinx.ext.autodoc.AttributeDocumenter.isinstanceattribute()``
+     - 3.5
+     - 5.0
+     - N/A
+
+   * - ``sphinx.ext.autodoc.importer.get_module_members()``
+     - 3.5
+     - 5.0
+     - ``sphinx.ext.autodoc.ModuleDocumenter.get_module_members()``
+
+   * - ``sphinx.ext.autosummary.generate._simple_info()``
+     - 3.5
+     - 5.0
+     - :ref:`logging-api`
+
+   * - ``sphinx.ext.autosummary.generate._simple_warn()``
+     - 3.5
+     - 5.0
+     - :ref:`logging-api`
+
+   * - ``sphinx.writers.html.HTMLTranslator.permalink_text``
+     - 3.5
+     - 5.0
+     - :confval:`html_permalinks_icon`
+
+   * - ``sphinx.writers.html5.HTML5Translator.permalink_text``
+     - 3.5
+     - 5.0
+     - :confval:`html_permalinks_icon`
+
    * - The ``follow_wrapped`` argument of ``sphinx.util.inspect.signature()``
      - 3.4
      - 5.0
      - N/A
 
+   * - The ``no_docstring`` argument of
+       ``sphinx.ext.autodoc.Documenter.add_content()``
+     - 3.4
+     - 5.0
+     - ``sphinx.ext.autodoc.Documenter.get_doc()``
+
+   * - ``sphinx.ext.autodoc.Documenter.get_object_members()``
+     - 3.4
+     - 6.0
+     - ``sphinx.ext.autodoc.ClassDocumenter.get_object_members()``
+
    * - ``sphinx.ext.autodoc.DataDeclarationDocumenter``
      - 3.4
      - 5.0
      - ``sphinx.ext.autodoc.DataDocumenter``
 
+   * - ``sphinx.ext.autodoc.GenericAliasDocumenter``
+     - 3.4
+     - 5.0
+     - ``sphinx.ext.autodoc.DataDocumenter``
+
+   * - ``sphinx.ext.autodoc.InstanceAttributeDocumenter``
+     - 3.4
+     - 5.0
+     - ``sphinx.ext.autodoc.AttributeDocumenter``
+
+   * - ``sphinx.ext.autodoc.SlotsAttributeDocumenter``
+     - 3.4
+     - 5.0
+     - ``sphinx.ext.autodoc.AttributeDocumenter``
+
+   * - ``sphinx.ext.autodoc.TypeVarDocumenter``
+     - 3.4
+     - 5.0
+     - ``sphinx.ext.autodoc.DataDocumenter``
+
+   * - ``sphinx.ext.autodoc.directive.DocumenterBridge.reporter``
+     - 3.5
+     - 5.0
+     - ``sphinx.util.logging``
+
+   * - ``sphinx.ext.autodoc.importer._getannotations()``
+     - 3.4
+     - 4.0
+     - ``sphinx.util.inspect.getannotations()``
+
+   * - ``sphinx.ext.autodoc.importer._getmro()``
+     - 3.4
+     - 4.0
+     - ``sphinx.util.inspect.getmro()``
+
    * - ``sphinx.pycode.ModuleAnalyzer.parse()``
      - 3.4
      - 5.0
      - ``sphinx.pycode.ModuleAnalyzer.analyze()``
 
+   * - ``sphinx.util.osutil.movefile()``
+     - 3.4
+     - 5.0
+     - ``os.replace()``
+
    * - ``sphinx.util.requests.is_ssl_error()``
      - 3.4
      - 5.0

doc/extdev/domainapi.rst

@@ -1,7 +1,7 @@
 .. _domain-api:
 
 Domain API
-----------
+==========
 
 .. module:: sphinx.domains
 
@@ -12,3 +12,16 @@ Domain API
 
 .. autoclass:: Index
    :members:
+
+
+Python Domain
+-------------
+
+.. module:: sphinx.domains.python
+
+.. autoclass:: PythonDomain
+
+   .. autoattribute:: objects
+   .. autoattribute:: modules
+   .. automethod:: note_object
+   .. automethod:: note_module

doc/internals/contributing.rst

@@ -138,10 +138,7 @@ Coding style
 
 Please follow these guidelines when writing code for Sphinx:
 
-* Try to use the same code style as used in the rest of the project.  See the
+* Try to use the same code style as used in the rest of the project.
-  `Pocoo Styleguide`__ for more information.
-
-  __ http://flask.pocoo.org/docs/styleguide/
 
 * For non-trivial changes, please update the :file:`CHANGES` file.  If your
   changes alter existing behavior, please document this.

doc/latex.rst

@@ -95,6 +95,12 @@ Keys that you may want to override include:
    A string which will be positioned early in the preamble, designed to
    contain ``\\PassOptionsToPackage{options}{foo}`` commands.
 
+   .. hint::
+
+      It may be also used for loading LaTeX packages very early in the
+      preamble.  For example package ``fancybox`` is incompatible with
+      being loaded via the ``'preamble'`` key, it must be loaded earlier.
+
    Default: ``''``
 
    .. versionadded:: 1.4
@@ -195,8 +201,8 @@ Keys that you may want to override include:
    "Bjornstrup".  You can also set this to ``''`` to disable fncychap.
 
    Default: ``'\\usepackage[Bjarne]{fncychap}'`` for English documents,
-       ``'\\usepackage[Sonny]{fncychap}'`` for internationalized documents, and
+   ``'\\usepackage[Sonny]{fncychap}'`` for internationalized documents, and
-       ``''`` for Japanese documents.
+   ``''`` for Japanese documents.
 
 ``'preamble'``
    Additional preamble content.  One may move all needed macros into some file
@@ -300,7 +306,7 @@ Keys that don't need to be overridden unless in special cases are:
    "inputenc" package inclusion.
 
    Default: ``'\\usepackage[utf8]{inputenc}'`` when using pdflatex, else
-       ``''``
+   ``''``
 
    .. versionchanged:: 1.4.3
       Previously ``'\\usepackage[utf8]{inputenc}'`` was used for all
@@ -389,7 +395,7 @@ Keys that don't need to be overridden unless in special cases are:
    key is ignored.
 
    Default: ``'\\usepackage{textalpha}'`` or ``''`` if ``fontenc`` does not
-       include the ``LGR`` option.
+   include the ``LGR`` option.
 
    .. versionadded:: 2.0
 
@@ -407,7 +413,7 @@ Keys that don't need to be overridden unless in special cases are:
    <latexsphinxsetup>`.
 
    Default: ``'\\usepackage{geometry}'`` (or
-       ``'\\usepackage[dvipdfm]{geometry}'`` for Japanese documents)
+   ``'\\usepackage[dvipdfm]{geometry}'`` for Japanese documents)
 
    .. versionadded:: 1.5
 
@@ -784,14 +790,14 @@ macros may be significant.
    |warningbdcolors|
        The colour for the admonition frame.
 
-       Default: ``{rgb}{0,0,0}`` (black)
+   Default: ``{rgb}{0,0,0}`` (black)
 
 .. only:: latex
 
    |wgbdcolorslatex|
        The colour for the admonition frame.
 
-       Default: ``{rgb}{0,0,0}`` (black)
+   Default: ``{rgb}{0,0,0}`` (black)
 
 |warningbgcolors|
     The background colours for the respective admonitions.
@@ -1018,6 +1024,14 @@ Environments
 Miscellany
 ~~~~~~~~~~
 
+- Every text paragraph in document body starts with ``\sphinxAtStartPar``.
+  Currently, this is used to insert a zero width horizontal skip which
+  is a trick to allow TeX hyphenation of the first word of a paragraph
+  in a narrow context (like a table cell). For ``'lualatex'`` which
+  does not need the trick, the ``\sphinxAtStartPar`` does nothing.
+
+  .. versionadded:: 3.5.0
+
 - The section, subsection, ... headings are set using  *titlesec*'s
   ``\titleformat`` command.
 

doc/usage/builders/index.rst

@@ -442,6 +442,10 @@ name is ``rinoh``. Refer to the `rinohtype manual`_ for details.
 
       Since Sphinx-1.5, the linkcheck builder comes to use requests module.
 
+   .. versionchanged:: 3.4
+
+      The linkcheck builder retries links when servers apply rate limits.
+
 .. module:: sphinx.builders.xml
 .. class:: XMLBuilder
 

doc/usage/configuration.rst

@@ -73,6 +73,12 @@ Project information
 
    A copyright statement in the style ``'2008, Author Name'``.
 
+.. confval:: project_copyright
+
+   An alias of :confval:`copyright`.
+
+   .. versionadded:: 3.5
+
 .. confval:: version
 
    The major project version, used as the replacement for ``|version|``.  For
@@ -556,7 +562,7 @@ General configuration
    * Otherwise, the current time is formatted using :func:`time.strftime` and
      the format given in :confval:`today_fmt`.
 
-   The default is now :confval:`today` and a :confval:`today_fmt` of ``'%B %d,
+   The default is now :confval:`today` and a :confval:`today_fmt` of ``'%b %d,
    %Y'`` (or, if translation is enabled with :confval:`language`, an equivalent
    format for the selected locale).
 
@@ -577,12 +583,27 @@ General configuration
 
 .. confval:: highlight_options
 
-   A dictionary of options that modify how the lexer specified by
+   A dictionary that maps language names to options for the lexer modules of
-   :confval:`highlight_language` generates highlighted source code. These are
+   Pygments.  These are lexer-specific; for the options understood by each,
-   lexer-specific; for the options understood by each, see the
+   see the `Pygments documentation <https://pygments.org/docs/lexers>`_.
-   `Pygments documentation <https://pygments.org/docs/lexers>`_.
+
+   Example::
+
+     highlight_options = {
+       'default': {'stripall': True},
+       'php': {'startinline': True},
+     }
+
+   A single dictionary of options are also allowed.  Then it is recognized
+   as options to the lexer specified by :confval:`highlight_language`::
+
+     # configuration for the ``highlight_language``
+     highlight_options = {'stripall': True}
 
    .. versionadded:: 1.3
+   .. versionchanged:: 3.5
+
+      Allow to configure highlight options for multiple languages
 
 .. confval:: pygments_style
 
@@ -937,8 +958,11 @@ that use Sphinx's HTMLWriter class.
 
 .. confval:: html_baseurl
 
-   The URL which points to the root of the HTML documentation.  It is used to
+   The base URL which points to the root of the HTML documentation.  It is used
-   indicate the location of document like ``canonical_url``.
+   to indicate the location of document using `The Canonical Link Relation`_.
+   Default: ``''``.
+
+   .. _The Canonical Link Relation: https://tools.ietf.org/html/rfc6596
 
    .. versionadded:: 1.8
 
@@ -996,7 +1020,14 @@ that use Sphinx's HTMLWriter class.
                          'https://example.com/css/custom.css',
                          ('print.css', {'media': 'print'})]
 
+   As a special attribute, *priority* can be set as an integer to load the CSS
+   file earlier or lazier step.  For more information, refer
+   :meth:`Sphinx.add_css_files()`.
+
    .. versionadded:: 1.8
+   .. versionchanged:: 3.5
+
+      Support priority attribute
 
 .. confval:: html_js_files
 
@@ -1012,7 +1043,14 @@ that use Sphinx's HTMLWriter class.
                         'https://example.com/scripts/custom.js',
                         ('custom.js', {'async': 'async'})]
 
+   As a special attribute, *priority* can be set as an integer to load the CSS
+   file earlier or lazier step.  For more information, refer
+   :meth:`Sphinx.add_css_files()`.
+
    .. versionadded:: 1.8
+   .. versionchanged:: 3.5
+
+      Support priority attribute
 
 .. confval:: html_static_path
 
@@ -1096,6 +1134,23 @@ that use Sphinx's HTMLWriter class.
       This can now be a string to select the actual text of the link.
       Previously, only boolean values were accepted.
 
+   .. deprecated:: 3.5
+      This has been replaced by :confval:`html_permalinks`
+
+.. confval:: html_permalinks
+
+   If true, Sphinx will add "permalinks" for each heading and description
+   environment.  Default: ``True``.
+
+   .. versionadded:: 3.5
+
+.. confval:: html_permalinks_icon
+
+   A text for permalinks for each heading and description environment.  HTML
+   tags are allowed.  Default: a paragraph sign; ``¶``
+
+   .. versionadded:: 3.5
+
 .. confval:: html_sidebars
 
    Custom sidebar templates, must be a dictionary that maps document names to
@@ -2525,6 +2580,23 @@ Options for the linkcheck builder
 
    .. versionadded:: 2.3
 
+.. confval:: linkcheck_rate_limit_timeout
+
+   The ``linkcheck`` builder may issue a large number of requests to the same
+   site over a short period of time. This setting controls the builder behavior
+   when servers indicate that requests are rate-limited.
+
+   If a server indicates when to retry (using the `Retry-After`_ header),
+   ``linkcheck`` always follows the server indication.
+
+   Otherwise, ``linkcheck`` waits for a minute before to retry and keeps
+   doubling the wait time between attempts until it succeeds or exceeds the
+   ``linkcheck_rate_limit_timeout``. By default, the timeout is 5 minutes.
+
+   .. _Retry-After: https://tools.ietf.org/html/rfc7231#section-7.1.3
+
+   .. versionadded:: 3.4
+
 
 Options for the XML builder
 ---------------------------

doc/usage/extensions/autodoc.rst

@@ -157,7 +157,7 @@ inserting them into the page source under a suitable :rst:dir:`py:module`,
      ``:meta private:`` in its :ref:`info-field-lists`.
      For example:
 
-     .. code-block:: rst
+     .. code-block:: python
 
         def my_function(my_arg, my_other_arg):
             """blah blah blah
@@ -172,7 +172,7 @@ inserting them into the page source under a suitable :rst:dir:`py:module`,
      an underscore.
      For example:
 
-     .. code-block:: rst
+     .. code-block:: python
 
         def _my_function(my_arg, my_other_arg):
             """blah blah blah
@@ -182,6 +182,16 @@ inserting them into the page source under a suitable :rst:dir:`py:module`,
 
      .. versionadded:: 3.1
 
+   * autodoc considers a variable member does not have any default value if its
+     docstring contains ``:meta hide-value:`` in its :ref:`info-field-lists`.
+     Example:
+
+     .. code-block:: python
+
+        var1 = None  #: :meta hide-value:
+
+     .. versionadded:: 3.5
+
    * Python "special" members (that is, those named like ``__special__``) will
      be included if the ``special-members`` flag option is given::
 
@@ -554,7 +564,7 @@ There are also config values that you can set:
         ...
 
    If you set ``autodoc_type_aliases`` as
-   ``{'AliasType': 'your.module.TypeAlias'}``, it generates a following document
+   ``{'AliasType': 'your.module.AliasType'}``, it generates the following document
    internally::
 
      .. py:function:: f() -> your.module.AliasType:

doc/usage/extensions/autosummary.rst

@@ -304,7 +304,7 @@ The following variables available in the templates:
 .. data:: modules
 
    List containing names of "public" modules in the package.  Only available for
-   modules that are packages.
+   modules that are packages and the ``recursive`` option is on.
 
    .. versionadded:: 3.1
 

doc/usage/extensions/example_google.py

@@ -294,3 +294,21 @@ class ExampleClass:
 
     def _private_without_docstring(self):
         pass
+
+class ExamplePEP526Class:
+    """The summary line for a class docstring should fit on one line.
+
+    If the class has public attributes, they may be documented here
+    in an ``Attributes`` section and follow the same formatting as a
+    function's ``Args`` section. If ``napoleon_attr_annotations``
+    is True, types can be specified in the class body using ``PEP 526``
+    annotations.
+
+    Attributes:
+        attr1: Description of `attr1`.
+        attr2: Description of `attr2`.
+
+    """
+
+    attr1: str
+    attr2: int

doc/usage/extensions/index.rst

@@ -41,22 +41,20 @@ These extensions are built in and can be activated by respective entries in the
 Third-party extensions
 ----------------------
 
-.. todo:: This should reference the GitHub organization now
+You can find several extensions contributed by users in the `sphinx-contrib`__
+organization. If you wish to include your extension in this organization,
+simply follow the instructions provided in the `github-administration`__
+project. This is optional and there are several extensions hosted elsewhere.
+The `awesome-sphinxdoc`__ project contains a curated list of Sphinx packages,
+and many packages use the `Framework :: Sphinx :: Extension`__ and
+`Framework :: Sphinx :: Theme`__ trove classifiers for Sphinx extensions and
+themes, respectively.
 
-You can find several extensions contributed by users in the `Sphinx Contrib`_
+.. __: https://github.com/sphinx-contrib/
-repository.  It is open for anyone who wants to maintain an extension publicly;
+.. __: https://github.com/sphinx-contrib/github-administration
-just send a short message asking for write permissions.
+.. __: https://github.com/yoloseem/awesome-sphinxdoc
-
+.. __: https://pypi.org/search/?c=Framework+%3A%3A+Sphinx+%3A%3A+Extension
-There are also several extensions hosted elsewhere.  The `Sphinx extension
+.. __: https://pypi.org/search/?c=Framework+%3A%3A+Sphinx+%3A%3A+Theme
-survey <https://sphinxext-survey.readthedocs.io/>`__ and `awesome-sphinxdoc
-<https://github.com/yoloseem/awesome-sphinxdoc>`__ contains a comprehensive
-list.
-
-If you write an extension that you think others will find useful or you think
-should be included as a part of Sphinx, please write to the project mailing
-list (`join here <https://groups.google.com/forum/#!forum/sphinx-dev>`_).
-
-.. _Sphinx Contrib: https://bitbucket.org/birkenfeld/sphinx-contrib
 
 Where to put your own extensions?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

doc/usage/extensions/intersphinx.rst

@@ -75,8 +75,9 @@ linking:
    A dictionary mapping unique identifiers to a tuple ``(target, inventory)``.
    Each ``target`` is the base URI of a foreign Sphinx documentation set and can
    be a local path or an HTTP URI.  The ``inventory`` indicates where the
-   inventory file can be found: it can be ``None`` (at the same location as
+   inventory file can be found: it can be ``None`` (an :file:`objects.inv` file
-   the base URI) or another local or HTTP URI.
+   at the same location as the base URI) or another local file path or a full
+   HTTP URI to an inventory file.
 
    The unique identifier can be used to prefix cross-reference targets, so that
    it is clear which intersphinx set the target belongs to.  A link like
@@ -106,7 +107,7 @@ linking:
    ``https://docs.python.org/3``.  It is up to you to update the inventory file
    as new objects are added to the Python documentation.
 
-   **Multiple target for the inventory**
+   **Multiple targets for the inventory**
 
    .. versionadded:: 1.3
 
@@ -120,6 +121,16 @@ linking:
       intersphinx_mapping = {'python': ('https://docs.python.org/3',
                                         (None, 'python-inv.txt'))}
 
+   For a set of books edited and tested locally and then published
+   together, it could be helpful to try a local inventory file first,
+   to check references before publication::
+
+      intersphinx_mapping = {
+          'otherbook':
+              ('https://myproj.readthedocs.io/projects/otherbook/en/latest',
+                  ('../../otherbook/build/html/objects.inv', None)),
+      }
+
 .. confval:: intersphinx_cache_limit
 
    The maximum number of days to cache remote inventories.  The default is

doc/usage/extensions/napoleon.rst

@@ -203,7 +203,8 @@ Type Annotations
 This is an alternative to expressing types directly in docstrings.
 One benefit of expressing types according to `PEP 484`_ is that
 type checkers and IDEs can take advantage of them for static code
-analysis.
+analysis. `PEP 484`_ was then extended by `PEP 526`_ which introduced
+a similar way to annotate variables (and attributes).
 
 Google style with Python 3 type annotations::
 
@@ -221,6 +222,19 @@ Google style with Python 3 type annotations::
 
         """
         return True
+    
+    class Class:
+        """Summary line.
+
+        Extended description of class
+
+        Attributes:
+            attr1: Description of attr1
+            attr2: Description of attr2
+        """
+
+        attr1: int
+        attr2: str
 
 Google style with types in docstrings::
 
@@ -238,6 +252,16 @@ Google style with types in docstrings::
 
         """
         return True
+    
+    class Class:
+        """Summary line.
+
+        Extended description of class
+
+        Attributes:
+            attr1 (int): Description of attr1
+            attr2 (str): Description of attr2
+        """
 
 .. Note::
    `Python 2/3 compatible annotations`_ aren't currently
@@ -246,6 +270,9 @@ Google style with types in docstrings::
 .. _PEP 484:
    https://www.python.org/dev/peps/pep-0484/
 
+.. _PEP 526:
+    https://www.python.org/dev/peps/pep-0526/
+
 .. _Python 2/3 compatible annotations:
    https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code
 
@@ -275,6 +302,7 @@ sure that "sphinx.ext.napoleon" is enabled in `conf.py`::
     napoleon_use_param = True
     napoleon_use_rtype = True
     napoleon_type_aliases = None
+    napoleon_attr_annotations = True
 
 .. _Google style:
    https://google.github.io/styleguide/pyguide.html
@@ -511,3 +539,35 @@ sure that "sphinx.ext.napoleon" is enabled in `conf.py`::
        :type arg2: :term:`dict-like <mapping>`
 
    .. versionadded:: 3.2
+
+.. confval:: napoleon_attr_annotations
+
+   True to allow using `PEP 526`_ attributes annotations in classes.
+   If an attribute is documented in the docstring without a type and
+   has an annotation in the class body, that type is used.
+
+   .. versionadded:: 3.4
+
+.. confval:: napoleon_custom_sections
+
+   Add a list of custom sections to include, expanding the list of parsed sections.
+   *Defaults to None.*
+
+   The entries can either be strings or tuples, depending on the intention:
+
+   * To create a custom "generic" section, just pass a string.
+   * To create an alias for an existing section, pass a tuple containing the
+     alias name and the original, in that order.
+   * To create a custom section that displays like the parameters or returns
+     section, pass a tuple containing the custom section name and a string
+     value, "params_style" or "returns_style".
+
+   If an entry is just a string, it is interpreted as a header for a generic
+   section. If the entry is a tuple/list/indexed container, the first entry
+   is the name of the section, the second is the section key to emulate. If the
+   second entry value is "params_style" or "returns_style", the custom section
+   will be displayed like the parameters section or returns section.
+
+   .. versionadded:: 1.8
+   .. versionchanged:: 3.5
+      Support ``params_style`` and ``returns_style``

doc/usage/restructuredtext/directives.rst

@@ -569,12 +569,28 @@ __ http://pygments.org/docs/lexers
 
            print 'Explicit is better than implicit.'
 
+      In order to cross-reference a code-block using either the
+      :rst:role:`ref` or the :rst:role:`numref` role, it is necessary
+      that both :strong:`name` and :strong:`caption` be defined. The
+      argument of :strong:`name` can then be given to :rst:role:`numref`
+      to generate the cross-reference. Example::
+
+        See :numref:`this-py` for an example.
+
+      When using :rst:role:`ref`, it is possible to generate a cross-reference
+      with only :strong:`name` defined, provided an explicit title is
+      given. Example::
+
+        See :ref:`this code snippet <this-py>` for an example.
+
       .. versionadded:: 1.3
 
    .. rst:directive:option:: dedent: number
-      :type: number
+      :type: number or no value
 
-      Strip indentation characters from the code block. For example::
+      Strip indentation characters from the code block.  When number given,
+      leading N characters are removed.  When no argument given, leading spaces
+      are removed via :func:`textwrap.dedent()`.  For example::
 
          .. code-block:: ruby
             :dedent: 4
@@ -582,6 +598,8 @@ __ http://pygments.org/docs/lexers
                 some ruby code
 
       .. versionadded:: 1.3
+      .. versionchanged:: 3.5
+         Support automatic dedent.
 
    .. rst:directive:option:: force
       :type: no value
@@ -742,6 +760,9 @@ __ http://pygments.org/docs/lexers
    .. versionchanged:: 2.1
       Added the ``force`` option.
 
+   .. versionchanged:: 3.5
+      Support automatic dedent.
+
 .. _glossary-directive:
 
 Glossary
@@ -1199,20 +1220,29 @@ the definition of the symbol.  There is this directive:
    the following definition.  If the definition spans multiple lines, each
    continuation line must begin with a colon placed at the same column as in
    the first line.
+   Blank lines are not allowed within ``productionlist`` directive arguments.
+
+   The definition can contain token names which are marked as interpreted text
+   (e.g., "``sum ::= `integer` "+" `integer```") -- this generates
+   cross-references to the productions of these tokens.  Outside of the
+   production list, you can reference to token productions using
+   :rst:role:`token`.
 
    The *productionGroup* argument to :rst:dir:`productionlist` serves to
    distinguish different sets of production lists that belong to different
    grammars.  Multiple production lists with the same *productionGroup* thus
    define rules in the same scope.
 
-   Blank lines are not allowed within ``productionlist`` directive arguments.
+   Inside of the production list, tokens implicitly refer to productions
+   from the current group. You can refer to the production of another
+   grammar by prefixing the token with its group name and a colon, e.g,
+   "``otherGroup:sum``". If the group of the token should not be shown in
+   the production, it can be prefixed by a tilde, e.g.,
+   "``~otherGroup:sum``". To refer to a production from an unnamed
+   grammar, the token should be prefixed by a colon, e.g., "``:sum``".
 
-   The definition can contain token names which are marked as interpreted text
+   Outside of the production list,
-   (e.g. "``sum ::= `integer` "+" `integer```") -- this generates
+   if you have given a *productionGroup* argument you must prefix the
-   cross-references to the productions of these tokens.  Outside of the
-   production list, you can reference to token productions using
-   :rst:role:`token`.
-   However, if you have given a *productionGroup* argument you must prefix the
    token name in the cross-reference with the group name and a colon,
    e.g., "``myGroup:sum``" instead of just "``sum``".
    If the group should not be shown in the title of the link either

karma.conf.js

@@ -18,6 +18,7 @@ module.exports = function(config) {
       'sphinx/themes/basic/static/underscore.js',
       'sphinx/themes/basic/static/jquery.js',
       'sphinx/themes/basic/static/doctools.js',
+      'sphinx/themes/basic/static/searchtools.js',
       'tests/js/*.js'
     ],
 

setup.cfg

@@ -60,6 +60,7 @@ filterwarnings =
     all
     ignore::DeprecationWarning:docutils.io
     ignore::DeprecationWarning:pyximport.pyximport
+    ignore::ImportWarning:importlib._bootstrap
     ignore::PendingDeprecationWarning:sphinx.util.pycompat
 markers =
     apidoc

setup.py

@@ -44,14 +44,14 @@ extras_require = {
     'lint': [
         'flake8>=3.5.0',
         'isort',
-        'mypy>=0.790',
+        'mypy>=0.800',
         'docutils-stubs',
     ],
     'test': [
         'pytest',
         'pytest-cov',
         'html5lib',
-        'typed_ast',  # for py35-37
+        "typed_ast; python_version < '3.8'",
         'cython',
     ],
 }

sphinx/__init__.py

@@ -4,7 +4,7 @@
 
     The Sphinx documentation toolchain.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -32,8 +32,8 @@ if 'PYTHONWARNINGS' not in os.environ:
 warnings.filterwarnings('ignore', "'U' mode is deprecated",
                         DeprecationWarning, module='docutils.io')
 
-__version__ = '3.4.0+'
+__version__ = '3.5.0+'
-__released__ = '3.4.0'  # used when Sphinx builds its own docs
+__released__ = '3.5.0'  # used when Sphinx builds its own docs
 
 #: Version info for better programmatic use.
 #:
@@ -43,7 +43,7 @@ __released__ = '3.4.0'  # used when Sphinx builds its own docs
 #:
 #: .. versionadded:: 1.2
 #:    Before version 1.2, check the string ``sphinx.__version__``.
-version_info = (3, 4, 0, 'beta', 0)
+version_info = (3, 5, 0, 'beta', 0)
 
 package_dir = path.abspath(path.dirname(__file__))
 

sphinx/__main__.py

@@ -4,7 +4,7 @@
 
     The Sphinx documentation toolchain.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/addnodes.py

@@ -4,7 +4,7 @@
 
     Additional docutils nodes.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/application.py

@@ -6,7 +6,7 @@
 
     Gracefully adapted from the TextPress system by Armin.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -404,9 +404,10 @@ class Sphinx:
     def require_sphinx(self, version: str) -> None:
         """Check the Sphinx version if requested.
 
-        Compare *version* (which must be a ``major.minor`` version string, e.g.
+        Compare *version* with the version of the running Sphinx, and abort the
-        ``'1.1'``) with the version of the running Sphinx, and abort the build
+        build when it is too old.
-        when it is too old.
+
+        :param version: The required version in the form of ``major.minor``.
 
         .. versionadded:: 1.0
         """
@@ -420,11 +421,11 @@ class Sphinx:
         For details on available core events and the arguments of callback
         functions, please see :ref:`events`.
 
-        Registered callbacks will be invoked on event in the order of *priority* and
+        :param event: The name of target event
-        registration.  The priority is ascending order.
+        :param callback: Callback function for the event
-
+        :param priority: The priority of the callback.  The callbacks will be invoked
-        The method returns a "listener ID" that can be used as an argument to
+                         in the order of *priority* in asending.
-        :meth:`disconnect`.
+        :return: A listener ID.  It can be used for :meth:`disconnect`.
 
         .. versionchanged:: 3.0
 
@@ -436,7 +437,10 @@ class Sphinx:
         return listener_id
 
     def disconnect(self, listener_id: int) -> None:
-        """Unregister callback by *listener_id*."""
+        """Unregister callback by *listener_id*.
+
+        :param listener_id: A listener_id that :meth:`connect` returns
+        """
         logger.debug('[app] disconnecting event: [id=%s]', listener_id)
         self.events.disconnect(listener_id)
 
@@ -447,6 +451,10 @@ class Sphinx:
         Return the return values of all callbacks as a list.  Do not emit core
         Sphinx events in extensions!
 
+        :param event: The name of event that will be emitted
+        :param args: The arguments for the event
+        :param allowed_exceptions: The list of exceptions that are allowed in the callbacks
+
         .. versionchanged:: 3.1
 
            Added *allowed_exceptions* to specify path-through exceptions
@@ -459,6 +467,10 @@ class Sphinx:
 
         Return the result of the first callback that doesn't return ``None``.
 
+        :param event: The name of event that will be emitted
+        :param args: The arguments for the event
+        :param allowed_exceptions: The list of exceptions that are allowed in the callbacks
+
         .. versionadded:: 0.5
         .. versionchanged:: 3.1
 
@@ -472,10 +484,9 @@ class Sphinx:
     def add_builder(self, builder: "Type[Builder]", override: bool = False) -> None:
         """Register a new builder.
 
-        *builder* must be a class that inherits from :class:`~sphinx.builders.Builder`.
+        :param builder: A builder class
-
+        :param override: If true, install the builder forcedly even if another builder
-        If *override* is True, the given *builder* is forcedly installed even if
+                         is already installed as the same name
-        a builder having the same name is already installed.
 
         .. versionchanged:: 1.8
            Add *override* keyword.
@@ -488,27 +499,34 @@ class Sphinx:
         """Register a configuration value.
 
         This is necessary for Sphinx to recognize new values and set default
-        values accordingly.  The *name* should be prefixed with the extension
+        values accordingly.
-        name, to avoid clashes.  The *default* value can be any Python object.
-        The string value *rebuild* must be one of those values:
 
-        * ``'env'`` if a change in the setting only takes effect when a
-          document is parsed -- this means that the whole environment must be
-          rebuilt.
-        * ``'html'`` if a change in the setting needs a full rebuild of HTML
-          documents.
-        * ``''`` if a change in the setting will not need any special rebuild.
 
-        .. versionchanged:: 0.6
+        :param name: The name of configuration value.  It is recommended to be prefixed
-           Changed *rebuild* from a simple boolean (equivalent to ``''`` or
+                     with the extension name (ex. ``html_logo``, ``epub_title``)
-           ``'env'``) to a string.  However, booleans are still accepted and
+        :param default: The default value of the configuration.
-           converted internally.
+        :param rebuild: The condition of rebuild.  It must be one of those values:
+
+                        * ``'env'`` if a change in the setting only takes effect when a
+                          document is parsed -- this means that the whole environment must be
+                          rebuilt.
+                        * ``'html'`` if a change in the setting needs a full rebuild of HTML
+                          documents.
+                        * ``''`` if a change in the setting will not need any special rebuild.
+        :param types: The type of configuration value.  A list of types can be specified.  For
+                      example, ``[str]`` is used to describe a configuration that takes string
+                      value.
 
         .. versionchanged:: 0.4
            If the *default* value is a callable, it will be called with the
            config object as its argument in order to get the default value.
            This can be used to implement config values whose default depends on
            other values.
+
+        .. versionchanged:: 0.6
+           Changed *rebuild* from a simple boolean (equivalent to ``''`` or
+           ``'env'``) to a string.  However, booleans are still accepted and
+           converted internally.
         """
         logger.debug('[app] adding config value: %r',
                      (name, default, rebuild) + ((types,) if types else ()))
@@ -520,6 +538,8 @@ class Sphinx:
         """Register an event called *name*.
 
         This is needed to be able to emit it.
+
+        :param name: The name of the event
         """
         logger.debug('[app] adding event: %r', name)
         self.events.add(name)
@@ -532,8 +552,10 @@ class Sphinx:
         builtin translator.  This allows extensions to use custom translator
         and define custom nodes for the translator (see :meth:`add_node`).
 
-        If *override* is True, the given *translator_class* is forcedly installed even if
+        :param name: The name of builder for the translator
-        a translator for *name* is already installed.
+        :param translator_class: A translator class
+        :param override: If true, install the translator forcedly even if another translator
+                         is already installed as the same name
 
         .. versionadded:: 1.3
         .. versionchanged:: 1.8
@@ -548,6 +570,11 @@ class Sphinx:
         This is necessary for Docutils internals.  It may also be used in the
         future to validate nodes in the parsed documents.
 
+        :param node: A node class
+        :param kwargs: Visitor functions for each builder (see below)
+        :param override: If true, install the node forcedly even if another node is already
+                         installed as the same name
+
         Node visitor functions for the Sphinx HTML, LaTeX, text and manpage
         writers can be given as keyword arguments: the keyword should be one or
         more of ``'html'``, ``'latex'``, ``'text'``, ``'man'``, ``'texinfo'``
@@ -569,9 +596,6 @@ class Sphinx:
         Obviously, translators for which you don't specify visitor methods will
         choke on the node when encountered in a document to translate.
 
-        If *override* is True, the given *node* is forcedly installed even if
-        a node having the same name is already installed.
-
         .. versionchanged:: 0.5
            Added the support for keyword arguments giving visit functions.
         """
@@ -591,24 +615,21 @@ class Sphinx:
         Sphinx numbers the node automatically. And then the users can refer it
         using :rst:role:`numref`.
 
-        *figtype* is a type of enumerable nodes.  Each figtypes have individual
+        :param node: A node class
-        numbering sequences.  As a system figtypes, ``figure``, ``table`` and
+        :param figtype: The type of enumerable nodes.  Each figtypes have individual numbering
-        ``code-block`` are defined.  It is able to add custom nodes to these
+                        sequences.  As a system figtypes, ``figure``, ``table`` and
-        default figtypes.  It is also able to define new custom figtype if new
+                        ``code-block`` are defined.  It is able to add custom nodes to these
-        figtype is given.
+                        default figtypes.  It is also able to define new custom figtype if new
-
+                        figtype is given.
-        *title_getter* is a getter function to obtain the title of node.  It
+        :param title_getter: A getter function to obtain the title of node.  It takes an
-        takes an instance of the enumerable node, and it must return its title
+                             instance of the enumerable node, and it must return its title as
-        as string.  The title is used to the default title of references for
+                             string.  The title is used to the default title of references for
-        :rst:role:`ref`.  By default, Sphinx searches
+                             :rst:role:`ref`.  By default, Sphinx searches
-        ``docutils.nodes.caption`` or ``docutils.nodes.title`` from the node as
+                             ``docutils.nodes.caption`` or ``docutils.nodes.title`` from the
-        a title.
+                             node as a title.
-
+        :param kwargs: Visitor functions for each builder (same as :meth:`add_node`)
-        Other keyword arguments are used for node visitor functions. See the
+        :param override: If true, install the node forcedly even if another node is already
-        :meth:`.Sphinx.add_node` for details.
+                         installed as the same name
-
-        If *override* is True, the given *node* is forcedly installed even if
-        a node having the same name is already installed.
 
         .. versionadded:: 1.4
         """
@@ -618,10 +639,10 @@ class Sphinx:
     def add_directive(self, name: str, cls: "Type[Directive]", override: bool = False) -> None:
         """Register a Docutils directive.
 
-        *name* must be the prospective directive name.  *cls* is a directive
+        :param name: The name of directive
-        class which inherits ``docutils.parsers.rst.Directive``.  For more
+        :param cls: A directive class
-        details, see `the Docutils docs
+        :param override: If true, install the directive forcedly even if another directive
-        <http://docutils.sourceforge.net/docs/howto/rst-directives.html>`_ .
+                         is already installed as the same name
 
         For example, a custom directive named ``my-directive`` would be added
         like this:
@@ -646,8 +667,8 @@ class Sphinx:
            def setup(app):
                add_directive('my-directive', MyDirective)
 
-        If *override* is True, the given *cls* is forcedly installed even if
+        For more details, see `the Docutils docs
-        a directive named as *name* is already installed.
+        <http://docutils.sourceforge.net/docs/howto/rst-directives.html>`__ .
 
         .. versionchanged:: 0.6
            Docutils 0.5-style directive classes are now supported.
@@ -666,13 +687,13 @@ class Sphinx:
     def add_role(self, name: str, role: Any, override: bool = False) -> None:
         """Register a Docutils role.
 
-        *name* must be the role name that occurs in the source, *role* the role
+        :param name: The name of role
-        function. Refer to the `Docutils documentation
+        :param role: A role function
-        <http://docutils.sourceforge.net/docs/howto/rst-roles.html>`_ for
+        :param override: If true, install the role forcedly even if another role is already
-        more information.
+                         installed as the same name
 
-        If *override* is True, the given *role* is forcedly installed even if
+        For more details about role functions, see `the Docutils docs
-        a role named as *name* is already installed.
+        <http://docutils.sourceforge.net/docs/howto/rst-roles.html>`__ .
 
         .. versionchanged:: 1.8
            Add *override* keyword.
@@ -708,11 +729,9 @@ class Sphinx:
     def add_domain(self, domain: "Type[Domain]", override: bool = False) -> None:
         """Register a domain.
 
-        Make the given *domain* (which must be a class; more precisely, a
+        :param domain: A domain class
-        subclass of :class:`~sphinx.domains.Domain`) known to Sphinx.
+        :param override: If true, install the domain forcedly even if another domain
-
+                         is already installed as the same name
-        If *override* is True, the given *domain* is forcedly installed even if
-        a domain having the same name is already installed.
 
         .. versionadded:: 1.0
         .. versionchanged:: 1.8
@@ -727,8 +746,11 @@ class Sphinx:
         Like :meth:`add_directive`, but the directive is added to the domain
         named *domain*.
 
-        If *override* is True, the given *directive* is forcedly installed even if
+        :param domain: The name of target domain
-        a directive named as *name* is already installed.
+        :param name: A name of directive
+        :param cls: A directive class
+        :param override: If true, install the directive forcedly even if another directive
+                         is already installed as the same name
 
         .. versionadded:: 1.0
         .. versionchanged:: 1.8
@@ -743,8 +765,11 @@ class Sphinx:
         Like :meth:`add_role`, but the role is added to the domain named
         *domain*.
 
-        If *override* is True, the given *role* is forcedly installed even if
+        :param domain: The name of target domain
-        a role named as *name* is already installed.
+        :param name: A name of role
+        :param role: A role function
+        :param override: If true, install the role forcedly even if another role is already
+                         installed as the same name
 
         .. versionadded:: 1.0
         .. versionchanged:: 1.8
@@ -756,11 +781,12 @@ class Sphinx:
                             ) -> None:
         """Register a custom index for a domain.
 
-        Add a custom *index* class to the domain named *domain*.  *index* must
+        Add a custom *index* class to the domain named *domain*.
-        be a subclass of :class:`~sphinx.domains.Index`.
 
-        If *override* is True, the given *index* is forcedly installed even if
+        :param domain: The name of target domain
-        an index having the same name is already installed.
+        :param index: A index class
+        :param override: If true, install the index forcedly even if another index is
+                         already installed as the same name
 
         .. versionadded:: 1.0
         .. versionchanged:: 1.8
@@ -881,6 +907,8 @@ class Sphinx:
         the list of transforms that are applied after Sphinx parses a reST
         document.
 
+        :param transform: A transform class
+
         .. list-table:: priority range categories for Sphinx transforms
            :widths: 20,80
 
@@ -913,25 +941,29 @@ class Sphinx:
         Add the standard docutils :class:`Transform` subclass *transform* to
         the list of transforms that are applied before Sphinx writes a
         document.
+
+        :param transform: A transform class
         """
         self.registry.add_post_transform(transform)
 
-    def add_javascript(self, filename: str, **kwargs: str) -> None:
+    def add_javascript(self, filename: str, **kwargs: Any) -> None:
         """An alias of :meth:`add_js_file`."""
         warnings.warn('The app.add_javascript() is deprecated. '
                       'Please use app.add_js_file() instead.',
                       RemovedInSphinx40Warning, stacklevel=2)
         self.add_js_file(filename, **kwargs)
 
-    def add_js_file(self, filename: str, **kwargs: str) -> None:
+    def add_js_file(self, filename: str, priority: int = 500, **kwargs: Any) -> None:
         """Register a JavaScript file to include in the HTML output.
 
         Add *filename* to the list of JavaScript files that the default HTML
-        template will include.  The filename must be relative to the HTML
+        template will include in order of *priority* (ascending).  The filename
-        static path , or a full URI with scheme.  If the keyword argument
+        must be relative to the HTML static path , or a full URI with scheme.
-        ``body`` is given, its value will be added between the
+        If the priority of JavaScript file is the same as others, the JavaScript
-        ``<script>`` tags. Extra keyword arguments are included as
+        files will be included in order of the registration.  If the keyword
-        attributes of the ``<script>`` tag.
+        argument ``body`` is given, its value will be added between the
+        ``<script>`` tags. Extra keyword arguments are included as attributes of
+        the ``<script>`` tag.
 
         Example::
 
@@ -944,23 +976,43 @@ class Sphinx:
             app.add_js_file(None, body="var myVariable = 'foo';")
             # => <script>var myVariable = 'foo';</script>
 
+        .. list-table:: priority range for JavaScript files
+           :widths: 20,80
+
+           * - Priority
+             - Main purpose in Sphinx
+           * - 200
+             - default priority for built-in JavaScript files
+           * - 500
+             - default priority for extensions
+           * - 800
+             - default priority for :confval:`html_js_files`
+
+        A JavaScript file can be added to the specific HTML page when on extension
+        calls this method on :event:`html-page-context` event.
+
         .. versionadded:: 0.5
 
         .. versionchanged:: 1.8
            Renamed from ``app.add_javascript()``.
            And it allows keyword arguments as attributes of script tag.
-        """
-        self.registry.add_js_file(filename, **kwargs)
-        if hasattr(self.builder, 'add_js_file'):
-            self.builder.add_js_file(filename, **kwargs)  # type: ignore
 
-    def add_css_file(self, filename: str, **kwargs: str) -> None:
+        .. versionchanged:: 3.5
+           Take priority argument.  Allow to add a JavaScript file to the specific page.
+        """
+        self.registry.add_js_file(filename, priority=priority, **kwargs)
+        if hasattr(self.builder, 'add_js_file'):
+            self.builder.add_js_file(filename, priority=priority, **kwargs)  # type: ignore
+
+    def add_css_file(self, filename: str, priority: int = 500, **kwargs: Any) -> None:
         """Register a stylesheet to include in the HTML output.
 
         Add *filename* to the list of CSS files that the default HTML template
-        will include.  The filename must be relative to the HTML static path,
+        will include in order of *priority* (ascending).  The filename must be
-        or a full URI with scheme.  The keyword arguments are also accepted for
+        relative to the HTML static path, or a full URI with scheme.  If the
-        attributes of ``<link>`` tag.
+        priority of CSS file is the same as others, the CSS files will be
+        included in order of the registration.  The keyword arguments are also
+        accepted for attributes of ``<link>`` tag.
 
         Example::
 
@@ -975,6 +1027,19 @@ class Sphinx:
             # => <link rel="alternate stylesheet" href="_static/fancy.css"
             #          type="text/css" title="fancy" />
 
+        .. list-table:: priority range for CSS files
+           :widths: 20,80
+
+           * - Priority
+             - Main purpose in Sphinx
+           * - 500
+             - default priority for extensions
+           * - 800
+             - default priority for :confval:`html_css_files`
+
+        A CSS file can be added to the specific HTML page when on extension calls
+        this method on :event:`html-page-context` event.
+
         .. versionadded:: 1.0
 
         .. versionchanged:: 1.6
@@ -987,11 +1052,14 @@ class Sphinx:
         .. versionchanged:: 1.8
            Renamed from ``app.add_stylesheet()``.
            And it allows keyword arguments as attributes of link tag.
+
+        .. versionchanged:: 3.5
+           Take priority argument.  Allow to add a CSS file to the specific page.
         """
         logger.debug('[app] adding stylesheet: %r', filename)
-        self.registry.add_css_files(filename, **kwargs)
+        self.registry.add_css_files(filename, priority=priority, **kwargs)
         if hasattr(self.builder, 'add_css_file'):
-            self.builder.add_css_file(filename, **kwargs)  # type: ignore
+            self.builder.add_css_file(filename, priority=priority, **kwargs)  # type: ignore
 
     def add_stylesheet(self, filename: str, alternate: bool = False, title: str = None
                        ) -> None:
@@ -1000,7 +1068,7 @@ class Sphinx:
                       'Please use app.add_css_file() instead.',
                       RemovedInSphinx40Warning, stacklevel=2)
 
-        attributes = {}  # type: Dict[str, str]
+        attributes = {}  # type: Dict[str, Any]
         if alternate:
             attributes['rel'] = 'alternate stylesheet'
         else:
@@ -1175,9 +1243,10 @@ class Sphinx:
     def add_message_catalog(self, catalog: str, locale_dir: str) -> None:
         """Register a message catalog.
 
-        The *catalog* is a name of catalog, and *locale_dir* is a base path
+        :param catalog: A name of catalog
-        of message catalog.  For more details, see
+        :param locale_dir: The base path of message catalog
-        :func:`sphinx.locale.get_translation()`.
+
+        For more details, see :func:`sphinx.locale.get_translation()`.
 
         .. versionadded:: 1.8
         """
@@ -1188,7 +1257,7 @@ class Sphinx:
     def is_parallel_allowed(self, typ: str) -> bool:
         """Check parallel processing is allowed or not.
 
-        ``typ`` is a type of processing; ``'read'`` or ``'write'``.
+        :param typ: A type of processing; ``'read'`` or ``'write'``.
         """
         if typ == 'read':
             attrname = 'parallel_read_safe'

sphinx/builders/__init__.py

@@ -4,7 +4,7 @@
 
     Builder superclass for all builders.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/_epub_base.py

@@ -4,7 +4,7 @@
 
     Base class of epub2/epub3 builders.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/applehelp.py

@@ -4,7 +4,7 @@
 
     Build Apple help books.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/changes.py

@@ -4,7 +4,7 @@
 
     Changelog builder.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/devhelp.py

@@ -6,7 +6,7 @@
 
     .. _Devhelp: https://wiki.gnome.org/Apps/Devhelp
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/dirhtml.py

@@ -4,7 +4,7 @@
 
     Directory HTML builders.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/dummy.py

@@ -4,7 +4,7 @@
 
     Do syntax checks, but no writing.
 
-    :copyright: Copyright 2007-2015 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/epub3.py

@@ -5,7 +5,7 @@
     Build epub3 files.
     Originally derived from epub.py.
 
-    :copyright: Copyright 2007-2015 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/gettext.py

@@ -4,7 +4,7 @@
 
     The MessageCatalogBuilder class.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -315,7 +315,7 @@ class MessageCatalogBuilder(I18nBuilder):
 def setup(app: Sphinx) -> Dict[str, Any]:
     app.add_builder(MessageCatalogBuilder)
 
-    app.add_config_value('gettext_compact', True, 'gettext', Any)
+    app.add_config_value('gettext_compact', True, 'gettext', {bool, str})
     app.add_config_value('gettext_location', True, 'gettext')
     app.add_config_value('gettext_uuid', False, 'gettext')
     app.add_config_value('gettext_auto_build', True, 'env')

sphinx/builders/html/__init__.py

@@ -4,11 +4,12 @@
 
     Several HTML builders.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
 import html
+import os
 import posixpath
 import re
 import sys
@@ -44,7 +45,7 @@ from sphinx.util.fileutil import copy_asset
 from sphinx.util.i18n import format_date
 from sphinx.util.inventory import InventoryFile
 from sphinx.util.matching import DOTFILES, Matcher, patmatch
-from sphinx.util.osutil import copyfile, ensuredir, movefile, os_path, relative_uri
+from sphinx.util.osutil import copyfile, ensuredir, os_path, relative_uri
 from sphinx.util.tags import Tags
 from sphinx.writers.html import HTMLTranslator, HTMLWriter
 
@@ -89,10 +90,13 @@ class Stylesheet(str):
 
     attributes = None   # type: Dict[str, str]
     filename = None     # type: str
+    priority = None     # type: int
 
-    def __new__(cls, filename: str, *args: str, **attributes: str) -> "Stylesheet":
+    def __new__(cls, filename: str, *args: str, priority: int = 500, **attributes: Any
-        self = str.__new__(cls, filename)  # type: ignore
+                ) -> "Stylesheet":
+        self = str.__new__(cls, filename)
         self.filename = filename
+        self.priority = priority
         self.attributes = attributes
         self.attributes.setdefault('rel', 'stylesheet')
         self.attributes.setdefault('type', 'text/css')
@@ -112,10 +116,12 @@ class JavaScript(str):
 
     attributes = None   # type: Dict[str, str]
     filename = None     # type: str
+    priority = None     # type: int
 
-    def __new__(cls, filename: str, **attributes: str) -> "JavaScript":
+    def __new__(cls, filename: str, priority: int = 500, **attributes: str) -> "JavaScript":
-        self = str.__new__(cls, filename)  # type: ignore
+        self = str.__new__(cls, filename)
         self.filename = filename
+        self.priority = priority
         self.attributes = attributes
 
         return self
@@ -289,29 +295,31 @@ class StandaloneHTMLBuilder(Builder):
             self.add_css_file(filename, **attrs)
 
         for filename, attrs in self.get_builder_config('css_files', 'html'):
+            attrs.setdefault('priority', 800)  # User's CSSs are loaded after extensions'
             self.add_css_file(filename, **attrs)
 
-    def add_css_file(self, filename: str, **kwargs: str) -> None:
+    def add_css_file(self, filename: str, **kwargs: Any) -> None:
         if '://' not in filename:
             filename = posixpath.join('_static', filename)
 
         self.css_files.append(Stylesheet(filename, **kwargs))  # type: ignore
 
     def init_js_files(self) -> None:
-        self.add_js_file('jquery.js')
+        self.add_js_file('jquery.js', priority=200)
-        self.add_js_file('underscore.js')
+        self.add_js_file('underscore.js', priority=200)
-        self.add_js_file('doctools.js')
+        self.add_js_file('doctools.js', priority=200)
 
         for filename, attrs in self.app.registry.js_files:
             self.add_js_file(filename, **attrs)
 
         for filename, attrs in self.get_builder_config('js_files', 'html'):
+            attrs.setdefault('priority', 800)  # User's JSs are loaded after extensions'
             self.add_js_file(filename, **attrs)
 
         if self.config.language and self._get_translations_js():
             self.add_js_file('translations.js')
 
-    def add_js_file(self, filename: str, **kwargs: str) -> None:
+    def add_js_file(self, filename: str, **kwargs: Any) -> None:
         if filename and '://' not in filename:
             filename = posixpath.join('_static', filename)
 
@@ -447,9 +455,6 @@ class StandaloneHTMLBuilder(Builder):
         logo = path.basename(self.config.html_logo) if self.config.html_logo else ''
         favicon = path.basename(self.config.html_favicon) if self.config.html_favicon else ''
 
-        if not isinstance(self.config.html_use_opensearch, str):
-            logger.warning(__('html_use_opensearch config value must now be a string'))
-
         self.relations = self.env.collect_relations()
 
         rellinks = []  # type: List[Tuple[str, str, str, str]]
@@ -461,6 +466,10 @@ class StandaloneHTMLBuilder(Builder):
                 rellinks.append((indexname, indexcls.localname,
                                  '', indexcls.shortname))
 
+        # back up script_files and css_files to allow adding JS/CSS files to a specific page.
+        self._script_files = list(self.script_files)
+        self._css_files = list(self.css_files)
+
         if self.config.html_style is not None:
             stylename = self.config.html_style
         elif self.theme:
@@ -1011,12 +1020,20 @@ class StandaloneHTMLBuilder(Builder):
         self.add_sidebars(pagename, ctx)
         ctx.update(addctx)
 
+        # revert script_files and css_files
+        self.script_files[:] = self._script_files
+        self.css_files[:] = self.css_files
+
         self.update_page_context(pagename, templatename, ctx, event_arg)
         newtmpl = self.app.emit_firstresult('html-page-context', pagename,
                                             templatename, ctx, event_arg)
         if newtmpl:
             templatename = newtmpl
 
+        # sort JS/CSS before rendering HTML
+        ctx['script_files'].sort(key=lambda js: js.priority)
+        ctx['css_files'].sort(key=lambda js: js.priority)
+
         try:
             output = self.templates.render(templatename, ctx)
         except UnicodeError:
@@ -1070,7 +1087,7 @@ class StandaloneHTMLBuilder(Builder):
             else:
                 with open(searchindexfn + '.tmp', 'wb') as fb:
                     self.indexer.dump(fb, self.indexer_format)
-            movefile(searchindexfn + '.tmp', searchindexfn)
+            os.replace(searchindexfn + '.tmp', searchindexfn)
 
 
 def convert_html_css_files(app: Sphinx, config: Config) -> None:
@@ -1188,6 +1205,16 @@ def validate_html_favicon(app: Sphinx, config: Config) -> None:
         config.html_favicon = None  # type: ignore
 
 
+def migrate_html_add_permalinks(app: Sphinx, config: Config) -> None:
+    """Migrate html_add_permalinks to html_permalinks*."""
+    if config.html_add_permalinks:
+        if (isinstance(config.html_add_permalinks, bool) and
+                config.html_add_permalinks is False):
+            config.html_permalinks = False  # type: ignore
+        else:
+            config.html_permalinks_icon = html.escape(config.html_add_permalinks)  # type: ignore  # NOQA
+
+
 # for compatibility
 import sphinxcontrib.serializinghtml  # NOQA
 
@@ -1218,7 +1245,9 @@ def setup(app: Sphinx) -> Dict[str, Any]:
     app.add_config_value('html_sidebars', {}, 'html')
     app.add_config_value('html_additional_pages', {}, 'html')
     app.add_config_value('html_domain_indices', True, 'html', [list])
-    app.add_config_value('html_add_permalinks', '¶', 'html')
+    app.add_config_value('html_add_permalinks', None, 'html')
+    app.add_config_value('html_permalinks', True, 'html')
+    app.add_config_value('html_permalinks_icon', '¶', 'html')
     app.add_config_value('html_use_index', True, 'html')
     app.add_config_value('html_split_index', False, 'html')
     app.add_config_value('html_copy_source', True, 'html')
@@ -1243,9 +1272,14 @@ def setup(app: Sphinx) -> Dict[str, Any]:
     app.add_config_value('html_math_renderer', None, 'env')
     app.add_config_value('html4_writer', False, 'html')
 
+    # events
+    app.add_event('html-collect-pages')
+    app.add_event('html-page-context')
+
     # event handlers
     app.connect('config-inited', convert_html_css_files, priority=800)
     app.connect('config-inited', convert_html_js_files, priority=800)
+    app.connect('config-inited', migrate_html_add_permalinks, priority=800)
     app.connect('config-inited', validate_html_extra_path, priority=800)
     app.connect('config-inited', validate_html_static_path, priority=800)
     app.connect('config-inited', validate_html_logo, priority=800)

sphinx/builders/html/transforms.py

@@ -4,12 +4,12 @@
 
     Transforms for HTML builder.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
 import re
-from typing import Any, Dict
+from typing import Any, Dict, List
 
 from docutils import nodes
 
@@ -28,7 +28,7 @@ class KeyboardTransform(SphinxPostTransform):
 
     After::
 
-        <literal class="kbd">
+        <literal class="kbd compound">
             <literal class="kbd">
                 Control
             -
@@ -37,18 +37,30 @@ class KeyboardTransform(SphinxPostTransform):
     """
     default_priority = 400
     builders = ('html',)
-    pattern = re.compile(r'(-|\+|\^|\s+)')
+    pattern = re.compile(r'(?<=.)(-|\+|\^|\s+)(?=.)')
+    multiwords_keys = (('caps', 'lock'),
+                       ('page' 'down'),
+                       ('page', 'up'),
+                       ('scroll' 'lock'),
+                       ('num', 'lock'),
+                       ('sys' 'rq'),
+                       ('back' 'space'))
 
     def run(self, **kwargs: Any) -> None:
         matcher = NodeMatcher(nodes.literal, classes=["kbd"])
         for node in self.document.traverse(matcher):  # type: nodes.literal
             parts = self.pattern.split(node[-1].astext())
-            if len(parts) == 1:
+            if len(parts) == 1 or self.is_multiwords_key(parts):
                 continue
 
+            node['classes'].append('compound')
             node.pop()
             while parts:
-                key = parts.pop(0)
+                if self.is_multiwords_key(parts):
+                    key = ''.join(parts[:3])
+                    parts[:3] = []
+                else:
+                    key = parts.pop(0)
                 node += nodes.literal('', key, classes=["kbd"])
 
                 try:
@@ -58,6 +70,16 @@ class KeyboardTransform(SphinxPostTransform):
                 except IndexError:
                     pass
 
+    def is_multiwords_key(self, parts: List[str]) -> bool:
+        if len(parts) >= 3 and parts[1].strip() == '':
+            name = parts[0].lower(), parts[2].lower()
+            if name in self.multiwords_keys:
+                return True
+            else:
+                return False
+        else:
+            return False
+
 
 def setup(app: Sphinx) -> Dict[str, Any]:
     app.add_post_transform(KeyboardTransform)

sphinx/builders/htmlhelp.py

@@ -5,7 +5,7 @@
     Build HTML help support files.
     Parts adapted from Python's Doc/tools/prechm.py.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/latex/__init__.py

@@ -4,7 +4,7 @@
 
     LaTeX builder.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/latex/constants.py

@@ -4,7 +4,7 @@
 
     consntants for LaTeX builder.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/latex/nodes.py

@@ -4,7 +4,7 @@
 
     Additional nodes for LaTeX writer.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/latex/theming.py

@@ -4,7 +4,7 @@
 
     Theming support for LaTeX builder.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/latex/transforms.py

@@ -4,7 +4,7 @@
 
     Transforms for LaTeX builder.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/latex/util.py

@@ -4,7 +4,7 @@
 
     Utilities for LaTeX builder.
 
-    :copyright: Copyright 2007-2018 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/linkcheck.py

@@ -4,7 +4,7 @@
 
     The CheckExternalLinksBuilder class.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -13,18 +13,25 @@ import queue
 import re
 import socket
 import threading
+import time
+import warnings
+from datetime import datetime, timezone
+from email.utils import parsedate_to_datetime
 from html.parser import HTMLParser
 from os import path
-from typing import Any, Dict, List, Set, Tuple
+from typing import Any, Dict, List, NamedTuple, Optional, Set, Tuple, cast
 from urllib.parse import unquote, urlparse
 
 from docutils import nodes
-from docutils.nodes import Node
+from docutils.nodes import Element
-from requests.exceptions import HTTPError
+from requests import Response
+from requests.exceptions import HTTPError, TooManyRedirects
 
 from sphinx.application import Sphinx
-from sphinx.builders import Builder
+from sphinx.builders.dummy import DummyBuilder
+from sphinx.deprecation import RemovedInSphinx50Warning
 from sphinx.locale import __
+from sphinx.transforms.post_transforms import SphinxPostTransform
 from sphinx.util import encode_uri, logging, requests
 from sphinx.util.console import darkgray, darkgreen, purple, red, turquoise  # type: ignore
 from sphinx.util.nodes import get_node_line
@@ -33,10 +40,28 @@ logger = logging.getLogger(__name__)
 
 uri_re = re.compile('([a-z]+:)?//')  # matches to foo:// and // (a protocol relative URL)
 
+Hyperlink = NamedTuple('Hyperlink', (('next_check', float),
+                                     ('uri', Optional[str]),
+                                     ('docname', Optional[str]),
+                                     ('lineno', Optional[int])))
+RateLimit = NamedTuple('RateLimit', (('delay', float), ('next_check', float)))
 
 DEFAULT_REQUEST_HEADERS = {
     'Accept': 'text/html,application/xhtml+xml;q=0.9,*/*;q=0.8',
 }
+CHECK_IMMEDIATELY = 0
+QUEUE_POLL_SECS = 1
+DEFAULT_DELAY = 60.0
+
+
+def node_line_or_0(node: Element) -> int:
+    """
+    PriorityQueue items must be comparable. The line number is part of the
+    tuple used by the PriorityQueue, keep an homogeneous type for comparison.
+    """
+    warnings.warn('node_line_or_0() is deprecated.',
+                  RemovedInSphinx50Warning, stacklevel=2)
+    return get_node_line(node) or 0
 
 
 class AnchorCheckParser(HTMLParser):
@@ -73,7 +98,7 @@ def check_anchor(response: requests.requests.Response, anchor: str) -> bool:
     return parser.found
 
 
-class CheckExternalLinksBuilder(Builder):
+class CheckExternalLinksBuilder(DummyBuilder):
     """
     Checks for broken external links.
     """
@@ -82,34 +107,59 @@ class CheckExternalLinksBuilder(Builder):
                 '%(outdir)s/output.txt')
 
     def init(self) -> None:
-        self.to_ignore = [re.compile(x) for x in self.app.config.linkcheck_ignore]
+        self.hyperlinks = {}    # type: Dict[str, Hyperlink]
+        self.to_ignore = [re.compile(x) for x in self.config.linkcheck_ignore]
         self.anchors_ignore = [re.compile(x)
-                               for x in self.app.config.linkcheck_anchors_ignore]
+                               for x in self.config.linkcheck_anchors_ignore]
         self.auth = [(re.compile(pattern), auth_info) for pattern, auth_info
-                     in self.app.config.linkcheck_auth]
+                     in self.config.linkcheck_auth]
-        self.good = set()       # type: Set[str]
+        self._good = set()       # type: Set[str]
-        self.broken = {}        # type: Dict[str, str]
+        self._broken = {}        # type: Dict[str, str]
-        self.redirected = {}    # type: Dict[str, Tuple[str, int]]
+        self._redirected = {}    # type: Dict[str, Tuple[str, int]]
         # set a timeout for non-responding servers
         socket.setdefaulttimeout(5.0)
-        # create output file
-        open(path.join(self.outdir, 'output.txt'), 'w').close()
-        # create JSON output file
-        open(path.join(self.outdir, 'output.json'), 'w').close()
 
         # create queues and worker threads
-        self.wqueue = queue.Queue()  # type: queue.Queue
+        self.rate_limits = {}  # type: Dict[str, RateLimit]
+        self.wqueue = queue.PriorityQueue()  # type: queue.PriorityQueue
         self.rqueue = queue.Queue()  # type: queue.Queue
         self.workers = []  # type: List[threading.Thread]
-        for i in range(self.app.config.linkcheck_workers):
+        for i in range(self.config.linkcheck_workers):
             thread = threading.Thread(target=self.check_thread, daemon=True)
             thread.start()
             self.workers.append(thread)
 
+    @property
+    def good(self) -> Set[str]:
+        warnings.warn(
+            "%s.%s is deprecated." % (self.__class__.__name__, "good"),
+            RemovedInSphinx50Warning,
+            stacklevel=2,
+        )
+        return self._good
+
+    @property
+    def broken(self) -> Dict[str, str]:
+        warnings.warn(
+            "%s.%s is deprecated." % (self.__class__.__name__, "broken"),
+            RemovedInSphinx50Warning,
+            stacklevel=2,
+        )
+        return self._broken
+
+    @property
+    def redirected(self) -> Dict[str, Tuple[str, int]]:
+        warnings.warn(
+            "%s.%s is deprecated." % (self.__class__.__name__, "redirected"),
+            RemovedInSphinx50Warning,
+            stacklevel=2,
+        )
+        return self._redirected
+
     def check_thread(self) -> None:
         kwargs = {}
-        if self.app.config.linkcheck_timeout:
+        if self.config.linkcheck_timeout:
-            kwargs['timeout'] = self.app.config.linkcheck_timeout
+            kwargs['timeout'] = self.config.linkcheck_timeout
 
         def get_request_headers() -> Dict:
             url = urlparse(uri)
@@ -155,9 +205,9 @@ class CheckExternalLinksBuilder(Builder):
             kwargs['headers'] = get_request_headers()
 
             try:
-                if anchor and self.app.config.linkcheck_anchors:
+                if anchor and self.config.linkcheck_anchors:
                     # Read the whole document and see if #anchor exists
-                    response = requests.get(req_url, stream=True, config=self.app.config,
+                    response = requests.get(req_url, stream=True, config=self.config,
                                             auth=auth_info, **kwargs)
                     response.raise_for_status()
                     found = check_anchor(response, unquote(anchor))
@@ -169,19 +219,28 @@ class CheckExternalLinksBuilder(Builder):
                         # try a HEAD request first, which should be easier on
                         # the server and the network
                         response = requests.head(req_url, allow_redirects=True,
-                                                 config=self.app.config, auth=auth_info,
+                                                 config=self.config, auth=auth_info,
                                                  **kwargs)
                         response.raise_for_status()
-                    except HTTPError:
+                    except (HTTPError, TooManyRedirects) as err:
+                        if isinstance(err, HTTPError) and err.response.status_code == 429:
+                            raise
                         # retry with GET request if that fails, some servers
                         # don't like HEAD requests.
-                        response = requests.get(req_url, stream=True, config=self.app.config,
+                        response = requests.get(req_url, stream=True,
+                                                config=self.config,
                                                 auth=auth_info, **kwargs)
                         response.raise_for_status()
             except HTTPError as err:
                 if err.response.status_code == 401:
                     # We'll take "Unauthorized" as working.
                     return 'working', ' - unauthorized', 0
+                elif err.response.status_code == 429:
+                    next_check = self.limit_rate(err.response)
+                    if next_check is not None:
+                        self.wqueue.put((next_check, uri, docname, lineno), False)
+                        return 'rate-limited', '', 0
+                    return 'broken', str(err), 0
                 elif err.response.status_code == 503:
                     # We'll take "Service Unavailable" as ignored.
                     return 'ignored', str(err), 0
@@ -189,6 +248,12 @@ class CheckExternalLinksBuilder(Builder):
                     return 'broken', str(err), 0
             except Exception as err:
                 return 'broken', str(err), 0
+            else:
+                netloc = urlparse(req_url).netloc
+                try:
+                    del self.rate_limits[netloc]
+                except KeyError:
+                    pass
             if response.url.rstrip('/') == req_url.rstrip('/'):
                 return 'working', '', 0
             else:
@@ -219,39 +284,97 @@ class CheckExternalLinksBuilder(Builder):
                             if rex.match(uri):
                                 return 'ignored', '', 0
                         else:
-                            self.broken[uri] = ''
+                            self._broken[uri] = ''
                             return 'broken', '', 0
-            elif uri in self.good:
+            elif uri in self._good:
                 return 'working', 'old', 0
-            elif uri in self.broken:
+            elif uri in self._broken:
-                return 'broken', self.broken[uri], 0
+                return 'broken', self._broken[uri], 0
-            elif uri in self.redirected:
+            elif uri in self._redirected:
-                return 'redirected', self.redirected[uri][0], self.redirected[uri][1]
+                return 'redirected', self._redirected[uri][0], self._redirected[uri][1]
             for rex in self.to_ignore:
                 if rex.match(uri):
                     return 'ignored', '', 0
 
             # need to actually check the URI
-            for _ in range(self.app.config.linkcheck_retries):
+            for _ in range(self.config.linkcheck_retries):
                 status, info, code = check_uri()
                 if status != "broken":
                     break
 
             if status == "working":
-                self.good.add(uri)
+                self._good.add(uri)
             elif status == "broken":
-                self.broken[uri] = info
+                self._broken[uri] = info
             elif status == "redirected":
-                self.redirected[uri] = (info, code)
+                self._redirected[uri] = (info, code)
 
             return (status, info, code)
 
         while True:
-            uri, docname, lineno = self.wqueue.get()
+            next_check, uri, docname, lineno = self.wqueue.get()
             if uri is None:
                 break
+            netloc = urlparse(uri).netloc
+            try:
+                # Refresh rate limit.
+                # When there are many links in the queue, workers are all stuck waiting
+                # for responses, but the builder keeps queuing. Links in the queue may
+                # have been queued before rate limits were discovered.
+                next_check = self.rate_limits[netloc].next_check
+            except KeyError:
+                pass
+            if next_check > time.time():
+                # Sleep before putting message back in the queue to avoid
+                # waking up other threads.
+                time.sleep(QUEUE_POLL_SECS)
+                self.wqueue.put((next_check, uri, docname, lineno), False)
+                self.wqueue.task_done()
+                continue
             status, info, code = check(docname)
-            self.rqueue.put((uri, docname, lineno, status, info, code))
+            if status == 'rate-limited':
+                logger.info(darkgray('-rate limited-   ') + uri + darkgray(' | sleeping...'))
+            else:
+                self.rqueue.put((uri, docname, lineno, status, info, code))
+            self.wqueue.task_done()
+
+    def limit_rate(self, response: Response) -> Optional[float]:
+        next_check = None
+        retry_after = response.headers.get("Retry-After")
+        if retry_after:
+            try:
+                # Integer: time to wait before next attempt.
+                delay = float(retry_after)
+            except ValueError:
+                try:
+                    # An HTTP-date: time of next attempt.
+                    until = parsedate_to_datetime(retry_after)
+                except (TypeError, ValueError):
+                    # TypeError: Invalid date format.
+                    # ValueError: Invalid date, e.g. Oct 52th.
+                    pass
+                else:
+                    next_check = datetime.timestamp(until)
+                    delay = (until - datetime.now(timezone.utc)).total_seconds()
+            else:
+                next_check = time.time() + delay
+        netloc = urlparse(response.url).netloc
+        if next_check is None:
+            max_delay = self.config.linkcheck_rate_limit_timeout
+            try:
+                rate_limit = self.rate_limits[netloc]
+            except KeyError:
+                delay = DEFAULT_DELAY
+            else:
+                last_wait_time = rate_limit.delay
+                delay = 2.0 * last_wait_time
+                if delay > max_delay and last_wait_time < max_delay:
+                    delay = max_delay
+            if delay > max_delay:
+                return None
+            next_check = time.time() + delay
+        self.rate_limits[netloc] = RateLimit(delay, next_check)
+        return next_check
 
     def process_result(self, result: Tuple[str, str, int, str, str, int]) -> None:
         uri, docname, lineno, status, info, code = result
@@ -305,62 +428,71 @@ class CheckExternalLinksBuilder(Builder):
             self.write_entry('redirected ' + text, docname, filename,
                              lineno, uri + ' to ' + info)
             self.write_linkstat(linkstat)
+        else:
+            raise ValueError("Unknown status %s." % status)
 
-    def get_target_uri(self, docname: str, typ: str = None) -> str:
+    def write_entry(self, what: str, docname: str, filename: str, line: int,
-        return ''
+                    uri: str) -> None:
+        self.txt_outfile.write("%s:%s: [%s] %s\n" % (filename, line, what, uri))
 
-    def get_outdated_docs(self) -> Set[str]:
+    def write_linkstat(self, data: dict) -> None:
-        return self.env.found_docs
+        self.json_outfile.write(json.dumps(data))
+        self.json_outfile.write('\n')
 
-    def prepare_writing(self, docnames: Set[str]) -> None:
+    def finish(self) -> None:
-        return
-
-    def write_doc(self, docname: str, doctree: Node) -> None:
         logger.info('')
-        n = 0
+
+        for hyperlink in self.hyperlinks.values():
+            self.wqueue.put(hyperlink, False)
+
+        total_links = len(self.hyperlinks)
+        done = 0
+        with open(path.join(self.outdir, 'output.txt'), 'w') as self.txt_outfile,\
+             open(path.join(self.outdir, 'output.json'), 'w') as self.json_outfile:
+            while done < total_links:
+                self.process_result(self.rqueue.get())
+                done += 1
+
+        if self._broken:
+            self.app.statuscode = 1
+
+        self.wqueue.join()
+        # Shutdown threads.
+        for worker in self.workers:
+            self.wqueue.put((CHECK_IMMEDIATELY, None, None, None), False)
+
+
+class HyperlinkCollector(SphinxPostTransform):
+    builders = ('linkcheck',)
+    default_priority = 800
+
+    def run(self, **kwargs: Any) -> None:
+        builder = cast(CheckExternalLinksBuilder, self.app.builder)
+        hyperlinks = builder.hyperlinks
 
         # reference nodes
-        for refnode in doctree.traverse(nodes.reference):
+        for refnode in self.document.traverse(nodes.reference):
             if 'refuri' not in refnode:
                 continue
             uri = refnode['refuri']
             lineno = get_node_line(refnode)
-            self.wqueue.put((uri, docname, lineno), False)
+            uri_info = Hyperlink(CHECK_IMMEDIATELY, uri, self.env.docname, lineno)
-            n += 1
+            if uri not in hyperlinks:
+                hyperlinks[uri] = uri_info
 
         # image nodes
-        for imgnode in doctree.traverse(nodes.image):
+        for imgnode in self.document.traverse(nodes.image):
             uri = imgnode['candidates'].get('?')
             if uri and '://' in uri:
                 lineno = get_node_line(imgnode)
-                self.wqueue.put((uri, docname, lineno), False)
+                uri_info = Hyperlink(CHECK_IMMEDIATELY, uri, self.env.docname, lineno)
-                n += 1
+                if uri not in hyperlinks:
-
+                    hyperlinks[uri] = uri_info
-        done = 0
-        while done < n:
-            self.process_result(self.rqueue.get())
-            done += 1
-
-        if self.broken:
-            self.app.statuscode = 1
-
-    def write_entry(self, what: str, docname: str, filename: str, line: int,
-                    uri: str) -> None:
-        with open(path.join(self.outdir, 'output.txt'), 'a') as output:
-            output.write("%s:%s: [%s] %s\n" % (filename, line, what, uri))
-
-    def write_linkstat(self, data: dict) -> None:
-        with open(path.join(self.outdir, 'output.json'), 'a') as output:
-            output.write(json.dumps(data))
-            output.write('\n')
-
-    def finish(self) -> None:
-        for worker in self.workers:
-            self.wqueue.put((None, None, None), False)
 
 
 def setup(app: Sphinx) -> Dict[str, Any]:
     app.add_builder(CheckExternalLinksBuilder)
+    app.add_post_transform(HyperlinkCollector)
 
     app.add_config_value('linkcheck_ignore', [], None)
     app.add_config_value('linkcheck_auth', [], None)
@@ -372,6 +504,7 @@ def setup(app: Sphinx) -> Dict[str, Any]:
     # Anchors starting with ! are ignored since they are
     # commonly used for dynamic pages
     app.add_config_value('linkcheck_anchors_ignore', ["^!"], None)
+    app.add_config_value('linkcheck_rate_limit_timeout', 300.0, None)
 
     return {
         'version': 'builtin',

sphinx/builders/manpage.py

@@ -4,7 +4,7 @@
 
     Manual pages builder.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/qthelp.py

@@ -4,7 +4,7 @@
 
     Build input files for the Qt collection generator.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/singlehtml.py

@@ -4,7 +4,7 @@
 
     Single HTML builders.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/texinfo.py

@@ -4,7 +4,7 @@
 
     Texinfo builder.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -179,7 +179,8 @@ class TexinfoBuilder(Builder):
                 try:
                     imagedir = path.join(self.outdir, targetname + '-figures')
                     ensuredir(imagedir)
-                    copy_asset_file(path.join(self.srcdir, dest), imagedir)
+                    copy_asset_file(path.join(self.srcdir, src),
+                                    path.join(imagedir, dest))
                 except Exception as err:
                     logger.warning(__('cannot copy image file %r: %s'),
                                    path.join(self.srcdir, src), err)

sphinx/builders/text.py

@@ -4,7 +4,7 @@
 
     Plain-text Sphinx builder.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/builders/xml.py

@@ -4,7 +4,7 @@
 
     Docutils-native XML and pseudo-XML builders.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/cmd/__init__.py

@@ -4,6 +4,6 @@
 
     Modules for command line executables.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """

sphinx/cmd/build.py

@@ -4,7 +4,7 @@
 
     Build documentation from a provided source.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/cmd/make_mode.py

@@ -10,7 +10,7 @@
     This is in its own module so that importing it is fast.  It should not
     import the main Sphinx modules (like sphinx.applications, sphinx.builders).
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/cmd/quickstart.py

@@ -4,7 +4,7 @@
 
     Quickly setup documentation source to work with Sphinx.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -29,6 +29,7 @@ try:
         readline.parse_and_bind("tab: complete")
         USE_LIBEDIT = False
 except ImportError:
+    readline = None
     USE_LIBEDIT = False
 
 from docutils.utils import column_width
@@ -169,8 +170,11 @@ def do_prompt(text: str, default: str = None, validator: Callable[[str], Any] =
             # sequence (see #5335).  To avoid the problem, all prompts are not colored
             # on libedit.
             pass
-        else:
+        elif readline:
+            # pass input_mode=True if readline available
             prompt = colorize(COLOR_QUESTION, prompt, input_mode=True)
+        else:
+            prompt = colorize(COLOR_QUESTION, prompt, input_mode=False)
         x = term_input(prompt).strip()
         if default and not x:
             x = default

sphinx/config.py

@@ -4,7 +4,7 @@
 
     Build configuration file handling.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -98,7 +98,8 @@ class Config:
         # general options
         'project': ('Python', 'env', []),
         'author': ('unknown', 'env', []),
-        'copyright': ('', 'html', []),
+        'project_copyright': ('', 'html', [str]),
+        'copyright': (lambda c: c.project_copyright, 'html', [str]),
         'version': ('', 'env', []),
         'release': ('', 'env', []),
         'today': ('', 'env', []),
@@ -180,6 +181,14 @@ class Config:
             defvalue = self.values[name][0]
             if self.values[name][2] == Any:
                 return value
+            elif self.values[name][2] == {bool, str}:
+                if value == '0':
+                    # given falsy string from command line option
+                    return False
+                elif value == '1':
+                    return True
+                else:
+                    return value
             elif type(defvalue) is bool or self.values[name][2] == [bool]:
                 if value == '0':
                     # given falsy string from command line option
@@ -358,6 +367,18 @@ def convert_source_suffix(app: "Sphinx", config: Config) -> None:
                           "But `%r' is given." % source_suffix))
 
 
+def convert_highlight_options(app: "Sphinx", config: Config) -> None:
+    """Convert old styled highlight_options to new styled one.
+
+    * old style: options
+    * new style: dict that maps language names to options
+    """
+    options = config.highlight_options
+    if options and not all(isinstance(v, dict) for v in options.values()):
+        # old styled option detected because all values are not dictionary.
+        config.highlight_options = {config.highlight_language: options}  # type: ignore
+
+
 def init_numfig_format(app: "Sphinx", config: Config) -> None:
     """Initialize :confval:`numfig_format`."""
     numfig_format = {'section': _('Section %s'),
@@ -478,6 +499,7 @@ def check_master_doc(app: "Sphinx", env: "BuildEnvironment", added: Set[str],
 
 def setup(app: "Sphinx") -> Dict[str, Any]:
     app.connect('config-inited', convert_source_suffix, priority=800)
+    app.connect('config-inited', convert_highlight_options, priority=800)
     app.connect('config-inited', init_numfig_format, priority=800)
     app.connect('config-inited', correct_copyright_year, priority=800)
     app.connect('config-inited', check_confval_types, priority=800)

sphinx/deprecation.py

@@ -4,7 +4,7 @@
 
     Sphinx deprecation classes and utilities.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -26,6 +26,10 @@ class RemovedInSphinx50Warning(PendingDeprecationWarning):
     pass
 
 
+class RemovedInSphinx60Warning(PendingDeprecationWarning):
+    pass
+
+
 RemovedInNextVersionWarning = RemovedInSphinx40Warning
 
 

sphinx/directives/__init__.py

@@ -4,12 +4,12 @@
 
     Handlers for additional ReST directives.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
 import re
-from typing import Any, Dict, List, Tuple, cast
+from typing import Any, Dict, Generic, List, Tuple, TypeVar, cast
 
 from docutils import nodes
 from docutils.nodes import Node
@@ -33,6 +33,8 @@ if False:
 nl_escape_re = re.compile(r'\\\n')
 strip_backslash_re = re.compile(r'\\(.)')
 
+T = TypeVar('T')
+
 
 def optional_int(argument: str) -> int:
     """
@@ -47,7 +49,7 @@ def optional_int(argument: str) -> int:
         return value
 
 
-class ObjectDescription(SphinxDirective):
+class ObjectDescription(SphinxDirective, Generic[T]):
     """
     Directive to describe a class, function or similar object.  Not used
     directly, but subclassed (in domain-specific directives) to add custom
@@ -97,7 +99,7 @@ class ObjectDescription(SphinxDirective):
         else:
             return [line.strip() for line in lines]
 
-    def handle_signature(self, sig: str, signode: desc_signature) -> Any:
+    def handle_signature(self, sig: str, signode: desc_signature) -> T:
         """
         Parse the signature *sig* into individual nodes and append them to
         *signode*. If ValueError is raised, parsing is aborted and the whole
@@ -109,7 +111,7 @@ class ObjectDescription(SphinxDirective):
         """
         raise ValueError
 
-    def add_target_and_index(self, name: Any, sig: str, signode: desc_signature) -> None:
+    def add_target_and_index(self, name: T, sig: str, signode: desc_signature) -> None:
         """
         Add cross-reference IDs and entries to self.indexnode, if applicable.
 
@@ -173,7 +175,7 @@ class ObjectDescription(SphinxDirective):
         if self.domain:
             node['classes'].append(self.domain)
 
-        self.names = []  # type: List[Any]
+        self.names = []  # type: List[T]
         signatures = self.get_signatures()
         for i, sig in enumerate(signatures):
             # add a signature node for each signature in the current unit

sphinx/directives/code.py

@@ -2,11 +2,12 @@
     sphinx.directives.code
     ~~~~~~~~~~~~~~~~~~~~~~
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
 import sys
+import textwrap
 import warnings
 from difflib import unified_diff
 from typing import Any, Dict, List, Tuple
@@ -19,6 +20,7 @@ from docutils.statemachine import StringList
 from sphinx import addnodes
 from sphinx.config import Config
 from sphinx.deprecation import RemovedInSphinx40Warning
+from sphinx.directives import optional_int
 from sphinx.locale import __
 from sphinx.util import logging, parselinenos
 from sphinx.util.docutils import SphinxDirective
@@ -68,7 +70,7 @@ class HighlightLang(Highlight):
 
 def dedent_lines(lines: List[str], dedent: int, location: Tuple[str, int] = None) -> List[str]:
     if not dedent:
-        return lines
+        return textwrap.dedent(''.join(lines)).splitlines(True)
 
     if any(s[:dedent].strip() for s in lines):
         logger.warning(__('non-whitespace stripped by dedent'), location=location)
@@ -117,7 +119,7 @@ class CodeBlock(SphinxDirective):
     option_spec = {
         'force': directives.flag,
         'linenos': directives.flag,
-        'dedent': int,
+        'dedent': optional_int,
         'lineno-start': int,
         'emphasize-lines': directives.unchanged_required,
         'caption': directives.unchanged_required,
@@ -391,7 +393,7 @@ class LiteralInclude(SphinxDirective):
     optional_arguments = 0
     final_argument_whitespace = True
     option_spec = {
-        'dedent': int,
+        'dedent': optional_int,
         'linenos': directives.flag,
         'lineno-start': int,
         'lineno-match': directives.flag,

sphinx/directives/other.py

@@ -2,7 +2,7 @@
     sphinx.directives.other
     ~~~~~~~~~~~~~~~~~~~~~~~
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -276,6 +276,7 @@ class HList(SphinxDirective):
         npercol, nmore = divmod(len(fulllist), ncolumns)
         index = 0
         newnode = addnodes.hlist()
+        newnode['ncolumns'] = str(ncolumns)
         for column in range(ncolumns):
             endindex = index + ((npercol + 1) if column < nmore else npercol)
             bullet_list = nodes.bullet_list()

sphinx/directives/patches.py

@@ -2,7 +2,7 @@
     sphinx.directives.patches
     ~~~~~~~~~~~~~~~~~~~~~~~~~
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/domains/__init__.py

@@ -5,7 +5,7 @@
     Support for domains, which are groupings of description directives
     and roles describing e.g. constructs of one programming language.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/domains/c.py

@@ -4,7 +4,7 @@
 
     The C language domain.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -137,8 +137,7 @@ class ASTIdentifier(ASTBaseBase):
                                           reftype='identifier',
                                           reftarget=targetText, modname=None,
                                           classname=None)
-            key = symbol.get_lookup_key()
+            pnode['c:parent_key'] = symbol.get_lookup_key()
-            pnode['c:parent_key'] = key
             if self.is_anon():
                 pnode += nodes.strong(text="[anonymous]")
             else:
@@ -1582,13 +1581,11 @@ class Symbol:
     def get_all_symbols(self) -> Iterator["Symbol"]:
         yield self
         for sChild in self._children:
-            for s in sChild.get_all_symbols():
+            yield from sChild.get_all_symbols()
-                yield s
 
     @property
     def children(self) -> Iterator["Symbol"]:
-        for c in self._children:
+        yield from self._children
-            yield c
 
     @property
     def children_recurse_anon(self) -> Iterator["Symbol"]:
@@ -3101,7 +3098,7 @@ def _make_phony_error_name() -> ASTNestedName:
     return ASTNestedName([ASTIdentifier("PhonyNameDueToError")], rooted=False)
 
 
-class CObject(ObjectDescription):
+class CObject(ObjectDescription[ASTDeclaration]):
     """
     Description of a C language object.
     """
@@ -3206,7 +3203,8 @@ class CObject(ObjectDescription):
     def parse_pre_v3_type_definition(self, parser: DefinitionParser) -> ASTDeclaration:
         return parser.parse_pre_v3_type_definition()
 
-    def describe_signature(self, signode: TextElement, ast: Any, options: Dict) -> None:
+    def describe_signature(self, signode: TextElement, ast: ASTDeclaration,
+                           options: Dict) -> None:
         ast.describe_signature(signode, 'lastIsName', self.env, options)
 
     def run(self) -> List[Node]:
@@ -3453,8 +3451,9 @@ class AliasNode(nodes.Element):
             assert parentKey is not None
             self.parentKey = parentKey
 
-    def copy(self: T) -> T:
+    def copy(self) -> 'AliasNode':
-        return self.__class__(self.sig, env=None, parentKey=self.parentKey)  # type: ignore
+        return self.__class__(self.sig, self.maxdepth, self.document,
+                              env=None, parentKey=self.parentKey)
 
 
 class AliasTransform(SphinxTransform):
@@ -3643,7 +3642,7 @@ class CExprRole(SphinxRole):
                            location=self.get_source_info())
             # see below
             return [self.node_type(text, text, classes=classes)], []
-        parentSymbol = self.env.temp_data.get('cpp:parent_symbol', None)
+        parentSymbol = self.env.temp_data.get('c:parent_symbol', None)
         if parentSymbol is None:
             parentSymbol = self.env.domaindata['c']['root_symbol']
         # ...most if not all of these classes should really apply to the individual references,
@@ -3658,15 +3657,18 @@ class CDomain(Domain):
     name = 'c'
     label = 'C'
     object_types = {
-        'function': ObjType(_('function'), 'func'),
+        # 'identifier' is the one used for xrefs generated in signatures, not in roles
-        'member': ObjType(_('member'), 'member'),
+        'member': ObjType(_('member'), 'var', 'member', 'data', 'identifier'),
-        'macro': ObjType(_('macro'), 'macro'),
+        'var': ObjType(_('variable'),  'var', 'member', 'data', 'identifier'),
-        'type': ObjType(_('type'), 'type'),
+        'function': ObjType(_('function'),     'func',          'identifier', 'type'),
-        'var': ObjType(_('variable'), 'data'),
+        'macro': ObjType(_('macro'),           'macro',         'identifier'),
-        'enum': ObjType(_('enum'), 'enum'),
+        'struct': ObjType(_('struct'),         'struct',        'identifier', 'type'),
-        'enumerator': ObjType(_('enumerator'), 'enumerator'),
+        'union': ObjType(_('union'),           'union',         'identifier', 'type'),
-        'struct': ObjType(_('struct'), 'struct'),
+        'enum': ObjType(_('enum'),             'enum',          'identifier', 'type'),
-        'union': ObjType(_('union'), 'union'),
+        'enumerator': ObjType(_('enumerator'), 'enumerator',    'identifier'),
+        'type': ObjType(_('type'),                              'identifier', 'type'),
+        # generated object types
+        'functionParam': ObjType(_('function parameter'),       'identifier', 'var', 'member', 'data'),  # noqa
     }
 
     directives = {

sphinx/domains/changeset.py

@@ -4,7 +4,7 @@
 
     The changeset domain.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/domains/citation.py

@@ -4,7 +4,7 @@
 
     The citation domain.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/domains/cpp.py

@@ -4,7 +4,7 @@
 
     The C++ language domain.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -1592,6 +1592,15 @@ class ASTOperator(ASTBase):
         identifier = str(self)
         if mode == 'lastIsName':
             signode += addnodes.desc_name(identifier, identifier)
+        elif mode == 'markType':
+            targetText = prefix + identifier + templateArgs
+            pnode = addnodes.pending_xref('', refdomain='cpp',
+                                          reftype='identifier',
+                                          reftarget=targetText, modname=None,
+                                          classname=None)
+            pnode['cpp:parent_key'] = symbol.get_lookup_key()
+            pnode += nodes.Text(identifier)
+            signode += pnode
         else:
             signode += addnodes.desc_addname(identifier, identifier)
 
@@ -3927,8 +3936,7 @@ class Symbol:
     def get_all_symbols(self) -> Iterator[Any]:
         yield self
         for sChild in self._children:
-            for s in sChild.get_all_symbols():
+            yield from sChild.get_all_symbols()
-                yield s
 
     @property
     def children_recurse_anon(self) -> Generator["Symbol", None, None]:
@@ -6671,7 +6679,7 @@ def _make_phony_error_name() -> ASTNestedName:
     return ASTNestedName([nne], [False], rooted=False)
 
 
-class CPPObject(ObjectDescription):
+class CPPObject(ObjectDescription[ASTDeclaration]):
     """Description of a C++ language object."""
 
     doc_field_types = [
@@ -7051,8 +7059,8 @@ class AliasNode(nodes.Element):
             assert parentKey is not None
             self.parentKey = parentKey
 
-    def copy(self: T) -> T:
+    def copy(self) -> 'AliasNode':
-        return self.__class__(self.sig, env=None, parentKey=self.parentKey)  # type: ignore
+        return self.__class__(self.sig, env=None, parentKey=self.parentKey)
 
 
 class AliasTransform(SphinxTransform):
@@ -7252,14 +7260,18 @@ class CPPDomain(Domain):
     name = 'cpp'
     label = 'C++'
     object_types = {
-        'class':      ObjType(_('class'),      'class',             'type', 'identifier'),
+        'class':      ObjType(_('class'),      'class', 'struct',   'identifier', 'type'),
-        'union':      ObjType(_('union'),      'union',             'type', 'identifier'),
+        'union':      ObjType(_('union'),      'union',             'identifier', 'type'),
-        'function':   ObjType(_('function'),   'function',  'func', 'type', 'identifier'),
+        'function':   ObjType(_('function'),   'func',              'identifier', 'type'),
-        'member':     ObjType(_('member'),     'member',    'var'),
+        'member':     ObjType(_('member'),     'member', 'var',     'identifier'),
-        'type':       ObjType(_('type'),                            'type', 'identifier'),
+        'type':       ObjType(_('type'),                            'identifier', 'type'),
-        'concept':    ObjType(_('concept'),    'concept',                   'identifier'),
+        'concept':    ObjType(_('concept'),    'concept',           'identifier'),
-        'enum':       ObjType(_('enum'),       'enum',              'type', 'identifier'),
+        'enum':       ObjType(_('enum'),       'enum',              'identifier', 'type'),
-        'enumerator': ObjType(_('enumerator'), 'enumerator')
+        'enumerator': ObjType(_('enumerator'), 'enumerator',        'identifier'),
+        # generated object types
+        'functionParam': ObjType(_('function parameter'),           'identifier', 'member', 'var'),  # noqa
+        'templateParam': ObjType(_('template parameter'),
+                                 'identifier', 'class', 'struct', 'union', 'member', 'var', 'type'),  # noqa
     }
 
     directives = {
@@ -7436,30 +7448,19 @@ class CPPDomain(Domain):
 
         if typ.startswith('cpp:'):
             typ = typ[4:]
-        origTyp = typ
-        if typ == 'func':
-            typ = 'function'
-        if typ == 'struct':
-            typ = 'class'
         declTyp = s.declaration.objectType
 
         def checkType() -> bool:
-            if typ == 'any' or typ == 'identifier':
+            if typ == 'any':
                 return True
-            if declTyp == 'templateParam':
-                # TODO: perhaps this should be strengthened one day
-                return True
-            if declTyp == 'functionParam':
-                if typ == 'var' or typ == 'member':
-                    return True
             objtypes = self.objtypes_for_role(typ)
             if objtypes:
                 return declTyp in objtypes
-            print("Type is %s (originally: %s), declType is %s" % (typ, origTyp, declTyp))
+            print("Type is %s, declaration type is %s" % (typ, declTyp))
             assert False
         if not checkType():
             logger.warning("cpp:%s targets a %s (%s).",
-                           origTyp, s.declaration.objectType,
+                           typ, s.declaration.objectType,
                            s.get_full_nested_name(),
                            location=node)
 
@@ -7489,10 +7490,10 @@ class CPPDomain(Domain):
                     if env.config.add_function_parentheses and typ == 'any':
                         addParen += 1
                     # and now this stuff for operator()
-                    if (env.config.add_function_parentheses and typ == 'function' and
+                    if (env.config.add_function_parentheses and typ == 'func' and
                             title.endswith('operator()')):
                         addParen += 1
-                    if ((typ == 'any' or typ == 'function') and
+                    if ((typ == 'any' or typ == 'func') and
                             title.endswith('operator') and
                             displayName.endswith('operator()')):
                         addParen += 1
@@ -7501,7 +7502,7 @@ class CPPDomain(Domain):
                     if env.config.add_function_parentheses:
                         if typ == 'any' and displayName.endswith('()'):
                             addParen += 1
-                        elif typ == 'function':
+                        elif typ == 'func':
                             if title.endswith('()') and not displayName.endswith('()'):
                                 title = title[:-2]
                     else:

sphinx/domains/index.py

@@ -4,7 +4,7 @@
 
     The index domain.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/domains/javascript.py

@@ -4,7 +4,7 @@
 
     The JavaScript domain.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -32,7 +32,7 @@ from sphinx.util.nodes import make_id, make_refnode
 logger = logging.getLogger(__name__)
 
 
-class JSObject(ObjectDescription):
+class JSObject(ObjectDescription[Tuple[str, str]]):
     """
     Description of a JavaScript object.
     """

sphinx/domains/math.py

@@ -4,7 +4,7 @@
 
     The math domain.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -157,8 +157,11 @@ class MathDomain(Domain):
         targets = [eq for eq in self.equations.values() if eq[0] == docname]
         return len(targets) + 1
 
-    def has_equations(self) -> bool:
+    def has_equations(self, docname: str = None) -> bool:
-        return any(self.data['has_equations'].values())
+        if docname:
+            return self.data['has_equations'].get(docname, False)
+        else:
+            return any(self.data['has_equations'].values())
 
 
 def setup(app: "Sphinx") -> Dict[str, Any]:

sphinx/domains/python.py

@@ -4,7 +4,7 @@
 
     The Python domain.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -272,6 +272,8 @@ class PyXrefMixin:
         result = super().make_xref(rolename, domain, target,  # type: ignore
                                    innernode, contnode, env)
         result['refspecific'] = True
+        result['py:module'] = env.ref_context.get('py:module')
+        result['py:class'] = env.ref_context.get('py:class')
         if target.startswith(('.', '~')):
             prefix, result['reftarget'] = target[0], target[1:]
             if prefix == '.':
@@ -332,7 +334,7 @@ class PyTypedField(PyXrefMixin, TypedField):
         return super().make_xref(rolename, domain, target, innernode, contnode, env)
 
 
-class PyObject(ObjectDescription):
+class PyObject(ObjectDescription[Tuple[str, str]]):
     """
     Description of a general Python object.
 

sphinx/domains/rst.py

@@ -4,7 +4,7 @@
 
     The reStructuredText domain.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -31,7 +31,7 @@ logger = logging.getLogger(__name__)
 dir_sig_re = re.compile(r'\.\. (.+?)::(.*)$')
 
 
-class ReSTMarkup(ObjectDescription):
+class ReSTMarkup(ObjectDescription[str]):
     """
     Description of generic reST markup.
     """

sphinx/domains/std.py

@@ -4,7 +4,7 @@
 
     The standard domain.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -43,12 +43,12 @@ logger = logging.getLogger(__name__)
 
 
 # RE for option descriptions
-option_desc_re = re.compile(r'((?:/|--|-|\+)?[^\s=[]+)(=?\s*.*)')
+option_desc_re = re.compile(r'((?:/|--|-|\+)?[^\s=]+)(=?\s*.*)')
 # RE for grammar tokens
-token_re = re.compile(r'`(\w+)`', re.U)
+token_re = re.compile(r'`((~?\w*:)?\w+)`', re.U)
 
 
-class GenericObject(ObjectDescription):
+class GenericObject(ObjectDescription[str]):
     """
     A generic x-ref directive registered with Sphinx.add_object_type().
     """
@@ -178,7 +178,7 @@ class Target(SphinxDirective):
         return self.name + '-' + name
 
 
-class Cmdoption(ObjectDescription):
+class Cmdoption(ObjectDescription[str]):
     """
     Description of a command-line option (.. option).
     """
@@ -197,6 +197,11 @@ class Cmdoption(ObjectDescription):
                                location=signode)
                 continue
             optname, args = m.groups()
+            if optname.endswith('[') and args.endswith(']'):
+                # optional value surrounded by brackets (ex. foo[=bar])
+                optname = optname[:-1]
+                args = '[' + args
+
             if count:
                 signode += addnodes.desc_addname(', ', ', ')
             signode += addnodes.desc_name(optname, optname)
@@ -318,7 +323,7 @@ def make_glossary_term(env: "BuildEnvironment", textnodes: Iterable[Node], index
         term['ids'].append(node_id)
 
     std = cast(StandardDomain, env.get_domain('std'))
-    std.note_object('term', termtext, node_id, location=term)
+    std._note_term(termtext, node_id, location=term)
 
     # add an index entry too
     indexnode = addnodes.index()
@@ -459,9 +464,23 @@ def token_xrefs(text: str, productionGroup: str = '') -> List[Node]:
         if m.start() > pos:
             txt = text[pos:m.start()]
             retnodes.append(nodes.Text(txt, txt))
-        refnode = pending_xref(m.group(1), reftype='token', refdomain='std',
+        token = m.group(1)
-                               reftarget=productionGroup + m.group(1))
+        if ':' in token:
-        refnode += nodes.literal(m.group(1), m.group(1), classes=['xref'])
+            if token[0] == '~':
+                _, title = token.split(':')
+                target = token[1:]
+            elif token[0] == ':':
+                title = token[1:]
+                target = title
+            else:
+                title = token
+                target = token
+        else:
+            title = token
+            target = productionGroup + token
+        refnode = pending_xref(title, reftype='token', refdomain='std',
+                               reftarget=target)
+        refnode += nodes.literal(token, title, classes=['xref'])
         retnodes.append(refnode)
         pos = m.end()
     if pos < len(text):
@@ -675,6 +694,20 @@ class StandardDomain(Domain):
                       RemovedInSphinx50Warning, stacklevel=2)
         self.objects[objtype, name] = (docname, labelid)
 
+    @property
+    def _terms(self) -> Dict[str, Tuple[str, str]]:
+        """.. note:: Will be removed soon. internal use only."""
+        return self.data.setdefault('terms', {})  # (name) -> docname, labelid
+
+    def _note_term(self, term: str, labelid: str, location: Any = None) -> None:
+        """Note a term for cross reference.
+
+        .. note:: Will be removed soon. internal use only.
+        """
+        self.note_object('term', term, labelid, location)
+
+        self._terms[term.lower()] = (self.env.docname, labelid)
+
     @property
     def progoptions(self) -> Dict[Tuple[str, str], Tuple[str, str]]:
         return self.data.setdefault('progoptions', {})  # (program, name) -> docname, labelid
@@ -695,6 +728,9 @@ class StandardDomain(Domain):
         for key, (fn, _l) in list(self.objects.items()):
             if fn == docname:
                 del self.objects[key]
+        for key, (fn, _l) in list(self._terms.items()):
+            if fn == docname:
+                del self._terms[key]
         for key, (fn, _l, _l) in list(self.labels.items()):
             if fn == docname:
                 del self.labels[key]
@@ -710,6 +746,9 @@ class StandardDomain(Domain):
         for key, data in otherdata['objects'].items():
             if data[0] in docnames:
                 self.objects[key] = data
+        for key, data in otherdata['terms'].items():
+            if data[0] in docnames:
+                self._terms[key] = data
         for key, data in otherdata['labels'].items():
             if data[0] in docnames:
                 self.labels[key] = data
@@ -740,9 +779,11 @@ class StandardDomain(Domain):
                                name, env.doc2path(self.labels[name][0]),
                                location=node)
             self.anonlabels[name] = docname, labelid
-            if node.tagname in ('section', 'rubric'):
+            if node.tagname == 'section':
                 title = cast(nodes.title, node[0])
                 sectname = clean_astext(title)
+            elif node.tagname == 'rubric':
+                sectname = clean_astext(node)
             elif self.is_enumerable_node(node):
                 sectname = self.get_numfig_title(node)
                 if not sectname:
@@ -852,8 +893,9 @@ class StandardDomain(Domain):
             if fignumber is None:
                 return contnode
         except ValueError:
-            logger.warning(__("no number is assigned for %s: %s"), figtype, labelid,
+            logger.warning(__("Failed to create a cross reference. Any number is not "
-                           location=node)
+                              "assigned: %s"),
+                           labelid, location=node)
             return contnode
 
         try:
@@ -945,19 +987,12 @@ class StandardDomain(Domain):
         if result:
             return result
         else:
-            for objtype, term in self.objects:
+            # fallback to case insentive match
-                if objtype == 'term' and term.lower() == target.lower():
+            if target.lower() in self._terms:
-                    docname, labelid = self.objects[objtype, term]
+                docname, labelid = self._terms[target.lower()]
-                    logger.warning(__('term %s not found in case sensitive match.'
+                return make_refnode(builder, fromdocname, docname, labelid, contnode)
-                                      'made a reference to %s instead.'),
-                                   target, term, location=node, type='ref', subtype='term')
-                    break
             else:
-                docname, labelid = '', ''
-            if not docname:
                 return None
-            return make_refnode(builder, fromdocname, docname,
-                                labelid, contnode)
 
     def _resolve_obj_xref(self, env: "BuildEnvironment", fromdocname: str,
                           builder: "Builder", typ: str, target: str,
@@ -1106,7 +1141,7 @@ class StandardDomain(Domain):
 
 
 def warn_missing_reference(app: "Sphinx", domain: Domain, node: pending_xref) -> bool:
-    if domain.name != 'std' or node['reftype'] != 'ref':
+    if (domain and domain.name != 'std') or node['reftype'] != 'ref':
         return None
     else:
         target = node['reftarget']
@@ -1125,7 +1160,7 @@ def setup(app: "Sphinx") -> Dict[str, Any]:
 
     return {
         'version': 'builtin',
-        'env_version': 1,
+        'env_version': 2,
         'parallel_read_safe': True,
         'parallel_write_safe': True,
     }

sphinx/environment/__init__.py

@@ -4,12 +4,13 @@
 
     Global creation environment.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
 import os
 import pickle
+import posixpath
 import warnings
 from collections import defaultdict
 from copy import copy
@@ -356,9 +357,9 @@ class BuildEnvironment:
             docdir = path.dirname(self.doc2path(docname or self.docname,
                                                 base=None))
             rel_fn = path.join(docdir, filename)
-        # the path.abspath() might seem redundant, but otherwise artifacts
+
-        # such as ".." will remain in the path
+        return (posixpath.normpath(rel_fn),
-        return rel_fn, path.abspath(path.join(self.srcdir, rel_fn))
+                path.normpath(path.join(self.srcdir, rel_fn)))
 
     @property
     def found_docs(self) -> Set[str]:

sphinx/environment/adapters/__init__.py

@@ -4,6 +4,6 @@
 
     Sphinx environment adapters
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """

sphinx/environment/adapters/asset.py

@@ -4,7 +4,7 @@
 
     Assets adapter for sphinx.environment.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/environment/adapters/indexentries.py

@@ -4,7 +4,7 @@
 
     Index entries adapters for sphinx.environment.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/environment/adapters/toctree.py

@@ -4,7 +4,7 @@
 
     Toctree adapter for sphinx.environment.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -320,8 +320,10 @@ class TocTree:
         toctrees = []  # type: List[Element]
         if 'includehidden' not in kwargs:
             kwargs['includehidden'] = True
-        if 'maxdepth' not in kwargs:
+        if 'maxdepth' not in kwargs or not kwargs['maxdepth']:
             kwargs['maxdepth'] = 0
+        else:
+            kwargs['maxdepth'] = int(kwargs['maxdepth'])
         kwargs['collapse'] = collapse
         for toctreenode in doctree.traverse(addnodes.toctree):
             toctree = self.resolve(docname, builder, toctreenode, prune=True, **kwargs)

sphinx/environment/collectors/__init__.py

@@ -4,7 +4,7 @@
 
     The data collector components for sphinx.environment.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/environment/collectors/asset.py

@@ -4,7 +4,7 @@
 
     The image collector for sphinx.environment.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/environment/collectors/dependencies.py

@@ -4,7 +4,7 @@
 
     The dependencies collector components for sphinx.environment.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/environment/collectors/indexentries.py

@@ -4,7 +4,7 @@
 
     Index entries collector for sphinx.environment.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/environment/collectors/metadata.py

@@ -4,7 +4,7 @@
 
     The metadata collector components for sphinx.environment.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/environment/collectors/title.py

@@ -4,7 +4,7 @@
 
     The title collector components for sphinx.environment.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/environment/collectors/toctree.py

@@ -4,7 +4,7 @@
 
     Toctree collector for sphinx.environment.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/errors.py

@@ -5,7 +5,7 @@
     Contains SphinxError and a few subclasses (in an extra module to avoid
     circular import problems).
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 

sphinx/events.py

@@ -6,7 +6,7 @@
 
     Gracefully adapted from the TextPress system by Armin.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -50,8 +50,6 @@ core_events = {
     'warn-missing-reference': 'domain, node',
     'doctree-resolved': 'doctree, docname',
     'env-updated': 'env',
-    'html-collect-pages': 'builder',
-    'html-page-context': 'pagename, context, doctree or None',
     'build-finished': 'exception',
 }
 

sphinx/ext/__init__.py

@@ -4,6 +4,6 @@
 
     Contains Sphinx features not activated by default.
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """

sphinx/ext/apidoc.py

@@ -10,7 +10,7 @@
     Copyright 2008 Société des arts technologiques (SAT),
     https://sat.qc.ca/
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -24,7 +24,7 @@ from copy import copy
 from fnmatch import fnmatch
 from importlib.machinery import EXTENSION_SUFFIXES
 from os import path
-from typing import Any, List, Tuple
+from typing import Any, Generator, List, Tuple
 
 import sphinx.locale
 from sphinx import __display_version__, package_dir
@@ -264,14 +264,46 @@ def is_skipped_module(filename: str, opts: Any, excludes: List[str]) -> bool:
         return False
 
 
+def walk(rootpath: str, excludes: List[str], opts: Any
+         ) -> Generator[Tuple[str, List[str], List[str]], None, None]:
+    """Walk through the directory and list files and subdirectories up."""
+    followlinks = getattr(opts, 'followlinks', False)
+    includeprivate = getattr(opts, 'includeprivate', False)
+
+    for root, subs, files in os.walk(rootpath, followlinks=followlinks):
+        # document only Python module files (that aren't excluded)
+        files = sorted(f for f in files
+                       if f.endswith(PY_SUFFIXES) and
+                       not is_excluded(path.join(root, f), excludes))
+
+        # remove hidden ('.') and private ('_') directories, as well as
+        # excluded dirs
+        if includeprivate:
+            exclude_prefixes = ('.',)  # type: Tuple[str, ...]
+        else:
+            exclude_prefixes = ('.', '_')
+
+        subs[:] = sorted(sub for sub in subs if not sub.startswith(exclude_prefixes) and
+                         not is_excluded(path.join(root, sub), excludes))
+
+        yield root, subs, files
+
+
+def has_child_module(rootpath: str, excludes: List[str], opts: Any) -> bool:
+    """Check the given directory contains child modules at least one."""
+    for root, subs, files in walk(rootpath, excludes, opts):
+        if files:
+            return True
+
+    return False
+
+
 def recurse_tree(rootpath: str, excludes: List[str], opts: Any,
                  user_template_dir: str = None) -> List[str]:
     """
     Look for every file in the directory tree and create the corresponding
     ReST files.
     """
-    followlinks = getattr(opts, 'followlinks', False)
-    includeprivate = getattr(opts, 'includeprivate', False)
     implicit_namespaces = getattr(opts, 'implicit_namespaces', False)
 
     # check if the base directory is a package and get its name
@@ -282,48 +314,36 @@ def recurse_tree(rootpath: str, excludes: List[str], opts: Any,
         root_package = None
 
     toplevels = []
-    for root, subs, files in os.walk(rootpath, followlinks=followlinks):
+    for root, subs, files in walk(rootpath, excludes, opts):
-        # document only Python module files (that aren't excluded)
+        is_pkg = is_packagedir(None, files)
-        py_files = sorted(f for f in files
-                          if f.endswith(PY_SUFFIXES) and
-                          not is_excluded(path.join(root, f), excludes))
-        is_pkg = is_packagedir(None, py_files)
         is_namespace = not is_pkg and implicit_namespaces
         if is_pkg:
-            for f in py_files[:]:
+            for f in files[:]:
                 if is_initpy(f):
-                    py_files.remove(f)
+                    files.remove(f)
-                    py_files.insert(0, f)
+                    files.insert(0, f)
         elif root != rootpath:
             # only accept non-package at toplevel unless using implicit namespaces
             if not implicit_namespaces:
                 del subs[:]
                 continue
-        # remove hidden ('.') and private ('_') directories, as well as
-        # excluded dirs
-        if includeprivate:
-            exclude_prefixes = ('.',)  # type: Tuple[str, ...]
-        else:
-            exclude_prefixes = ('.', '_')
-        subs[:] = sorted(sub for sub in subs if not sub.startswith(exclude_prefixes) and
-                         not is_excluded(path.join(root, sub), excludes))
 
         if is_pkg or is_namespace:
             # we are in a package with something to document
-            if subs or len(py_files) > 1 or not is_skipped_package(root, opts):
+            if subs or len(files) > 1 or not is_skipped_package(root, opts):
                 subpackage = root[len(rootpath):].lstrip(path.sep).\
                     replace(path.sep, '.')
                 # if this is not a namespace or
                 # a namespace and there is something there to document
-                if not is_namespace or len(py_files) > 0:
+                if not is_namespace or has_child_module(root, excludes, opts):
                     create_package_file(root, root_package, subpackage,
-                                        py_files, opts, subs, is_namespace, excludes,
+                                        files, opts, subs, is_namespace, excludes,
                                         user_template_dir)
                     toplevels.append(module_join(root_package, subpackage))
         else:
             # if we are at the root level, we don't require it to be a package
             assert root == rootpath and root_package is None
-            for py_file in py_files:
+            for py_file in files:
                 if not is_skipped_module(path.join(rootpath, py_file), opts, excludes):
                     module = py_file.split('.')[0]
                     create_module_file(root_package, module, opts, user_template_dir)

sphinx/ext/autodoc/deprecated.py

@@ -0,0 +1,126 @@
+"""
+    sphinx.ext.autodoc.deprecated
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    The deprecated Documenters for autodoc.
+
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+import warnings
+from typing import Any
+
+from sphinx.deprecation import RemovedInSphinx50Warning
+from sphinx.ext.autodoc import (AttributeDocumenter, DataDocumenter, FunctionDocumenter,
+                                MethodDocumenter)
+
+
+class SingledispatchFunctionDocumenter(FunctionDocumenter):
+    """
+    Used to be a specialized Documenter subclass for singledispatch'ed functions.
+
+    Retained for backwards compatibility, now does the same as the FunctionDocumenter
+    """
+
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        warnings.warn("%s is deprecated." % self.__class__.__name__,
+                      RemovedInSphinx50Warning, stacklevel=2)
+        super().__init__(*args, **kwargs)
+
+
+class DataDeclarationDocumenter(DataDocumenter):
+    """
+    Specialized Documenter subclass for data that cannot be imported
+    because they are declared without initial value (refs: PEP-526).
+    """
+    objtype = 'datadecl'
+    directivetype = 'data'
+    member_order = 60
+
+    # must be higher than AttributeDocumenter
+    priority = 11
+
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        warnings.warn("%s is deprecated." % self.__class__.__name__,
+                      RemovedInSphinx50Warning, stacklevel=2)
+        super().__init__(*args, **kwargs)
+
+
+class TypeVarDocumenter(DataDocumenter):
+    """
+    Specialized Documenter subclass for TypeVars.
+    """
+
+    objtype = 'typevar'
+    directivetype = 'data'
+    priority = DataDocumenter.priority + 1  # type: ignore
+
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        warnings.warn("%s is deprecated." % self.__class__.__name__,
+                      RemovedInSphinx50Warning, stacklevel=2)
+        super().__init__(*args, **kwargs)
+
+
+class SingledispatchMethodDocumenter(MethodDocumenter):
+    """
+    Used to be a specialized Documenter subclass for singledispatch'ed methods.
+
+    Retained for backwards compatibility, now does the same as the MethodDocumenter
+    """
+
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        warnings.warn("%s is deprecated." % self.__class__.__name__,
+                      RemovedInSphinx50Warning, stacklevel=2)
+        super().__init__(*args, **kwargs)
+
+
+class InstanceAttributeDocumenter(AttributeDocumenter):
+    """
+    Specialized Documenter subclass for attributes that cannot be imported
+    because they are instance attributes (e.g. assigned in __init__).
+    """
+    objtype = 'instanceattribute'
+    directivetype = 'attribute'
+    member_order = 60
+
+    # must be higher than AttributeDocumenter
+    priority = 11
+
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        warnings.warn("%s is deprecated." % self.__class__.__name__,
+                      RemovedInSphinx50Warning, stacklevel=2)
+        super().__init__(*args, **kwargs)
+
+
+class SlotsAttributeDocumenter(AttributeDocumenter):
+    """
+    Specialized Documenter subclass for attributes that cannot be imported
+    because they are attributes in __slots__.
+    """
+    objtype = 'slotsattribute'
+    directivetype = 'attribute'
+    member_order = 60
+
+    # must be higher than AttributeDocumenter
+    priority = 11
+
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        warnings.warn("%s is deprecated." % self.__class__.__name__,
+                      RemovedInSphinx50Warning, stacklevel=2)
+        super().__init__(*args, **kwargs)
+
+
+class GenericAliasDocumenter(DataDocumenter):
+    """
+    Specialized Documenter subclass for GenericAliases.
+    """
+
+    objtype = 'genericalias'
+    directivetype = 'data'
+    priority = DataDocumenter.priority + 1  # type: ignore
+
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        warnings.warn("%s is deprecated." % self.__class__.__name__,
+                      RemovedInSphinx50Warning, stacklevel=2)
+        super().__init__(*args, **kwargs)

sphinx/ext/autodoc/directive.py

@@ -2,7 +2,7 @@
     sphinx.ext.autodoc.directive
     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-    :copyright: Copyright 2007-2017 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -16,7 +16,7 @@ from docutils.statemachine import StringList
 from docutils.utils import Reporter, assemble_option_dict
 
 from sphinx.config import Config
-from sphinx.deprecation import RemovedInSphinx40Warning
+from sphinx.deprecation import RemovedInSphinx40Warning, RemovedInSphinx50Warning
 from sphinx.environment import BuildEnvironment
 from sphinx.ext.autodoc import Documenter, Options
 from sphinx.util import logging
@@ -55,7 +55,7 @@ class DocumenterBridge:
     def __init__(self, env: BuildEnvironment, reporter: Reporter, options: Options,
                  lineno: int, state: Any = None) -> None:
         self.env = env
-        self.reporter = reporter
+        self._reporter = reporter
         self.genopt = options
         self.lineno = lineno
         self.filename_set = set()  # type: Set[str]
@@ -74,6 +74,12 @@ class DocumenterBridge:
     def warn(self, msg: str) -> None:
         logger.warning(msg, location=(self.env.docname, self.lineno))
 
+    @property
+    def reporter(self) -> Reporter:
+        warnings.warn('DocumenterBridge.reporter is deprecated.',
+                      RemovedInSphinx50Warning, stacklevel=2)
+        return self._reporter
+
 
 def process_documenter_options(documenter: "Type[Documenter]", config: Config, options: Dict
                                ) -> Options:

sphinx/ext/autodoc/importer.py

@@ -4,7 +4,7 @@
 
     Importer utilities for autodoc
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -13,15 +13,20 @@ import traceback
 import warnings
 from typing import Any, Callable, Dict, List, Mapping, NamedTuple, Optional, Tuple
 
-from sphinx.deprecation import RemovedInSphinx40Warning, deprecated_alias
+from sphinx.deprecation import (RemovedInSphinx40Warning, RemovedInSphinx50Warning,
-from sphinx.pycode import ModuleAnalyzer
+                                deprecated_alias)
+from sphinx.ext.autodoc.mock import ismock, undecorate
+from sphinx.pycode import ModuleAnalyzer, PycodeError
 from sphinx.util import logging
-from sphinx.util.inspect import getslots, isclass, isenumclass, safe_getattr
+from sphinx.util.inspect import (getannotations, getmro, getslots, isclass, isenumclass,
+                                 safe_getattr)
 
 if False:
     # For type annotation
     from typing import Type  # NOQA
 
+    from sphinx.ext.autodoc import ObjectMember
+
 logger = logging.getLogger(__name__)
 
 
@@ -140,6 +145,9 @@ def get_module_members(module: Any) -> List[Tuple[str, Any]]:
     """Get members of target module."""
     from sphinx.ext.autodoc import INSTANCEATTR
 
+    warnings.warn('sphinx.ext.autodoc.importer.get_module_members() is deprecated.',
+                  RemovedInSphinx50Warning)
+
     members = {}  # type: Dict[str, Tuple[str, Any]]
     for name in dir(module):
         try:
@@ -149,10 +157,12 @@ def get_module_members(module: Any) -> List[Tuple[str, Any]]:
             continue
 
     # annotation only member (ex. attr: int)
-    if hasattr(module, '__annotations__'):
+    try:
-        for name in module.__annotations__:
+        for name in getannotations(module):
             if name not in members:
                 members[name] = (name, INSTANCEATTR)
+    except AttributeError:
+        pass
 
     return sorted(list(members.values()))
 
@@ -163,21 +173,15 @@ Attribute = NamedTuple('Attribute', [('name', str),
 
 
 def _getmro(obj: Any) -> Tuple["Type", ...]:
-    """Get __mro__ from given *obj* safely."""
+    warnings.warn('sphinx.ext.autodoc.importer._getmro() is deprecated.',
-    __mro__ = safe_getattr(obj, '__mro__', None)
+                  RemovedInSphinx40Warning)
-    if isinstance(__mro__, tuple):
+    return getmro(obj)
-        return __mro__
-    else:
-        return tuple()
 
 
 def _getannotations(obj: Any) -> Mapping[str, Any]:
-    """Get __annotations__ from given *obj* safely."""
+    warnings.warn('sphinx.ext.autodoc.importer._getannotations() is deprecated.',
-    __annotations__ = safe_getattr(obj, '__annotations__', None)
+                  RemovedInSphinx40Warning)
-    if isinstance(__annotations__, Mapping):
+    return getannotations(obj)
-        return __annotations__
-    else:
-        return {}
 
 
 def get_object_members(subject: Any, objpath: List[str], attrgetter: Callable,
@@ -225,11 +229,14 @@ def get_object_members(subject: Any, objpath: List[str], attrgetter: Callable,
             continue
 
     # annotation only member (ex. attr: int)
-    for i, cls in enumerate(_getmro(subject)):
+    for i, cls in enumerate(getmro(subject)):
-        for name in _getannotations(cls):
+        try:
-            name = unmangle(cls, name)
+            for name in getannotations(cls):
-            if name and name not in members:
+                name = unmangle(cls, name)
-                members[name] = Attribute(name, i == 0, INSTANCEATTR)
+                if name and name not in members:
+                    members[name] = Attribute(name, i == 0, INSTANCEATTR)
+        except AttributeError:
+            pass
 
     if analyzer:
         # append instance attributes (cf. self.attr1) if analyzer knows
@@ -241,6 +248,85 @@ def get_object_members(subject: Any, objpath: List[str], attrgetter: Callable,
     return members
 
 
+def get_class_members(subject: Any, objpath: List[str], attrgetter: Callable
+                      ) -> Dict[str, "ObjectMember"]:
+    """Get members and attributes of target class."""
+    from sphinx.ext.autodoc import INSTANCEATTR, ObjectMember
+
+    # the members directly defined in the class
+    obj_dict = attrgetter(subject, '__dict__', {})
+
+    members = {}  # type: Dict[str, ObjectMember]
+
+    # enum members
+    if isenumclass(subject):
+        for name, value in subject.__members__.items():
+            if name not in members:
+                members[name] = ObjectMember(name, value, class_=subject)
+
+        superclass = subject.__mro__[1]
+        for name in obj_dict:
+            if name not in superclass.__dict__:
+                value = safe_getattr(subject, name)
+                members[name] = ObjectMember(name, value, class_=subject)
+
+    # members in __slots__
+    try:
+        __slots__ = getslots(subject)
+        if __slots__:
+            from sphinx.ext.autodoc import SLOTSATTR
+
+            for name, docstring in __slots__.items():
+                members[name] = ObjectMember(name, SLOTSATTR, class_=subject,
+                                             docstring=docstring)
+    except (AttributeError, TypeError, ValueError):
+        pass
+
+    # other members
+    for name in dir(subject):
+        try:
+            value = attrgetter(subject, name)
+            if ismock(value):
+                value = undecorate(value)
+
+            unmangled = unmangle(subject, name)
+            if unmangled and unmangled not in members:
+                if name in obj_dict:
+                    members[unmangled] = ObjectMember(unmangled, value, class_=subject)
+                else:
+                    members[unmangled] = ObjectMember(unmangled, value)
+        except AttributeError:
+            continue
+
+    try:
+        for cls in getmro(subject):
+            # annotation only member (ex. attr: int)
+            try:
+                for name in getannotations(cls):
+                    name = unmangle(cls, name)
+                    if name and name not in members:
+                        members[name] = ObjectMember(name, INSTANCEATTR, class_=cls)
+            except AttributeError:
+                pass
+
+            # append instance attributes (cf. self.attr1) if analyzer knows
+            try:
+                modname = safe_getattr(cls, '__module__')
+                qualname = safe_getattr(cls, '__qualname__')
+                analyzer = ModuleAnalyzer.for_module(modname)
+                analyzer.analyze()
+                for (ns, name), docstring in analyzer.attr_docs.items():
+                    if ns == qualname and name not in members:
+                        members[name] = ObjectMember(name, INSTANCEATTR, class_=cls,
+                                                     docstring='\n'.join(docstring))
+            except (AttributeError, PycodeError):
+                pass
+    except AttributeError:
+        pass
+
+    return members
+
+
 from sphinx.ext.autodoc.mock import (MockFinder, MockLoader, _MockModule, _MockObject,  # NOQA
                                      mock)
 

sphinx/ext/autodoc/mock.py

@@ -4,7 +4,7 @@
 
     mock for autodoc
 
-    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
 
@@ -13,10 +13,11 @@ import os
 import sys
 from importlib.abc import Loader, MetaPathFinder
 from importlib.machinery import ModuleSpec
-from types import FunctionType, MethodType, ModuleType
+from types import ModuleType
 from typing import Any, Generator, Iterator, List, Sequence, Tuple, Union
 
 from sphinx.util import logging
+from sphinx.util.inspect import safe_getattr
 
 logger = logging.getLogger(__name__)
 
@@ -26,6 +27,7 @@ class _MockObject:
 
     __display_name__ = '_MockObject'
     __sphinx_mock__ = True
+    __sphinx_decorator_args__ = ()  # type: Tuple[Any, ...]
 
     def __new__(cls, *args: Any, **kwargs: Any) -> Any:
         if len(args) == 3 and isinstance(args[1], tuple):
@@ -59,18 +61,19 @@ class _MockObject:
         return _make_subclass(key, self.__display_name__, self.__class__)()
 
     def __call__(self, *args: Any, **kwargs: Any) -> Any:
-        if args and type(args[0]) in [type, FunctionType, MethodType]:
+        call = self.__class__()
-            # Appears to be a decorator, pass through unchanged
+        call.__sphinx_decorator_args__ = args
-            return args[0]
+        return call
-        return self
 
     def __repr__(self) -> str:
         return self.__display_name__
 
 
 def _make_subclass(name: str, module: str, superclass: Any = _MockObject,
-                   attributes: Any = None) -> Any:
+                   attributes: Any = None, decorator_args: Tuple = ()) -> Any:
-    attrs = {'__module__': module, '__display_name__': module + '.' + name}
+    attrs = {'__module__': module,
+             '__display_name__': module + '.' + name,
+             '__sphinx_decorator_args__': decorator_args}
     attrs.update(attributes or {})
 
     return type(name, (superclass,), attrs)
@@ -147,3 +150,38 @@ def mock(modnames: List[str]) -> Generator[None, None, None]:
     finally:
         sys.meta_path.remove(finder)
         finder.invalidate_caches()
+
+
+def ismock(subject: Any) -> bool:
+    """Check if the object is mocked."""
+    # check the object has '__sphinx_mock__' attribute
+    try:
+        if safe_getattr(subject, '__sphinx_mock__', None) is None:
+            return False
+    except AttributeError:
+        return False
+
+    # check the object is mocked module
+    if isinstance(subject, _MockModule):
+        return True
+
+    try:
+        # check the object is mocked object
+        __mro__ = safe_getattr(type(subject), '__mro__', [])
+        if len(__mro__) > 2 and __mro__[1] is _MockObject:
+            return True
+    except AttributeError:
+        pass
+
+    return False
+
+
+def undecorate(subject: _MockObject) -> Any:
+    """Unwrap mock if *subject* is decorated by mocked object.
+
+    If not decorated, returns given *subject* itself.
+    """
+    if ismock(subject) and subject.__sphinx_decorator_args__:
+        return subject.__sphinx_decorator_args__[0]
+    else:
+        return subject