Merge pull request #9534 from astrojuanlu/new-tutorial-describing-code

New Sphinx tutorial, part III

doc/tutorial/automatic-doc-generation.rst

@@ -0,0 +1,166 @@
+Automatic documentation generation from code
+============================================
+
+In the :ref:`previous section <tutorial-describing-objects>` of the tutorial
+you manually documented a Python function in Sphinx. However, the description
+was out of sync with the code itself, since the function signature was not
+the same. Besides, it would be nice to reuse `Python
+docstrings <https://www.python.org/dev/peps/pep-0257/#what-is-a-docstring>`_
+in the documentation, rather than having to write the information in two
+places.
+
+Fortunately, :doc:`the autodoc extension </usage/extensions/autodoc>` provides this
+functionality.
+
+Reusing signatures and docstrings with autodoc
+----------------------------------------------
+
+To use autodoc, first add it to the list of enabled extensions:
+
+.. code-block:: python
+   :caption: docs/source/conf.py
+   :emphasize-lines: 4
+
+   extensions = [
+       'sphinx.ext.duration',
+       'sphinx.ext.doctest',
+       'sphinx.ext.autodoc',
+   ]
+
+Next, move the content of the ``.. py:function`` directive to the function
+docstring in the original Python file, as follows:
+
+.. code-block:: python
+   :caption: lumache.py
+   :emphasize-lines: 2-11
+
+   def get_random_ingredients(kind=None):
+       """
+       Return a list of random ingredients as strings.
+
+       :param kind: Optional "kind" of ingredients.
+       :type kind: list[str] or None
+       :raise lumache.InvalidKindError: If the kind is invalid.
+       :return: The ingredients list.
+       :rtype: list[str]
+
+       """
+       return ["shells", "gorgonzola", "parsley"]
+
+Finally, replace the ``.. py:function`` directive from the Sphinx documentation
+with :rst:dir:`autofunction`:
+
+.. code-block:: rst
+   :caption: docs/source/usage.rst
+   :emphasize-lines: 3
+
+   you can use the ``lumache.get_random_ingredients()`` function:
+
+   .. autofunction:: lumache.get_random_ingredients
+
+If you now build the HTML documentation, the output will be the same!
+With the advantage that it is generated from the code itself.
+Sphinx took the reStructuredText from the docstring and included it,
+also generating proper cross-references.
+
+You can also autogenerate documentation from other objects. For example, add
+the code for the ``InvalidKindError`` exception:
+
+.. code-block:: python
+   :caption: lumache.py
+
+   class InvalidKindError(Exception):
+       """Raised if the kind is invalid."""
+       pass
+
+And replace the ``.. py:exception`` directive with :rst:dir:`autoexception`
+as follows:
+
+.. code-block:: rst
+   :caption: docs/source/usage.rst
+   :emphasize-lines: 4
+
+   or ``"veggies"``. Otherwise, :py:func:`lumache.get_random_ingredients`
+   will raise an exception.
+
+   .. autoexception:: lumache.InvalidKindError
+
+And again, after running ``make html``, the output will be the same as before.
+
+Generating comprehensive API references
+---------------------------------------
+
+While using ``sphinx.ext.autodoc`` makes keeping the code and the documentation
+in sync much easier, it still requires you to write an ``auto*`` directive
+for every object you want to document. Sphinx provides yet another level of
+automation: the :doc:`autosummary </usage/extensions/autosummary>` extension.
+
+The :rst:dir:`autosummary` directive generates documents that contain all the
+necessary ``autodoc`` directives. To use it, first enable the autosummary
+extension:
+
+.. code-block:: python
+   :caption: docs/source/conf.py
+   :emphasize-lines: 5
+
+   extensions = [
+      'sphinx.ext.duration',
+      'sphinx.ext.doctest',
+      'sphinx.ext.autodoc',
+      'sphinx.ext.autosummary',
+   ]
+
+Next, create a new ``api.rst`` file with these contents:
+
+.. code-block:: rst
+   :caption: docs/source/api.rst
+
+   API
+   ===
+
+   .. autosummary::
+      :toctree: generated
+
+      lumache
+
+Remember to include the new document in the root toctree:
+
+.. code-block:: rst
+   :caption: docs/source/index.rst
+   :emphasize-lines: 7
+
+   Contents
+   --------
+
+   .. toctree::
+
+      usage
+      api
+
+Finally, after you build the HTML documentation running ``make html``, it will
+contain two new pages:
+
+- ``api.html``, corresponding to ``docs/source/api.rst`` and containing a table
+  with the objects you included in the ``autosummary`` directive (in this case,
+  only one).
+- ``generated/lumache.html``, corresponding to a newly created reST file
+  ``generated/lumache.rst`` and containing a summary of members of the module,
+  in this case one function and one exception.
+
+.. figure:: /_static/tutorial/lumache-autosummary.png
+   :width: 80%
+   :align: center
+   :alt: Summary page created by autosummary
+
+   Summary page created by autosummary
+
+Each of the links in the summary page will take you to the places where you
+originally used the corresponding ``autodoc`` directive, in this case in the
+``usage.rst`` document.
+
+.. note::
+
+   The generated files are based on `Jinja2
+   templates <https://jinja2docs.readthedocs.io/>`_ that
+   :ref:`can be customized <autosummary-customizing-templates>`,
+   but that is out of scope for this tutorial.

doc/tutorial/describing-code.rst

@@ -0,0 +1,231 @@
+Describing code in Sphinx
+=========================
+
+In the :doc:`previous sections of the tutorial </tutorial/index>` you can read
+how to write narrative or prose documentation in Sphinx. In this section you
+will describe code objects instead.
+
+Sphinx supports documenting code objects in several languages, namely Python,
+C, C++, JavaScript, and reStructuredText. Each of them can be documented using
+a series of directives and roles grouped by
+:doc:`domain </usage/restructuredtext/domains>`. For the remainder of the
+tutorial you will use the Python domain, but all the concepts seen in this
+section apply for the other domains as well.
+
+.. _tutorial-describing-objects:
+
+Documenting Python objects
+--------------------------
+
+Sphinx offers several roles and directives to document Python objects,
+all grouped together in :ref:`the Python domain <python-domain>`. For example,
+you can use the :rst:dir:`py:function` directive to document a Python function,
+as follows:
+
+.. code-block:: rst
+   :caption: docs/source/usage.rst
+
+   Creating recipes
+   ----------------
+
+   To retrieve a list of random ingredients,
+   you can use the ``lumache.get_random_ingredients()`` function:
+
+   .. py:function:: lumache.get_random_ingredients(kind=None)
+
+      Return a list of random ingredients as strings.
+
+      :param kind: Optional "kind" of ingredients.
+      :type kind: list[str] or None
+      :return: The ingredients list.
+      :rtype: list[str]
+
+Which will render like this:
+
+.. figure:: /_static/tutorial/lumache-py-function.png
+   :width: 80%
+   :align: center
+   :alt: HTML result of documenting a Python function in Sphinx
+
+   The rendered result of documenting a Python function in Sphinx
+
+Notice several things:
+
+- Sphinx parsed the argument of the ``.. py:function`` directive and
+  highlighted the module, the function name, and the parameters appropriately.
+- The directive content includes a one-line description of the function,
+  as well as a :ref:`info field list <info-field-lists>` containing the function
+  parameter, its expected type, the return value, and the return type.
+
+.. note::
+
+   The ``py:`` prefix specifies the :term:`domain`. You may configure the
+   default domain so you can omit the prefix, either globally using the
+   :confval:`primary_domain` configuration, or use the
+   :rst:dir:`default-domain` directive to change it from the point it is called
+   until the end of the file.
+   For example, if you set it to ``py`` (the default), you can write
+   ``.. function::`` directly.
+
+Cross-referencing Python objects
+--------------------------------
+
+By default, most of these directives generate entities that can be
+cross-referenced from any part of the documentation by using
+:ref:`a corresponding role <python-roles>`. For the case of functions,
+you can use :rst:role:`py:func` for that, as follows:
+
+.. code-block:: rst
+   :caption: docs/source/usage.rst
+
+   The ``kind`` parameter should be either ``"meat"``, ``"fish"``,
+   or ``"veggies"``. Otherwise, :py:func:`lumache.get_random_ingredients`
+   will raise an exception.
+
+When generating code documentation, Sphinx will generate a cross-reference automatically just
+by using the name of the object, without you having to explicitly use a role
+for that. For example, you can describe the custom exception raised by the
+function using the :rst:dir:`py:exception` directive:
+
+.. code-block:: rst
+   :caption: docs/source/usage.rst
+
+   .. py:exception:: lumache.InvalidKindError
+
+      Raised if the kind is invalid.
+
+Then, add this exception to the original description of the function:
+
+.. code-block:: rst
+   :caption: docs/source/usage.rst
+   :emphasize-lines: 7
+
+   .. py:function:: lumache.get_random_ingredients(kind=None)
+
+      Return a list of random ingredients as strings.
+
+      :param kind: Optional "kind" of ingredients.
+      :type kind: list[str] or None
+      :raise lumache.InvalidKindError: If the kind is invalid.
+      :return: The ingredients list.
+      :rtype: list[str]
+
+And finally, this is how the result would look:
+
+.. figure:: /_static/tutorial/lumache-py-function-full.png
+   :width: 80%
+   :align: center
+   :alt: HTML result of documenting a Python function in Sphinx
+         with cross-references
+
+   HTML result of documenting a Python function in Sphinx with cross-references
+
+Beautiful, isn't it?
+
+Including doctests in your documentation
+----------------------------------------
+
+Since you are now describing code from a Python library, it will become useful
+to keep both the documentation and the code as synchronized as possible.
+One of the ways to do that in Sphinx is to include code snippets in the
+documentation, called *doctests*, that are executed when the documentation is
+built.
+
+To demonstrate doctests and other Sphinx features covered in this tutorial,
+Sphinx will need to be able to import the code. To achieve that, write this
+at the beginning of ``conf.py``:
+
+.. code-block:: python
+   :caption: docs/source/conf.py
+   :emphasize-lines: 3-5
+
+   # If extensions (or modules to document with autodoc) are in another directory,
+   # add these directories to sys.path here.
+   import pathlib
+   import sys
+   sys.path.insert(0, pathlib.Path(__file__).parents[2].resolve().as_posix())
+
+.. note::
+
+   An alternative to changing the :py:data:`sys.path` variable is to create a
+   ``pyproject.toml`` file and make the code installable,
+   so it behaves like any other Python library. However, the ``sys.path``
+   approach is simpler.
+
+Then, before adding doctests to your documentation, enable the
+:doc:`doctest </usage/extensions/doctest>` extension in ``conf.py``:
+
+.. code-block:: python
+   :caption: docs/source/conf.py
+   :emphasize-lines: 3
+
+   extensions = [
+       'sphinx.ext.duration',
+       'sphinx.ext.doctest',
+   ]
+
+Next, write a doctest block as follows:
+
+.. code-block:: rst
+   :caption: docs/source/usage.rst
+
+   >>> import lumache
+   >>> lumache.get_random_ingredients()
+   ['shells', 'gorgonzola', 'parsley']
+
+Doctests include the Python instructions to be run preceded by ``>>>``,
+the standard Python interpreter prompt, as well as the expected output
+of each instruction. This way, Sphinx can check whether the actual output
+matches the expected one.
+
+To observe how a doctest failure looks like (rather than a code error as
+above), let's write the return value incorrectly first. Therefore, add a
+function ``get_random_ingredients`` like this:
+
+.. code-block:: python
+   :caption: lumache.py
+
+   def get_random_ingredients(kind=None):
+       return ["eggs", "bacon", "spam"]
+
+You can now run ``make doctest`` to execute the doctests of your documentation.
+Initially this will display an error, since the actual code does not behave
+as specified:
+
+.. code-block:: console
+
+   (.venv) $ make doctest
+   Running Sphinx v4.2.0
+   loading pickled environment... done
+   ...
+   running tests...
+
+   Document: usage
+   ---------------
+   **********************************************************************
+   File "usage.rst", line 44, in default
+   Failed example:
+       lumache.get_random_ingredients()
+   Expected:
+       ['shells', 'gorgonzola', 'parsley']
+   Got:
+       ['eggs', 'bacon', 'spam']
+   **********************************************************************
+   ...
+   make: *** [Makefile:20: doctest] Error 1
+
+As you can see, doctest reports the expected and the actual results,
+for easy examination. It is now time to fix the function:
+
+.. code-block:: python
+   :caption: lumache.py
+   :emphasize-lines: 2
+
+   def get_random_ingredients(kind=None):
+       return ["shells", "gorgonzola", "parsley"]
+
+And finally, ``make test`` reports success!
+
+For big projects though, this manual approach can become a bit tedious.
+In the next section, you will see :doc:`how to automate the
+process </tutorial/automatic-doc-generation>`.

doc/tutorial/first-steps.rst

@@ -79,7 +79,8 @@ behavior by adding the following code at the end of your ``conf.py``:
 
 With this configuration value, and after running ``make epub`` again, you will
 notice that URLs appear now as footnotes, which avoids cluttering the text.
-Sweet!
+Sweet! Read on to explore :doc:`other ways to customize
+Sphinx </tutorial/more-sphinx-customization>`.
 
 .. note::
 

doc/tutorial/getting-started.rst

@@ -117,3 +117,4 @@ something like this:
    Freshly created documentation of Lumache
 
 There we go! You created your first HTML documentation using Sphinx.
+Now you can start :doc:`customizing it </tutorial/first-steps>`.

doc/tutorial/index.rst

@@ -33,4 +33,6 @@ project.
    first-steps
    more-sphinx-customization
    narrative-documentation
+   describing-code
+   automatic-doc-generation
    end

doc/tutorial/more-sphinx-customization.rst

@@ -73,3 +73,6 @@ appearance:
    :alt: HTML documentation of Lumache with the Furo theme
 
    HTML documentation of Lumache with the Furo theme
+
+It is now time to :doc:`expand the narrative documentation and split it into
+several documents </tutorial/narrative-documentation>`.

doc/tutorial/narrative-documentation.rst

@@ -85,8 +85,9 @@ introduction paragraph in ``index.rst``:
 
    Check out the :doc:`usage` section for further information.
 
-The :rst:role:`doc` role you used automatically references a specific document
-in the project, in this case the ``usage.rst`` you created earlier.
+The :rst:role:`doc` :ref:`role <rst-roles-alt>` you used automatically
+references a specific document in the project, in this case the ``usage.rst``
+you created earlier.
 
 Alternatively, you can also add a cross-reference to an arbitrary part of the
 project. For that, you need to use the :rst:role:`ref` role, and add an
@@ -126,3 +127,6 @@ cross-reference to. If you do not include an explicit title, hence using
 ``:ref:`installation```, the section title will be used (in this case,
 ``Installation``). Both the ``:doc:`` and the ``:ref:`` roles will be rendered
 as hyperlinks in the HTML documentation.
+
+What about :doc:`documenting code objects in Sphinx </tutorial/describing-code>`?
+Read on!

doc/usage/extensions/autosummary.rst

@@ -210,6 +210,7 @@ also use these config values:
 
    .. versionadded:: 3.2
 
+.. _autosummary-customizing-templates:
 
 Customizing templates
 ---------------------

doc/usage/restructuredtext/domains.rst

@@ -125,6 +125,7 @@ In short:
   component of the target.  For example, ``:py:meth:`~Queue.Queue.get``` will
   refer to ``Queue.Queue.get`` but only display ``get`` as the link text.
 
+.. _python-domain:
 
 The Python Domain
 -----------------