Allow explicitly specifying modules in the coverage builder (#11592)

Currently there is no mechanism to identify totally undocumented modules
in the coverage builder, unlike with partially documented modules.
Resolve this by introducing a new ``coverage_modules`` config option.
This is a list of modules that should be documented somewhere
within the documentation tree. 
Any modules that are specified in the configuration value but 
are not documented anywhere will result in a warning. 
Likewise, any modules that are not in the config option but 
are documented somewhere will result in a warning.

Signed-off-by: Stephen Finucane <stephen@that.guru>

CHANGES.rst

@@ -71,6 +71,9 @@ Features added
 * #12523: Added configuration option, :confval:`math_numsep`, to define the
   separator for math numbering.
   Patch by Thomas Fanning
+* #11592: Add :confval:`coverage_modules` to the coverage builder
+  to allow explicitly specifying which modules should be documented.
+  Patch by Stephen Finucane.
 
 Bugs fixed
 ----------

doc/usage/extensions/coverage.rst

@@ -6,15 +6,64 @@
 
 This extension features one additional builder, the :class:`CoverageBuilder`.
 
-.. class:: CoverageBuilder
-
-   To use this builder, activate the coverage extension in your configuration
-   file and give ``-M coverage`` on the command line.
-
 .. todo:: Write this section.
 
-Several configuration values can be used to specify what the builder
-should check:
+.. note::
+
+   The :doc:`sphinx-apidoc </man/sphinx-apidoc>` command can be used to
+   automatically generate API documentation for all code in a project,
+   avoiding the need to manually author these documents and keep them up-to-date.
+
+.. warning::
+
+   :mod:`~sphinx.ext.coverage` **imports** the modules to be documented.
+   If any modules have side effects on import,
+   these will be executed by the coverage builder when ``sphinx-build`` is run.
+
+   If you document scripts (as opposed to library modules),
+   make sure their main routine is protected by a
+   ``if __name__ == '__main__'`` condition.
+
+.. note::
+
+   For Sphinx (actually, the Python interpreter that executes Sphinx)
+   to find your module, it must be importable.
+   That means that the module or the package must be in
+   one of the directories on :data:`sys.path` -- adapt your :data:`sys.path`
+   in the configuration file accordingly.
+
+To use this builder, activate the coverage extension in your configuration file
+and run ``sphinx-build -M coverage`` on the command line.
+
+
+Builder
+-------
+
+.. py:class:: CoverageBuilder
+
+
+Configuration
+-------------
+
+Several configuration values can be used to specify
+what the builder should check:
+
+.. confval:: coverage_modules
+   :type: ``list[str]``
+   :default: ``[]``
+
+   List of Python packages or modules to test coverage for.
+   When this is provided, Sphinx will introspect each package
+   or module provided in this list as well
+   as all sub-packages and sub-modules found in each.
+   When this is not provided, Sphinx will only provide coverage
+   for Python packages and modules that it is aware of:
+   that is, any modules documented using the :rst:dir:`py:module` directive
+   provided in the :doc:`Python domain </usage/domains/python>`
+   or the :rst:dir:`automodule` directive provided by the
+   :mod:`~sphinx.ext.autodoc` extension.
+
+   .. versionadded:: 7.4
 
 .. confval:: coverage_ignore_modules
 

sphinx/ext/coverage.py

@@ -9,6 +9,7 @@ from __future__ import annotations
 import glob
 import inspect
 import pickle
+import pkgutil
 import re
 import sys
 from importlib import import_module
@@ -23,7 +24,7 @@ from sphinx.util.console import red
 from sphinx.util.inspect import safe_getattr
 
 if TYPE_CHECKING:
-    from collections.abc import Iterator
+    from collections.abc import Iterable, Iterator, Sequence, Set
 
     from sphinx.application import Sphinx
     from sphinx.util.typing import ExtensionMetadata
@@ -66,6 +67,93 @@ def _add_row(col_widths: list[int], columns: list[str], separator: str) -> Itera
     yield _add_line(col_widths, separator)
 
 
+def _load_modules(mod_name: str, ignored_module_exps: Iterable[re.Pattern[str]]) -> Set[str]:
+    """Recursively load all submodules.
+
+    :param mod_name: The name of a module to load submodules for.
+    :param ignored_module_exps: A list of regexes for modules to ignore.
+    :returns: A set of modules names including the provided module name,
+        ``mod_name``
+    :raises ImportError: If the module indicated by ``mod_name`` could not be
+        loaded.
+    """
+    if any(exp.match(mod_name) for exp in ignored_module_exps):
+        return set()
+
+    # This can raise an exception, which must be handled by the caller.
+    mod = import_module(mod_name)
+    modules = {mod_name}
+    if mod.__spec__ is None:
+        return modules
+
+    search_locations = mod.__spec__.submodule_search_locations
+    for (_, sub_mod_name, sub_mod_ispkg) in pkgutil.iter_modules(search_locations):
+        if sub_mod_name == '__main__':
+            continue
+
+        if sub_mod_ispkg:
+            modules |= _load_modules(f'{mod_name}.{sub_mod_name}', ignored_module_exps)
+        else:
+            if any(exp.match(sub_mod_name) for exp in ignored_module_exps):
+                continue
+            modules.add(f'{mod_name}.{sub_mod_name}')
+
+    return modules
+
+
+def _determine_py_coverage_modules(
+    coverage_modules: Sequence[str],
+    seen_modules: Set[str],
+    ignored_module_exps: Iterable[re.Pattern[str]],
+    py_undoc: dict[str, dict[str, Any]],
+) -> list[str]:
+    """Return a sorted list of modules to check for coverage.
+
+    Figure out which of the two operating modes to use:
+
+    - If 'coverage_modules' is not specified, we check coverage for all modules
+      seen in the documentation tree. Any objects found in these modules that are
+      not documented will be noted. This will therefore only identify missing
+      objects, but it requires no additional configuration.
+
+    - If 'coverage_modules' is specified, we check coverage for all modules
+      specified in this configuration value. Any objects found in these modules
+      that are not documented will be noted. In addition, any objects from other
+      modules that are documented will be noted. This will therefore identify both
+      missing modules and missing objects, but it requires manual configuration.
+    """
+    if not coverage_modules:
+        return sorted(seen_modules)
+
+    modules: set[str] = set()
+    for mod_name in coverage_modules:
+        try:
+            modules |= _load_modules(mod_name, ignored_module_exps)
+        except ImportError as err:
+            # TODO(stephenfin): Define a subtype for all logs in this module
+            logger.warning(__('module %s could not be imported: %s'), mod_name, err)
+            py_undoc[mod_name] = {'error': err}
+            continue
+
+    # if there are additional modules then we warn but continue scanning
+    if additional_modules := seen_modules - modules:
+        logger.warning(
+            __('the following modules are documented but were not specified '
+               'in coverage_modules: %s'),
+            ', '.join(additional_modules),
+        )
+
+    # likewise, if there are missing modules we warn but continue scanning
+    if missing_modules := modules - seen_modules:
+        logger.warning(
+            __('the following modules are specified in coverage_modules '
+               'but were not documented'),
+            ', '.join(missing_modules),
+        )
+
+    return sorted(modules)
+
+
 class CoverageBuilder(Builder):
     """
     Evaluates coverage of code in the documentation.
@@ -106,12 +194,12 @@ class CoverageBuilder(Builder):
 
     def write(self, *ignored: Any) -> None:
         self.py_undoc: dict[str, dict[str, Any]] = {}
-        self.py_undocumented: dict[str, set[str]] = {}
-        self.py_documented: dict[str, set[str]] = {}
+        self.py_undocumented: dict[str, Set[str]] = {}
+        self.py_documented: dict[str, Set[str]] = {}
         self.build_py_coverage()
         self.write_py_coverage()
 
-        self.c_undoc: dict[str, set[tuple[str, str]]] = {}
+        self.c_undoc: dict[str, Set[tuple[str, str]]] = {}
         self.build_c_coverage()
         self.write_c_coverage()
 
@@ -169,11 +257,14 @@ class CoverageBuilder(Builder):
         )
 
     def build_py_coverage(self) -> None:
-        objects = self.env.domaindata['py']['objects']
-        modules = self.env.domaindata['py']['modules']
+        seen_objects = frozenset(self.env.domaindata['py']['objects'])
+        seen_modules = frozenset(self.env.domaindata['py']['modules'])
 
         skip_undoc = self.config.coverage_skip_undoc_in_source
 
+        modules = _determine_py_coverage_modules(
+            self.config.coverage_modules, seen_modules, self.mod_ignorexps, self.py_undoc,
+        )
         for mod_name in modules:
             ignore = False
             for exp in self.mod_ignorexps:
@@ -213,7 +304,7 @@ class CoverageBuilder(Builder):
                     continue
 
                 if inspect.isfunction(obj):
-                    if full_name not in objects:
+                    if full_name not in seen_objects:
                         for exp in self.fun_ignorexps:
                             if exp.match(name):
                                 break
@@ -229,7 +320,7 @@ class CoverageBuilder(Builder):
                         if exp.match(name):
                             break
                     else:
-                        if full_name not in objects:
+                        if full_name not in seen_objects:
                             if skip_undoc and not obj.__doc__:
                                 continue
                             # not documented at all
@@ -257,7 +348,7 @@ class CoverageBuilder(Builder):
                             full_attr_name = f'{full_name}.{attr_name}'
                             if self.ignore_pyobj(full_attr_name):
                                 continue
-                            if full_attr_name not in objects:
+                            if full_attr_name not in seen_objects:
                                 attrs.append(attr_name)
                                 undocumented_objects.add(full_attr_name)
                             else:
@@ -273,19 +364,17 @@ class CoverageBuilder(Builder):
 
     def _write_py_statistics(self, op: TextIO) -> None:
         """Outputs the table of ``op``."""
-        all_modules = set(self.py_documented.keys()).union(
-            set(self.py_undocumented.keys()))
-        all_objects: set[str] = set()
-        all_documented_objects: set[str] = set()
+        all_modules = frozenset(self.py_documented.keys() | self.py_undocumented.keys())
+        all_objects: Set[str] = set()
+        all_documented_objects: Set[str] = set()
         for module in all_modules:
-            all_module_objects = self.py_documented[module].union(self.py_undocumented[module])
-            all_objects = all_objects.union(all_module_objects)
-            all_documented_objects = all_documented_objects.union(self.py_documented[module])
+            all_objects |= self.py_documented[module] | self.py_undocumented[module]
+            all_documented_objects |= self.py_documented[module]
 
         # prepare tabular
         table = [['Module', 'Coverage', 'Undocumented']]
-        for module in all_modules:
-            module_objects = self.py_documented[module].union(self.py_undocumented[module])
+        for module in sorted(all_modules):
+            module_objects = self.py_documented[module] | self.py_undocumented[module]
             if len(module_objects):
                 value = 100.0 * len(self.py_documented[module]) / len(module_objects)
             else:
@@ -391,6 +480,7 @@ class CoverageBuilder(Builder):
 
 def setup(app: Sphinx) -> ExtensionMetadata:
     app.add_builder(CoverageBuilder)
+    app.add_config_value('coverage_modules', (), '', types={tuple, list})
     app.add_config_value('coverage_ignore_modules', [], '')
     app.add_config_value('coverage_ignore_functions', [], '')
     app.add_config_value('coverage_ignore_classes', [], '')

tests/roots/test-ext-coverage/conf.py

@@ -5,8 +5,11 @@ sys.path.insert(0, os.path.abspath('.'))
 
 extensions = ['sphinx.ext.autodoc', 'sphinx.ext.coverage']
 
+coverage_modules = [
+    'grog',
+]
 coverage_ignore_pyobjects = [
-    r'^coverage_ignored(\..*)?$',
+    r'^grog\.coverage_ignored(\..*)?$',
     r'\.Ignored$',
     r'\.Documented\.ignored\d$',
 ]

tests/roots/test-ext-coverage/grog/coverage_missing.py

@@ -0,0 +1,7 @@
+"""This module is intentionally not documented."""
+
+class Missing:
+    """An undocumented class."""
+
+    def missing_a(self):
+        """An undocumented method."""

tests/roots/test-ext-coverage/index.rst

@@ -1,6 +1,6 @@
-.. automodule:: coverage_ignored
+.. automodule:: grog.coverage_ignored
    :members:
 
 
-.. automodule:: coverage_not_ignored
+.. automodule:: grog.coverage_not_ignored
    :members:

tests/test_extensions/test_ext_coverage.py

@@ -10,8 +10,10 @@ def test_build(app, status, warning):
     app.build(force_all=True)
 
     py_undoc = (app.outdir / 'python.txt').read_text(encoding='utf8')
-    assert py_undoc.startswith('Undocumented Python objects\n'
-                               '===========================\n')
+    assert py_undoc.startswith(
+        'Undocumented Python objects\n'
+        '===========================\n',
+    )
     assert 'autodoc_target\n--------------\n' in py_undoc
     assert ' * Class -- missing methods:\n' in py_undoc
     assert ' * raises\n' in py_undoc
@@ -23,8 +25,10 @@ def test_build(app, status, warning):
     assert "undocumented  py" not in status.getvalue()
 
     c_undoc = (app.outdir / 'c.txt').read_text(encoding='utf8')
-    assert c_undoc.startswith('Undocumented C API elements\n'
-                              '===========================\n')
+    assert c_undoc.startswith(
+        'Undocumented C API elements\n'
+        '===========================\n',
+    )
     assert 'api.h' in c_undoc
     assert ' * Py_SphinxTest' in c_undoc
 
@@ -54,16 +58,26 @@ Undocumented Python objects
 Statistics
 ----------
 
-+----------------------+----------+--------------+
-| Module               | Coverage | Undocumented |
-+======================+==========+==============+
-| coverage_not_ignored | 0.00%    | 2            |
-+----------------------+----------+--------------+
-| TOTAL                | 0.00%    | 2            |
-+----------------------+----------+--------------+
++---------------------------+----------+--------------+
+| Module                    | Coverage | Undocumented |
++===========================+==========+==============+
+| grog                      | 100.00%  | 0            |
++---------------------------+----------+--------------+
+| grog.coverage_missing     | 100.00%  | 0            |
++---------------------------+----------+--------------+
+| grog.coverage_not_ignored | 0.00%    | 2            |
++---------------------------+----------+--------------+
+| TOTAL                     | 0.00%    | 2            |
++---------------------------+----------+--------------+
 
-coverage_not_ignored
---------------------
+grog.coverage_missing
+---------------------
+
+Classes:
+ * Missing
+
+grog.coverage_not_ignored
+-------------------------
 
 Classes:
  * Documented -- missing methods: