Merge pull request #7319 from jakobandersen/c_revamp

C domain rewrite

CHANGES

@@ -25,7 +25,7 @@ Incompatible changes
 * Due to the scoping changes for :rst:dir:`productionlist` some uses of
   :rst:role:`token` must be modified to include the scope which was previously
   ignored.
-* #6903: Internal data structure of C, Python, reST and standard domains have
+* #6903: Internal data structure of Python, reST and standard domains have
   changed.  The node_id is added to the index of objects and modules.  Now they
   contains a pair of docname and node_id for cross reference.
 * #7276: C++ domain: Non intended behavior is removed such as ``say_hello_``
@@ -38,6 +38,10 @@ Incompatible changes
   links to ``.. py:function:: say_hello()``
 * #7246: py domain: Drop special cross reference helper for exceptions,
   functions and methods
+* The C domain has been rewritten, with additional directives and roles.
+  The existing ones are now more strict, resulting in new warnings.
+* The attribute ``sphinx_cpp_tagname`` in the ``desc_signature_line`` node
+  has been renamed to ``sphinx_line_type``.
 
 Deprecated
 ----------
@@ -93,6 +97,14 @@ Features added
   ``SearchLanguage.js_splitter_code``
 * #7142: html theme: Add a theme option: ``pygments_dark_style`` to switch the
   style of code-blocks in dark mode
+* The C domain has been rewritten adding for example:
+
+  - Cross-referencing respecting the current scope.
+  - Possible to document anonymous entities.
+  - More specific directives and roles for each type of entitiy,
+    e.g., handling scoping of enumerators.
+  - New role :rst:role:`c:expr` for rendering expressions and types
+    in text.
 
 Bugs fixed
 ----------
@@ -111,6 +123,8 @@ Bugs fixed
   ``html_link_suffix`` in search results
 * #7179: std domain: Fix whitespaces are suppressed on referring GenericObject
 * #7289: console: use bright colors instead of bold
+* #1539: C, parse array types.
+* #2377: C, parse function pointers even in complex types.
 
 Testing
 --------

doc/usage/restructuredtext/domains.rst

@@ -552,47 +552,62 @@ The C Domain
 
 The C domain (name **c**) is suited for documentation of C API.
 
+.. rst:directive:: .. c:member:: declaration
+                   .. c:var:: declaration
+
+   Describes a C struct member or variable. Example signature::
+
+      .. c:member:: PyObject *PyTypeObject.tp_bases
+
+   The difference between the two directives is only cosmetic.
+
 .. rst:directive:: .. c:function:: function prototype
 
    Describes a C function. The signature should be given as in C, e.g.::
 
-      .. c:function:: PyObject* PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems)
+      .. c:function:: PyObject *PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems)
-
-   This is also used to describe function-like preprocessor macros.  The names
-   of the arguments should be given so they may be used in the description.
 
    Note that you don't have to backslash-escape asterisks in the signature, as
    it is not parsed by the reST inliner.
 
-.. rst:directive:: .. c:member:: declaration
-
-   Describes a C struct member. Example signature::
-
-      .. c:member:: PyObject* PyTypeObject.tp_bases
-
-   The text of the description should include the range of values allowed, how
-   the value should be interpreted, and whether the value can be changed.
-   References to structure members in text should use the ``member`` role.
-
 .. rst:directive:: .. c:macro:: name
+                   .. c:macro:: name(arg list)
 
-   Describes a "simple" C macro.  Simple macros are macros which are used for
+   Describes a C macro, i.e., a C-language ``#define``, without the replacement
-   code expansion, but which do not take arguments so cannot be described as
+   text.
-   functions.  This is a simple C-language ``#define``.  Examples of its use in
-   the Python documentation include :c:macro:`PyObject_HEAD` and
-   :c:macro:`Py_BEGIN_ALLOW_THREADS`.
 
-.. rst:directive:: .. c:type:: name
+   .. versionadded:: 3.0
+      The function style variant.
 
-   Describes a C type (whether defined by a typedef or struct). The signature
+.. rst:directive:: .. c:struct:: name
-   should just be the type name.
 
-.. rst:directive:: .. c:var:: declaration
+   Describes a C struct.
 
-   Describes a global C variable.  The signature should include the type, such
+   .. versionadded:: 3.0
-   as::
 
-      .. c:var:: PyObject* PyClass_Type
+.. rst:directive:: .. c:union:: name
+
+   Describes a C union.
+
+   .. versionadded:: 3.0
+
+.. rst:directive:: .. c:enum:: name
+
+   Describes a C enum.
+
+   .. versionadded:: 3.0
+
+.. rst:directive:: .. c:enumerator:: name
+
+   Describes a C enumerator.
+
+   .. versionadded:: 3.0
+
+.. rst:directive:: .. c:type:: typedef-like declaration
+                   .. c:type:: name
+
+   Describes a C type, either as a typedef, or the alias for an unspecified
+   type.
 
 .. _c-roles:
 
@@ -602,25 +617,93 @@ Cross-referencing C constructs
 The following roles create cross-references to C-language constructs if they
 are defined in the documentation:
 
-.. rst:role:: c:func
-
-   Reference a C-language function. Should include trailing parentheses.
-
 .. rst:role:: c:member
+              c:data
+              c:var
+              c:func
+              c:macro
+              c:struct
+              c:union
+              c:enum
+              c:enumerator
+              c:type
 
-   Reference a C-language member of a struct.
+   Reference a C declaration, as defined above.
+   Note that :rst:role:`c:member`, :rst:role:`c:data`, and
+   :rst:role:`c:var` are equivalent.
 
-.. rst:role:: c:macro
+   .. versionadded:: 3.0
+      The var, struct, union, enum, and enumerator roles.
 
-   Reference a "simple" C macro, as defined above.
 
-.. rst:role:: c:type
+Anonymous Entities
+~~~~~~~~~~~~~~~~~~
 
-   Reference a C-language type.
+C supports anonymous structs, enums, and unions.
+For the sake of documentation they must be given some name that starts with
+``@``, e.g., ``@42`` or ``@data``.
+These names can also be used in cross-references,
+though nested symbols will be found even when omitted.
+The ``@...`` name will always be rendered as **[anonymous]** (possibly as a
+link).
 
-.. rst:role:: c:data
+Example::
 
-   Reference a C-language variable.
+   .. c:struct:: Data
+
+      .. c:union:: @data
+
+         .. c:var:: int a
+
+         .. c:var:: double b
+
+   Explicit ref: :c:var:`Data.@data.a`. Short-hand ref: :c:var:`Data.a`.
+
+This will be rendered as:
+
+.. c:struct:: Data
+
+   .. c:union:: @data
+
+      .. c:var:: int a
+
+      .. c:var:: double b
+
+Explicit ref: :c:var:`Data.@data.a`. Short-hand ref: :c:var:`Data.a`.
+
+.. versionadded:: 3.0
+
+
+Inline Expressions and Types
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. rst:role:: c:expr
+              c:texpr
+
+   Insert a C expression or type either as inline code (``cpp:expr``)
+   or inline text (``cpp:texpr``). For example::
+
+      .. c:var:: int a = 42
+
+      .. c:function:: int f(int i)
+
+      An expression: :c:expr:`a * f(a)` (or as text: :c:texpr:`a * f(a)`).
+
+      A type: :c:expr:`const Data*`
+      (or as text :c:texpr:`const Data*`).
+
+   will be rendered as follows:
+
+   .. c:var:: int a = 42
+
+   .. c:function:: int f(int i)
+
+   An expression: :c:expr:`a * f(a)` (or as text: :c:texpr:`a * f(a)`).
+
+   A type: :c:expr:`const Data*`
+   (or as text :c:texpr:`const Data*`).
+
+   .. versionadded:: 3.0
 
 
 .. _cpp-domain:

sphinx/addnodes.py

@@ -119,7 +119,7 @@ class desc_signature_line(nodes.Part, nodes.Inline, nodes.FixedTextElement):
     It should only be used in a ``desc_signature`` with ``is_multiline`` set.
     Set ``add_permalink = True`` for the line that should get the permalink.
     """
-    sphinx_cpp_tagname = ''
+    sphinx_line_type = ''
 
 
 # nodes to use within a desc_signature or desc_signature_line

sphinx/domains/cpp.py

@@ -9,10 +9,8 @@
 """
 
 import re
-import warnings
-from copy import deepcopy
 from typing import (
-    Any, Callable, Dict, Iterator, List, Match, Pattern, Tuple, Type, TypeVar, Union
+    Any, Callable, Dict, Iterator, List, Tuple, Type, TypeVar, Union
 )
 
 from docutils import nodes, utils
@@ -25,7 +23,6 @@ from sphinx.addnodes import desc_signature, pending_xref
 from sphinx.application import Sphinx
 from sphinx.builders import Builder
 from sphinx.config import Config
-from sphinx.deprecation import RemovedInSphinx40Warning
 from sphinx.directives import ObjectDescription
 from sphinx.domains import Domain, ObjType
 from sphinx.environment import BuildEnvironment
@@ -35,13 +32,19 @@ from sphinx.roles import XRefRole
 from sphinx.transforms import SphinxTransform
 from sphinx.transforms.post_transforms import ReferencesResolver
 from sphinx.util import logging
+from sphinx.util.cfamily import (
+    NoOldIdError, ASTBaseBase, verify_description_mode, StringifyTransform,
+    BaseParser, DefinitionError, UnsupportedMultiCharacterCharLiteral,
+    identifier_re, anon_identifier_re, integer_literal_re, octal_literal_re,
+    hex_literal_re, binary_literal_re, float_literal_re,
+    char_literal_re
+)
 from sphinx.util.docfields import Field, GroupedField
 from sphinx.util.docutils import SphinxDirective
 from sphinx.util.nodes import make_refnode
 
 
 logger = logging.getLogger(__name__)
-StringifyTransform = Callable[[Any], str]
 T = TypeVar('T')
 
 """
@@ -65,7 +68,7 @@ T = TypeVar('T')
 
     Each signature is in a desc_signature node, where all children are
     desc_signature_line nodes. Each of these lines will have the attribute
-    'sphinx_cpp_tagname' set to one of the following (prioritized):
+    'sphinx_line_type' set to one of the following (prioritized):
     - 'declarator', if the line contains the name of the declared object.
     - 'templateParams', if the line starts a template parameter list,
     - 'templateParams', if the line has template parameters
@@ -294,47 +297,6 @@ T = TypeVar('T')
             nested-name
 """
 
-_integer_literal_re = re.compile(r'[1-9][0-9]*')
-_octal_literal_re = re.compile(r'0[0-7]*')
-_hex_literal_re = re.compile(r'0[xX][0-9a-fA-F][0-9a-fA-F]*')
-_binary_literal_re = re.compile(r'0[bB][01][01]*')
-_integer_suffix_re = re.compile(r'')
-_float_literal_re = re.compile(r'''(?x)
-    [+-]?(
-    # decimal
-      ([0-9]+[eE][+-]?[0-9]+)
-    | ([0-9]*\.[0-9]+([eE][+-]?[0-9]+)?)
-    | ([0-9]+\.([eE][+-]?[0-9]+)?)
-    # hex
-    | (0[xX][0-9a-fA-F]+[pP][+-]?[0-9a-fA-F]+)
-    | (0[xX][0-9a-fA-F]*\.[0-9a-fA-F]+([pP][+-]?[0-9a-fA-F]+)?)
-    | (0[xX][0-9a-fA-F]+\.([pP][+-]?[0-9a-fA-F]+)?)
-    )
-''')
-_char_literal_re = re.compile(r'''(?x)
-    ((?:u8)|u|U|L)?
-    '(
-      (?:[^\\'])
-    | (\\(
-        (?:['"?\\abfnrtv])
-      | (?:[0-7]{1,3})
-      | (?:x[0-9a-fA-F]{2})
-      | (?:u[0-9a-fA-F]{4})
-      | (?:U[0-9a-fA-F]{8})
-      ))
-    )'
-''')
-
-_anon_identifier_re = re.compile(r'(@[a-zA-Z0-9_])[a-zA-Z0-9_]*\b')
-_identifier_re = re.compile(r'''(?x)
-    (   # This 'extends' _anon_identifier_re with the ordinary identifiers,
-        # make sure they are in sync.
-        (~?\b[a-zA-Z_])  # ordinary identifiers
-    |   (@[a-zA-Z0-9_])  # our extension for names of anonymous entities
-    )
-    [a-zA-Z0-9_]*\b
-''')
-_whitespace_re = re.compile(r'(?u)\s+')
 _string_re = re.compile(r"[LuU8]?('([^'\\]*(?:\\.[^'\\]*)*)'"
                         r'|"([^"\\]*(?:\\.[^"\\]*)*)")', re.S)
 _visibility_re = re.compile(r'\b(public|private|protected)\b')
@@ -582,25 +544,6 @@ _id_explicit_cast = {
 }
 
 
-class NoOldIdError(Exception):
-    # Used to avoid implementing unneeded id generation for old id schemes.
-    @property
-    def description(self) -> str:
-        warnings.warn('%s.description is deprecated. '
-                      'Coerce the instance to a string instead.' % self.__class__.__name__,
-                      RemovedInSphinx40Warning, stacklevel=2)
-        return str(self)
-
-
-class DefinitionError(Exception):
-    @property
-    def description(self) -> str:
-        warnings.warn('%s.description is deprecated. '
-                      'Coerce the instance to a string instead.' % self.__class__.__name__,
-                      RemovedInSphinx40Warning, stacklevel=2)
-        return str(self)
-
-
 class _DuplicateSymbolError(Exception):
     def __init__(self, symbol: "Symbol", declaration: Any) -> None:
         assert symbol
@@ -612,40 +555,8 @@ class _DuplicateSymbolError(Exception):
         return "Internal C++ duplicate symbol error:\n%s" % self.symbol.dump(0)
 
 
-class ASTBase:
+class ASTBase(ASTBaseBase):
-    def __eq__(self, other: Any) -> bool:
+    pass
-        if type(self) is not type(other):
-            return False
-        try:
-            for key, value in self.__dict__.items():
-                if value != getattr(other, key):
-                    return False
-        except AttributeError:
-            return False
-        return True
-
-    __hash__ = None  # type: Callable[[], int]
-
-    def clone(self) -> Any:
-        """Clone a definition expression node."""
-        return deepcopy(self)
-
-    def _stringify(self, transform: StringifyTransform) -> str:
-        raise NotImplementedError(repr(self))
-
-    def __str__(self) -> str:
-        return self._stringify(lambda ast: str(ast))
-
-    def get_display_string(self) -> str:
-        return self._stringify(lambda ast: ast.get_display_string())
-
-    def __repr__(self) -> str:
-        return '<%s>' % self.__class__.__name__
-
-
-def _verify_description_mode(mode: str) -> None:
-    if mode not in ('lastIsName', 'noneIsName', 'markType', 'markName', 'param'):
-        raise Exception("Description mode '%s' is invalid." % mode)
 
 
 ################################################################################
@@ -779,15 +690,6 @@ class ASTNumberLiteral(ASTBase):
         signode.append(nodes.Text(txt, txt))
 
 
-class UnsupportedMultiCharacterCharLiteral(Exception):
-    @property
-    def decoded(self) -> str:
-        warnings.warn('%s.decoded is deprecated. '
-                      'Coerce the instance to a string instead.' % self.__class__.__name__,
-                      RemovedInSphinx40Warning, stacklevel=2)
-        return str(self)
-
-
 class ASTCharLiteral(ASTBase):
     def __init__(self, prefix: str, data: str) -> None:
         self.prefix = prefix  # may be None when no prefix
@@ -1441,7 +1343,7 @@ class ASTIdentifier(ASTBase):
 
     def describe_signature(self, signode: Any, mode: str, env: "BuildEnvironment",
                            prefix: str, templateArgs: str, symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         if mode == 'markType':
             targetText = prefix + self.identifier + templateArgs
             pnode = addnodes.pending_xref('', refdomain='cpp',
@@ -1704,7 +1606,7 @@ class ASTTemplateParams(ASTBase):
         def makeLine(parentNode: desc_signature = parentNode) -> addnodes.desc_signature_line:
             signode = addnodes.desc_signature_line()
             parentNode += signode
-            signode.sphinx_cpp_tagname = 'templateParams'
+            signode.sphinx_line_type = 'templateParams'
             return signode
         if self.isNested:
             lineNode = parentNode  # type: Element
@@ -1813,7 +1715,7 @@ class ASTTemplateIntroduction(ASTBase):
         # Note: 'lineSpec' has no effect on template introductions.
         signode = addnodes.desc_signature_line()
         parentNode += signode
-        signode.sphinx_cpp_tagname = 'templateIntroduction'
+        signode.sphinx_line_type = 'templateIntroduction'
         self.concept.describe_signature(signode, 'markType', env, symbol)
         signode += nodes.Text('{')
         first = True
@@ -1847,7 +1749,7 @@ class ASTTemplateDeclarationPrefix(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol", lineSpec: bool) -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         for t in self.templates:
             t.describe_signature(signode, 'lastIsName', env, symbol, lineSpec)
 
@@ -1868,7 +1770,7 @@ class ASTOperator(ASTBase):
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", prefix: str, templateArgs: str,
                            symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         identifier = str(self)
         if mode == 'lastIsName':
             signode += addnodes.desc_name(identifier, identifier)
@@ -1947,7 +1849,7 @@ class ASTTemplateArgConstant(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         self.value.describe_signature(signode, mode, env, symbol)
 
 
@@ -1977,7 +1879,7 @@ class ASTTemplateArgs(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         signode += nodes.Text('<')
         first = True
         for a in self.args:
@@ -2072,7 +1974,7 @@ class ASTNestedName(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         # just print the name part, with template args, not template params
         if mode == 'noneIsName':
             signode += nodes.Text(str(self))
@@ -2243,7 +2145,7 @@ class ASTFunctionParameter(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         if self.ellipsis:
             signode += nodes.Text('...')
         else:
@@ -2323,7 +2225,7 @@ class ASTParametersQualifiers(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         paramlist = addnodes.desc_parameterlist()
         for arg in self.args:
             param = addnodes.desc_parameter('', '', noemph=True)
@@ -2489,7 +2391,7 @@ class ASTDeclSpecs(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         modifiers = []  # type: List[Node]
 
         def _add(modifiers: List[Node], text: str) -> None:
@@ -2539,7 +2441,7 @@ class ASTArray(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         signode.append(nodes.Text("["))
         if self.size:
             self.size.describe_signature(signode, mode, env, symbol)
@@ -2623,7 +2525,7 @@ class ASTDeclaratorPtr(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         signode += nodes.Text("*")
         for a in self.attrs:
             a.describe_signature(signode)
@@ -2696,7 +2598,7 @@ class ASTDeclaratorRef(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         signode += nodes.Text("&")
         for a in self.attrs:
             a.describe_signature(signode)
@@ -2749,7 +2651,7 @@ class ASTDeclaratorParamPack(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         signode += nodes.Text("...")
         if self.next.name:
             signode += nodes.Text(' ')
@@ -2826,7 +2728,7 @@ class ASTDeclaratorMemPtr(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         self.className.describe_signature(signode, mode, env, symbol)
         signode += nodes.Text('::*')
 
@@ -2896,7 +2798,7 @@ class ASTDeclaratorParen(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         signode += nodes.Text('(')
         self.inner.describe_signature(signode, mode, env, symbol)
         signode += nodes.Text(')')
@@ -2972,7 +2874,7 @@ class ASTDeclaratorNameParamQual(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         if self.declId:
             self.declId.describe_signature(signode, mode, env, symbol)
         for op in self.arrayOps:
@@ -3014,7 +2916,7 @@ class ASTDeclaratorNameBitField(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         if self.declId:
             self.declId.describe_signature(signode, mode, env, symbol)
         signode.append(nodes.Text(' : ', ' : '))
@@ -3034,7 +2936,7 @@ class ASTParenExprList(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         signode.append(nodes.Text('('))
         first = True
         for e in self.exprs:
@@ -3061,7 +2963,7 @@ class ASTBracedInitList(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         signode.append(nodes.Text('{'))
         first = True
         for e in self.exprs:
@@ -3089,7 +2991,7 @@ class ASTInitializer(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         if self.hasAssign:
             signode.append(nodes.Text(' = '))
         self.value.describe_signature(signode, 'markType', env, symbol)
@@ -3184,7 +3086,7 @@ class ASTType(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         self.declSpecs.describe_signature(signode, 'markType', env, symbol)
         if (self.decl.require_space_after_declSpecs() and
                 len(str(self.declSpecs)) > 0):
@@ -3227,7 +3129,7 @@ class ASTTypeWithInit(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         self.type.describe_signature(signode, mode, env, symbol)
         if self.init:
             self.init.describe_signature(signode, mode, env, symbol)
@@ -3257,7 +3159,7 @@ class ASTTypeUsing(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         self.name.describe_signature(signode, mode, env, symbol=symbol)
         if self.type:
             signode += nodes.Text(' = ')
@@ -3315,7 +3217,7 @@ class ASTBaseClass(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         if self.visibility is not None:
             signode += addnodes.desc_annotation(self.visibility,
                                                 self.visibility)
@@ -3354,7 +3256,7 @@ class ASTClass(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         self.name.describe_signature(signode, mode, env, symbol=symbol)
         if self.final:
             signode += nodes.Text(' ')
@@ -3381,7 +3283,7 @@ class ASTUnion(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         self.name.describe_signature(signode, mode, env, symbol=symbol)
 
 
@@ -3409,7 +3311,7 @@ class ASTEnum(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         # self.scoped has been done by the CPPEnumObject
         self.name.describe_signature(signode, mode, env, symbol=symbol)
         if self.underlyingType:
@@ -3437,7 +3339,7 @@ class ASTEnumerator(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", symbol: "Symbol") -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         self.name.describe_signature(signode, mode, env, symbol)
         if self.init:
             self.init.describe_signature(signode, 'markType', env, symbol)
@@ -3509,14 +3411,14 @@ class ASTDeclaration(ASTBase):
 
     def describe_signature(self, signode: desc_signature, mode: str,
                            env: "BuildEnvironment", options: Dict) -> None:
-        _verify_description_mode(mode)
+        verify_description_mode(mode)
         assert self.symbol
         # The caller of the domain added a desc_signature node.
         # Always enable multiline:
         signode['is_multiline'] = True
         # Put each line in a desc_signature_line node.
         mainDeclNode = addnodes.desc_signature_line()
-        mainDeclNode.sphinx_cpp_tagname = 'declarator'
+        mainDeclNode.sphinx_line_type = 'declarator'
         mainDeclNode['add_permalink'] = not self.symbol.isRedeclaration
 
         if self.templatePrefix:
@@ -4497,7 +4399,7 @@ class Symbol:
         return ''.join(res)
 
 
-class DefinitionParser:
+class DefinitionParser(BaseParser):
     # those without signedness and size modifiers
     # see https://en.cppreference.com/w/cpp/language/types
     _simple_fundemental_types = (
@@ -4508,128 +4410,9 @@ class DefinitionParser:
     _prefix_keys = ('class', 'struct', 'enum', 'union', 'typename')
 
     def __init__(self, definition: Any, warnEnv: Any, config: "Config") -> None:
-        self.definition = definition.strip()
+        super().__init__(definition, warnEnv)
-        self.pos = 0
-        self.end = len(self.definition)
-        self.last_match = None  # type: Match
-        self._previous_state = (0, None)  # type: Tuple[int, Match]
-        self.otherErrors = []  # type: List[DefinitionError]
-        # in our tests the following is set to False to capture bad parsing
-        self.allowFallbackExpressionParsing = True
-
-        self.warnEnv = warnEnv
         self.config = config
 
-    def _make_multi_error(self, errors: List[Any], header: str) -> DefinitionError:
-        if len(errors) == 1:
-            if len(header) > 0:
-                return DefinitionError(header + '\n' + str(errors[0][0]))
-            else:
-                return DefinitionError(str(errors[0][0]))
-        result = [header, '\n']
-        for e in errors:
-            if len(e[1]) > 0:
-                ident = '  '
-                result.append(e[1])
-                result.append(':\n')
-                for line in str(e[0]).split('\n'):
-                    if len(line) == 0:
-                        continue
-                    result.append(ident)
-                    result.append(line)
-                    result.append('\n')
-            else:
-                result.append(str(e[0]))
-        return DefinitionError(''.join(result))
-
-    def status(self, msg: str) -> None:
-        # for debugging
-        indicator = '-' * self.pos + '^'
-        print("%s\n%s\n%s" % (msg, self.definition, indicator))
-
-    def fail(self, msg: str) -> None:
-        errors = []
-        indicator = '-' * self.pos + '^'
-        exMain = DefinitionError(
-            'Invalid definition: %s [error at %d]\n  %s\n  %s' %
-            (msg, self.pos, self.definition, indicator))
-        errors.append((exMain, "Main error"))
-        for err in self.otherErrors:
-            errors.append((err, "Potential other error"))
-        self.otherErrors = []
-        raise self._make_multi_error(errors, '')
-
-    def warn(self, msg: str) -> None:
-        if self.warnEnv:
-            self.warnEnv.warn(msg)
-        else:
-            print("Warning: %s" % msg)
-
-    def match(self, regex: Pattern) -> bool:
-        match = regex.match(self.definition, self.pos)
-        if match is not None:
-            self._previous_state = (self.pos, self.last_match)
-            self.pos = match.end()
-            self.last_match = match
-            return True
-        return False
-
-    def backout(self) -> None:
-        self.pos, self.last_match = self._previous_state
-
-    def skip_string(self, string: str) -> bool:
-        strlen = len(string)
-        if self.definition[self.pos:self.pos + strlen] == string:
-            self.pos += strlen
-            return True
-        return False
-
-    def skip_word(self, word: str) -> bool:
-        return self.match(re.compile(r'\b%s\b' % re.escape(word)))
-
-    def skip_ws(self) -> bool:
-        return self.match(_whitespace_re)
-
-    def skip_word_and_ws(self, word: str) -> bool:
-        if self.skip_word(word):
-            self.skip_ws()
-            return True
-        return False
-
-    def skip_string_and_ws(self, string: str) -> bool:
-        if self.skip_string(string):
-            self.skip_ws()
-            return True
-        return False
-
-    @property
-    def eof(self) -> bool:
-        return self.pos >= self.end
-
-    @property
-    def current_char(self) -> str:
-        try:
-            return self.definition[self.pos]
-        except IndexError:
-            return 'EOF'
-
-    @property
-    def matched_text(self) -> str:
-        if self.last_match is not None:
-            return self.last_match.group()
-        else:
-            return None
-
-    def read_rest(self) -> str:
-        rv = self.definition[self.pos:]
-        self.pos = self.end
-        return rv
-
-    def assert_end(self) -> None:
-        self.skip_ws()
-        if not self.eof:
-            self.fail('Expected end of definition.')
-
     def _parse_string(self):
         if self.current_char != '"':
             return None
@@ -4693,7 +4476,7 @@ class DefinitionParser:
                 self.fail("Expected '(' after '__attribute__('.")
             attrs = []
             while 1:
-                if self.match(_identifier_re):
+                if self.match(identifier_re):
                     name = self.matched_text
                     self.skip_ws()
                     if self.skip_string_and_ws('('):
@@ -4743,8 +4526,8 @@ class DefinitionParser:
             return ASTBooleanLiteral(True)
         if self.skip_word('false'):
             return ASTBooleanLiteral(False)
-        for regex in [_float_literal_re, _binary_literal_re, _hex_literal_re,
+        for regex in [float_literal_re, binary_literal_re, hex_literal_re,
-                      _integer_literal_re, _octal_literal_re]:
+                      integer_literal_re, octal_literal_re]:
             pos = self.pos
             if self.match(regex):
                 while self.current_char in 'uUlLfF':
@@ -4756,7 +4539,7 @@ class DefinitionParser:
             return ASTStringLiteral(string)
 
         # character-literal
-        if self.match(_char_literal_re):
+        if self.match(char_literal_re):
             prefix = self.last_match.group(1)  # may be None when no prefix
             data = self.last_match.group(2)
             try:
@@ -5085,7 +4868,7 @@ class DefinitionParser:
             if self.skip_string_and_ws('...'):
                 if not self.skip_string_and_ws('('):
                     self.fail("Expecting '(' after 'sizeof...'.")
-                if not self.match(_identifier_re):
+                if not self.match(identifier_re):
                     self.fail("Expecting identifier for 'sizeof...'.")
                 ident = ASTIdentifier(self.matched_text)
                 self.skip_ws()
@@ -5335,7 +5118,7 @@ class DefinitionParser:
         # user-defined literal?
         if self.skip_string('""'):
             self.skip_ws()
-            if not self.match(_identifier_re):
+            if not self.match(identifier_re):
                 self.fail("Expected user-defined literal suffix.")
             identifier = ASTIdentifier(self.matched_text)
             return ASTOperatorLiteral(identifier)
@@ -5415,7 +5198,7 @@ class DefinitionParser:
             if self.skip_word_and_ws('operator'):
                 identOrOp = self._parse_operator()
             else:
-                if not self.match(_identifier_re):
+                if not self.match(identifier_re):
                     if memberPointer and len(names) > 0:
                         templates.pop()
                         break
@@ -5703,7 +5486,7 @@ class DefinitionParser:
                 self.pos = pos
                 declId = None
         elif named == 'single':
-            if self.match(_identifier_re):
+            if self.match(identifier_re):
                 identifier = ASTIdentifier(self.matched_text)
                 nne = ASTNestedNameElement(identifier, None)
                 declId = ASTNestedName([nne], [False], rooted=False)
@@ -6133,7 +5916,7 @@ class DefinitionParser:
                 self.skip_ws()
                 parameterPack = self.skip_string('...')
                 self.skip_ws()
-                if self.match(_identifier_re):
+                if self.match(identifier_re):
                     identifier = ASTIdentifier(self.matched_text)
                 else:
                     identifier = None
@@ -6192,7 +5975,7 @@ class DefinitionParser:
             self.skip_ws()
             parameterPack = self.skip_string('...')
             self.skip_ws()
-            if not self.match(_identifier_re):
+            if not self.match(identifier_re):
                 self.fail("Expected identifier in template introduction list.")
             txt_identifier = self.matched_text
             # make sure there isn't a keyword
@@ -6911,7 +6694,7 @@ class CPPXRefRole(XRefRole):
         if not has_explicit_title:
             # major hax: replace anon names via simple string manipulation.
             # Can this actually fail?
-            title = _anon_identifier_re.sub("[anonymous]", str(title))
+            title = anon_identifier_re.sub("[anonymous]", str(title))
 
         if refnode['reftype'] == 'any':
             # Assume the removal part of fix_parens for :any: refs.

sphinx/util/cfamily.py

@@ -0,0 +1,247 @@
+"""
+    sphinx.util.cfamily
+    ~~~~~~~~~~~~~~~~~~~
+
+    Utility functions common to the C and C++ domains.
+
+    :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+import re
+import warnings
+from copy import deepcopy
+from typing import (
+    Any, Callable, List, Match, Pattern, Tuple
+)
+
+from sphinx.deprecation import RemovedInSphinx40Warning
+
+
+StringifyTransform = Callable[[Any], str]
+
+
+_whitespace_re = re.compile(r'(?u)\s+')
+anon_identifier_re = re.compile(r'(@[a-zA-Z0-9_])[a-zA-Z0-9_]*\b')
+identifier_re = re.compile(r'''(?x)
+    (   # This 'extends' _anon_identifier_re with the ordinary identifiers,
+        # make sure they are in sync.
+        (~?\b[a-zA-Z_])  # ordinary identifiers
+    |   (@[a-zA-Z0-9_])  # our extension for names of anonymous entities
+    )
+    [a-zA-Z0-9_]*\b
+''')
+integer_literal_re = re.compile(r'[1-9][0-9]*')
+octal_literal_re = re.compile(r'0[0-7]*')
+hex_literal_re = re.compile(r'0[xX][0-9a-fA-F][0-9a-fA-F]*')
+binary_literal_re = re.compile(r'0[bB][01][01]*')
+float_literal_re = re.compile(r'''(?x)
+    [+-]?(
+    # decimal
+      ([0-9]+[eE][+-]?[0-9]+)
+    | ([0-9]*\.[0-9]+([eE][+-]?[0-9]+)?)
+    | ([0-9]+\.([eE][+-]?[0-9]+)?)
+    # hex
+    | (0[xX][0-9a-fA-F]+[pP][+-]?[0-9a-fA-F]+)
+    | (0[xX][0-9a-fA-F]*\.[0-9a-fA-F]+([pP][+-]?[0-9a-fA-F]+)?)
+    | (0[xX][0-9a-fA-F]+\.([pP][+-]?[0-9a-fA-F]+)?)
+    )
+''')
+char_literal_re = re.compile(r'''(?x)
+    ((?:u8)|u|U|L)?
+    '(
+      (?:[^\\'])
+    | (\\(
+        (?:['"?\\abfnrtv])
+      | (?:[0-7]{1,3})
+      | (?:x[0-9a-fA-F]{2})
+      | (?:u[0-9a-fA-F]{4})
+      | (?:U[0-9a-fA-F]{8})
+      ))
+    )'
+''')
+
+
+def verify_description_mode(mode: str) -> None:
+    if mode not in ('lastIsName', 'noneIsName', 'markType', 'markName', 'param'):
+        raise Exception("Description mode '%s' is invalid." % mode)
+
+
+class NoOldIdError(Exception):
+    # Used to avoid implementing unneeded id generation for old id schemes.
+    @property
+    def description(self) -> str:
+        warnings.warn('%s.description is deprecated. '
+                      'Coerce the instance to a string instead.' % self.__class__.__name__,
+                      RemovedInSphinx40Warning, stacklevel=2)
+        return str(self)
+
+
+class ASTBaseBase:
+    def __eq__(self, other: Any) -> bool:
+        if type(self) is not type(other):
+            return False
+        try:
+            for key, value in self.__dict__.items():
+                if value != getattr(other, key):
+                    return False
+        except AttributeError:
+            return False
+        return True
+
+    __hash__ = None  # type: Callable[[], int]
+
+    def clone(self) -> Any:
+        """Clone a definition expression node."""
+        return deepcopy(self)
+
+    def _stringify(self, transform: StringifyTransform) -> str:
+        raise NotImplementedError(repr(self))
+
+    def __str__(self) -> str:
+        return self._stringify(lambda ast: str(ast))
+
+    def get_display_string(self) -> str:
+        return self._stringify(lambda ast: ast.get_display_string())
+
+    def __repr__(self) -> str:
+        return '<%s>' % self.__class__.__name__
+
+
+class UnsupportedMultiCharacterCharLiteral(Exception):
+    @property
+    def decoded(self) -> str:
+        warnings.warn('%s.decoded is deprecated. '
+                      'Coerce the instance to a string instead.' % self.__class__.__name__,
+                      RemovedInSphinx40Warning, stacklevel=2)
+        return str(self)
+
+
+class DefinitionError(Exception):
+    @property
+    def description(self) -> str:
+        warnings.warn('%s.description is deprecated. '
+                      'Coerce the instance to a string instead.' % self.__class__.__name__,
+                      RemovedInSphinx40Warning, stacklevel=2)
+        return str(self)
+
+
+class BaseParser:
+    def __init__(self, definition: Any, warnEnv: Any) -> None:
+        self.warnEnv = warnEnv
+        self.definition = definition.strip()
+        self.pos = 0
+        self.end = len(self.definition)
+        self.last_match = None  # type: Match
+        self._previous_state = (0, None)  # type: Tuple[int, Match]
+        self.otherErrors = []  # type: List[DefinitionError]
+
+        # in our tests the following is set to False to capture bad parsing
+        self.allowFallbackExpressionParsing = True
+
+    def _make_multi_error(self, errors: List[Any], header: str) -> DefinitionError:
+        if len(errors) == 1:
+            if len(header) > 0:
+                return DefinitionError(header + '\n' + str(errors[0][0]))
+            else:
+                return DefinitionError(str(errors[0][0]))
+        result = [header, '\n']
+        for e in errors:
+            if len(e[1]) > 0:
+                ident = '  '
+                result.append(e[1])
+                result.append(':\n')
+                for line in str(e[0]).split('\n'):
+                    if len(line) == 0:
+                        continue
+                    result.append(ident)
+                    result.append(line)
+                    result.append('\n')
+            else:
+                result.append(str(e[0]))
+        return DefinitionError(''.join(result))
+
+    def status(self, msg: str) -> None:
+        # for debugging
+        indicator = '-' * self.pos + '^'
+        print("%s\n%s\n%s" % (msg, self.definition, indicator))
+
+    def fail(self, msg: str) -> None:
+        errors = []
+        indicator = '-' * self.pos + '^'
+        exMain = DefinitionError(
+            'Invalid definition: %s [error at %d]\n  %s\n  %s' %
+            (msg, self.pos, self.definition, indicator))
+        errors.append((exMain, "Main error"))
+        for err in self.otherErrors:
+            errors.append((err, "Potential other error"))
+        self.otherErrors = []
+        raise self._make_multi_error(errors, '')
+
+    def warn(self, msg: str) -> None:
+        if self.warnEnv:
+            self.warnEnv.warn(msg)
+        else:
+            print("Warning: %s" % msg)
+
+    def match(self, regex: Pattern) -> bool:
+        match = regex.match(self.definition, self.pos)
+        if match is not None:
+            self._previous_state = (self.pos, self.last_match)
+            self.pos = match.end()
+            self.last_match = match
+            return True
+        return False
+
+    def skip_string(self, string: str) -> bool:
+        strlen = len(string)
+        if self.definition[self.pos:self.pos + strlen] == string:
+            self.pos += strlen
+            return True
+        return False
+
+    def skip_word(self, word: str) -> bool:
+        return self.match(re.compile(r'\b%s\b' % re.escape(word)))
+
+    def skip_ws(self) -> bool:
+        return self.match(_whitespace_re)
+
+    def skip_word_and_ws(self, word: str) -> bool:
+        if self.skip_word(word):
+            self.skip_ws()
+            return True
+        return False
+
+    def skip_string_and_ws(self, string: str) -> bool:
+        if self.skip_string(string):
+            self.skip_ws()
+            return True
+        return False
+
+    @property
+    def eof(self) -> bool:
+        return self.pos >= self.end
+
+    @property
+    def current_char(self) -> str:
+        try:
+            return self.definition[self.pos]
+        except IndexError:
+            return 'EOF'
+
+    @property
+    def matched_text(self) -> str:
+        if self.last_match is not None:
+            return self.last_match.group()
+        else:
+            return None
+
+    def read_rest(self) -> str:
+        rv = self.definition[self.pos:]
+        self.pos = self.end
+        return rv
+
+    def assert_end(self) -> None:
+        self.skip_ws()
+        if not self.eof:
+            self.fail('Expected end of definition.')

tests/roots/test-domain-c/conf.py

@@ -0,0 +1 @@
+exclude_patterns = ['_build']

tests/roots/test-domain-c/index.rst

@@ -0,0 +1,52 @@
+test-domain-c
+=============
+
+directives
+----------
+
+.. c:function:: int hello(const char *name)
+
+   :rtype: int
+
+.. c:function:: MyStruct hello2(char *name)
+
+   :rtype: MyStruct
+
+.. c:member:: float Sphinx.version
+.. c:var:: int version
+
+.. c:macro::  IS_SPHINX
+.. c:macro::  SPHINX(arg1, arg2)
+
+.. c:struct:: MyStruct
+.. c:union:: MyUnion
+.. c:enum:: MyEnum
+
+   .. c:enumerator:: MyEnumerator
+
+      :c:enumerator:`MyEnumerator`
+
+   :c:enumerator:`MyEnumerator`
+
+:c:enumerator:`MyEnumerator`
+
+.. c:type:: Sphinx
+.. c:type:: int SphinxVersionNum
+
+
+.. c:struct:: A
+
+   .. c:union:: @data
+
+      .. c:member:: int a
+
+- :c:member:`A.@data.a`
+- :c:member:`A.a`
+
+- :c:expr:`unsigned int`
+- :c:texpr:`unsigned int`
+
+.. c:var:: A a
+
+- :c:expr:`a->b`
+- :c:texpr:`a->b`

tests/roots/test-root/objects.txt

@@ -107,15 +107,15 @@ Referring to :func:`nothing <>`.
 C items
 =======
 
-.. c:function:: Sphinx_DoSomething()
+.. c:function:: void Sphinx_DoSomething()
 
-.. c:member:: SphinxStruct.member
+.. c:member:: int SphinxStruct.member
 
 .. c:macro:: SPHINX_USE_PYTHON
 
 .. c:type:: SphinxType
 
-.. c:var:: sphinx_global
+.. c:var:: int sphinx_global
 
 
 Javascript items

tests/test_build_changes.py

@@ -28,7 +28,7 @@ def test_build(app):
     assert path_html in htmltext
 
     malloc_html = (
-        '<b>Test_Malloc</b>: <i>changed:</i> Changed in version 0.6:'
+        '<b>void *Test_Malloc(size_t n)</b>: <i>changed:</i> Changed in version 0.6:'
         ' Can now be replaced with a different allocator.</a>')
     assert malloc_html in htmltext
 

tests/test_build_html.py

@@ -289,11 +289,11 @@ def test_html4_output(app, status, warning):
         (".//a[@class='reference internal'][@href='#errmod-error']/strong", 'Error'),
         # C references
         (".//span[@class='pre']", 'CFunction()'),
-        (".//a[@href='#c-sphinx-dosomething']", ''),
+        (".//a[@href='#c.Sphinx_DoSomething']", ''),
-        (".//a[@href='#c-sphinxstruct-member']", ''),
+        (".//a[@href='#c.SphinxStruct.member']", ''),
-        (".//a[@href='#c-sphinx-use-python']", ''),
+        (".//a[@href='#c.SPHINX_USE_PYTHON']", ''),
-        (".//a[@href='#c-sphinxtype']", ''),
+        (".//a[@href='#c.SphinxType']", ''),
-        (".//a[@href='#c-sphinx-global']", ''),
+        (".//a[@href='#c.sphinx_global']", ''),
         # test global TOC created by toctree()
         (".//ul[@class='current']/li[@class='toctree-l1 current']/a[@href='#']",
          'Testing object descriptions'),

tests/test_domain_c.py

@@ -8,73 +8,505 @@
     :license: BSD, see LICENSE for details.
 """
 
-from docutils import nodes
+import re
 
+import pytest
+
+from docutils import nodes
+import sphinx.domains.c as cDomain
 from sphinx import addnodes
 from sphinx.addnodes import (
     desc, desc_addname, desc_annotation, desc_content, desc_name, desc_optional,
     desc_parameter, desc_parameterlist, desc_returns, desc_signature, desc_type,
     pending_xref
 )
+from sphinx.domains.c import DefinitionParser, DefinitionError
+from sphinx.domains.c import _max_id, _id_prefix, Symbol
 from sphinx.testing import restructuredtext
 from sphinx.testing.util import assert_node
+from sphinx.util import docutils
+
+
+def parse(name, string):
+    parser = DefinitionParser(string, None)
+    parser.allowFallbackExpressionParsing = False
+    ast = parser.parse_declaration(name, name)
+    parser.assert_end()
+    return ast
+
+
+def check(name, input, idDict, output=None):
+    # first a simple check of the AST
+    if output is None:
+        output = input
+    ast = parse(name, input)
+    res = str(ast)
+    if res != output:
+        print("")
+        print("Input:    ", input)
+        print("Result:   ", res)
+        print("Expected: ", output)
+        raise DefinitionError("")
+    rootSymbol = Symbol(None, None, None, None)
+    symbol = rootSymbol.add_declaration(ast, docname="TestDoc")
+    parentNode = addnodes.desc()
+    signode = addnodes.desc_signature(input, '')
+    parentNode += signode
+    ast.describe_signature(signode, 'lastIsName', symbol, options={})
+
+    idExpected = [None]
+    for i in range(1, _max_id + 1):
+        if i in idDict:
+            idExpected.append(idDict[i])
+        else:
+            idExpected.append(idExpected[i - 1])
+    idActual = [None]
+    for i in range(1, _max_id + 1):
+        #try:
+        id = ast.get_id(version=i)
+        assert id is not None
+        idActual.append(id[len(_id_prefix[i]):])
+        #except NoOldIdError:
+        #    idActual.append(None)
+
+    res = [True]
+    for i in range(1, _max_id + 1):
+        res.append(idExpected[i] == idActual[i])
+
+    if not all(res):
+        print("input:    %s" % input.rjust(20))
+        for i in range(1, _max_id + 1):
+            if res[i]:
+                continue
+            print("Error in id version %d." % i)
+            print("result:   %s" % idActual[i])
+            print("expected: %s" % idExpected[i])
+        #print(rootSymbol.dump(0))
+        raise DefinitionError("")
+
+
+def test_expressions():
+    def exprCheck(expr, output=None):
+        parser = DefinitionParser(expr, None)
+        parser.allowFallbackExpressionParsing = False
+        ast = parser.parse_expression()
+        parser.assert_end()
+        # first a simple check of the AST
+        if output is None:
+            output = expr
+        res = str(ast)
+        if res != output:
+            print("")
+            print("Input:    ", input)
+            print("Result:   ", res)
+            print("Expected: ", output)
+            raise DefinitionError("")
+    # type expressions
+    exprCheck('int*')
+    exprCheck('int *const*')
+    exprCheck('int *volatile*')
+    exprCheck('int *restrict*')
+    exprCheck('int *(*)(double)')
+    exprCheck('const int*')
+    exprCheck('__int64')
+    exprCheck('unsigned __int64')
+
+    # actual expressions
+
+    # primary
+    exprCheck('true')
+    exprCheck('false')
+    ints = ['5', '0', '075', '0x0123456789ABCDEF', '0XF', '0b1', '0B1']
+    unsignedSuffix = ['', 'u', 'U']
+    longSuffix = ['', 'l', 'L', 'll', 'LL']
+    for i in ints:
+        for u in unsignedSuffix:
+            for l in longSuffix:
+                expr = i + u + l
+                exprCheck(expr)
+                expr = i + l + u
+                exprCheck(expr)
+    for suffix in ['', 'f', 'F', 'l', 'L']:
+        for e in [
+                '5e42', '5e+42', '5e-42',
+                '5.', '5.e42', '5.e+42', '5.e-42',
+                '.5', '.5e42', '.5e+42', '.5e-42',
+                '5.0', '5.0e42', '5.0e+42', '5.0e-42']:
+            expr = e + suffix
+            exprCheck(expr)
+        for e in [
+                'ApF', 'Ap+F', 'Ap-F',
+                'A.', 'A.pF', 'A.p+F', 'A.p-F',
+                '.A', '.ApF', '.Ap+F', '.Ap-F',
+                'A.B', 'A.BpF', 'A.Bp+F', 'A.Bp-F']:
+            expr = "0x" + e + suffix
+            exprCheck(expr)
+    exprCheck('"abc\\"cba"')  # string
+    # character literals
+    for p in ['', 'u8', 'u', 'U', 'L']:
+        exprCheck(p + "'a'")
+        exprCheck(p + "'\\n'")
+        exprCheck(p + "'\\012'")
+        exprCheck(p + "'\\0'")
+        exprCheck(p + "'\\x0a'")
+        exprCheck(p + "'\\x0A'")
+        exprCheck(p + "'\\u0a42'")
+        exprCheck(p + "'\\u0A42'")
+        exprCheck(p + "'\\U0001f34c'")
+        exprCheck(p + "'\\U0001F34C'")
+
+    exprCheck('(5)')
+    exprCheck('C')
+    # postfix
+    exprCheck('A(2)')
+    exprCheck('A[2]')
+    exprCheck('a.b.c')
+    exprCheck('a->b->c')
+    exprCheck('i++')
+    exprCheck('i--')
+    # unary
+    exprCheck('++5')
+    exprCheck('--5')
+    exprCheck('*5')
+    exprCheck('&5')
+    exprCheck('+5')
+    exprCheck('-5')
+    exprCheck('!5')
+    exprCheck('~5')
+    exprCheck('sizeof(T)')
+    exprCheck('sizeof -42')
+    exprCheck('alignof(T)')
+    # cast
+    exprCheck('(int)2')
+    # binary op
+    exprCheck('5 || 42')
+    exprCheck('5 && 42')
+    exprCheck('5 | 42')
+    exprCheck('5 ^ 42')
+    exprCheck('5 & 42')
+    # ['==', '!=']
+    exprCheck('5 == 42')
+    exprCheck('5 != 42')
+    # ['<=', '>=', '<', '>']
+    exprCheck('5 <= 42')
+    exprCheck('5 >= 42')
+    exprCheck('5 < 42')
+    exprCheck('5 > 42')
+    # ['<<', '>>']
+    exprCheck('5 << 42')
+    exprCheck('5 >> 42')
+    # ['+', '-']
+    exprCheck('5 + 42')
+    exprCheck('5 - 42')
+    # ['*', '/', '%']
+    exprCheck('5 * 42')
+    exprCheck('5 / 42')
+    exprCheck('5 % 42')
+    # ['.*', '->*']
+    # conditional
+    # TODO
+    # assignment
+    exprCheck('a = 5')
+    exprCheck('a *= 5')
+    exprCheck('a /= 5')
+    exprCheck('a %= 5')
+    exprCheck('a += 5')
+    exprCheck('a -= 5')
+    exprCheck('a >>= 5')
+    exprCheck('a <<= 5')
+    exprCheck('a &= 5')
+    exprCheck('a ^= 5')
+    exprCheck('a |= 5')
+
+
+def test_type_definitions():
+    check('type', "T", {1: "T"})
+
+    check('type', "bool *b", {1: 'b'})
+    check('type', "bool *const b", {1: 'b'})
+    check('type', "bool *const *b", {1: 'b'})
+    check('type', "bool *volatile *b", {1: 'b'})
+    check('type', "bool *restrict *b", {1: 'b'})
+    check('type', "bool *volatile const b", {1: 'b'})
+    check('type', "bool *volatile const b", {1: 'b'})
+    check('type', "bool *volatile const *b", {1: 'b'})
+    check('type', "bool b[]", {1: 'b'})
+    check('type', "long long int foo", {1: 'foo'})
+    # test decl specs on right
+    check('type', "bool const b", {1: 'b'})
+
+    # from breathe#267 (named function parameters for function pointers
+    check('type', 'void (*gpio_callback_t)(struct device *port, uint32_t pin)',
+          {1: 'gpio_callback_t'})
+
+
+def test_macro_definitions():
+    check('macro', 'M', {1: 'M'})
+    check('macro', 'M()', {1: 'M'})
+    check('macro', 'M(arg)', {1: 'M'})
+    check('macro', 'M(arg1, arg2)', {1: 'M'})
+    check('macro', 'M(arg1, arg2, arg3)', {1: 'M'})
+    check('macro', 'M(...)', {1: 'M'})
+    check('macro', 'M(arg, ...)', {1: 'M'})
+    check('macro', 'M(arg1, arg2, ...)', {1: 'M'})
+    check('macro', 'M(arg1, arg2, arg3, ...)', {1: 'M'})
+
+
+def test_member_definitions():
+    check('member', 'void a', {1: 'a'})
+    check('member', '_Bool a', {1: 'a'})
+    check('member', 'bool a', {1: 'a'})
+    check('member', 'char a', {1: 'a'})
+    check('member', 'int a', {1: 'a'})
+    check('member', 'float a', {1: 'a'})
+    check('member', 'double a', {1: 'a'})
+
+    check('member', 'unsigned long a', {1: 'a'})
+    check('member', '__int64 a', {1: 'a'})
+    check('member', 'unsigned __int64 a', {1: 'a'})
+
+    check('member', 'int .a', {1: 'a'})
+
+    check('member', 'int *a', {1: 'a'})
+    check('member', 'int **a', {1: 'a'})
+    check('member', 'const int a', {1: 'a'})
+    check('member', 'volatile int a', {1: 'a'})
+    check('member', 'restrict int a', {1: 'a'})
+    check('member', 'volatile const int a', {1: 'a'})
+    check('member', 'restrict const int a', {1: 'a'})
+    check('member', 'restrict volatile int a', {1: 'a'})
+    check('member', 'restrict volatile const int a', {1: 'a'})
+
+    check('member', 'T t', {1: 't'})
+
+    check('member', 'int a[]', {1: 'a'})
+
+    check('member', 'int (*p)[]', {1: 'p'})
+
+    check('member', 'int a[42]', {1: 'a'})
+    check('member', 'int a = 42', {1: 'a'})
+    check('member', 'T a = {}', {1: 'a'})
+    check('member', 'T a = {1}', {1: 'a'})
+    check('member', 'T a = {1, 2}', {1: 'a'})
+    check('member', 'T a = {1, 2, 3}', {1: 'a'})
+
+    # test from issue #1539
+    check('member', 'CK_UTF8CHAR model[16]', {1: 'model'})
+
+    check('member', 'auto int a', {1: 'a'})
+    check('member', 'register int a', {1: 'a'})
+    check('member', 'extern int a', {1: 'a'})
+    check('member', 'static int a', {1: 'a'})
+
+    check('member', 'thread_local int a', {1: 'a'})
+    check('member', '_Thread_local int a', {1: 'a'})
+    check('member', 'extern thread_local int a', {1: 'a'})
+    check('member', 'thread_local extern int a', {1: 'a'},
+          'extern thread_local int a')
+    check('member', 'static thread_local int a', {1: 'a'})
+    check('member', 'thread_local static int a', {1: 'a'},
+          'static thread_local int a')
+
+    check('member', 'int b : 3', {1: 'b'})
+
+
+def test_function_definitions():
+    check('function', 'void f()', {1: 'f'})
+    check('function', 'void f(int)', {1: 'f'})
+    check('function', 'void f(int i)', {1: 'f'})
+    check('function', 'void f(int i, int j)', {1: 'f'})
+    check('function', 'void f(...)', {1: 'f'})
+    check('function', 'void f(int i, ...)', {1: 'f'})
+    check('function', 'void f(struct T)', {1: 'f'})
+    check('function', 'void f(struct T t)', {1: 'f'})
+    check('function', 'void f(union T)', {1: 'f'})
+    check('function', 'void f(union T t)', {1: 'f'})
+    check('function', 'void f(enum T)', {1: 'f'})
+    check('function', 'void f(enum T t)', {1: 'f'})
+
+    # test from issue #1539
+    check('function', 'void f(A x[])', {1: 'f'})
+
+    # test from issue #2377
+    check('function', 'void (*signal(int sig, void (*func)(int)))(int)', {1: 'signal'})
+
+    check('function', 'extern void f()', {1: 'f'})
+    check('function', 'static void f()', {1: 'f'})
+    check('function', 'inline void f()', {1: 'f'})
+
+    # tests derived from issue #1753 (skip to keep sanity)
+    check('function', "void f(float *q(double))", {1: 'f'})
+    check('function', "void f(float *(*q)(double))", {1: 'f'})
+    check('function', "void f(float (*q)(double))", {1: 'f'})
+    check('function', "int (*f(double d))(float)", {1: 'f'})
+    check('function', "int (*f(bool b))[5]", {1: 'f'})
+    check('function', "void f(int *const p)", {1: 'f'})
+    check('function', "void f(int *volatile const p)", {1: 'f'})
+
+    # from breathe#223
+    check('function', 'void f(struct E e)', {1: 'f'})
+    check('function', 'void f(enum E e)', {1: 'f'})
+    check('function', 'void f(union E e)', {1: 'f'})
+
+
+def test_union_definitions():
+    check('struct', 'A', {1: 'A'})
+
+
+def test_union_definitions():
+    check('union', 'A', {1: 'A'})
+
+
+def test_enum_definitions():
+    check('enum', 'A', {1: 'A'})
+
+    check('enumerator', 'A', {1: 'A'})
+    check('enumerator', 'A = 42', {1: 'A'})
+
+
+def test_anon_definitions():
+    return  # TODO
+    check('class', '@a', {3: "Ut1_a"})
+    check('union', '@a', {3: "Ut1_a"})
+    check('enum', '@a', {3: "Ut1_a"})
+    check('class', '@1', {3: "Ut1_1"})
+    check('class', '@a::A', {3: "NUt1_a1AE"})
+
+
+def test_initializers():
+    idsMember = {1: 'v'}
+    idsFunction = {1: 'f'}
+    # no init
+    check('member', 'T v', idsMember)
+    check('function', 'void f(T v)', idsFunction)
+    # with '=', assignment-expression
+    check('member', 'T v = 42', idsMember)
+    check('function', 'void f(T v = 42)', idsFunction)
+    # with '=', braced-init
+    check('member', 'T v = {}', idsMember)
+    check('function', 'void f(T v = {})', idsFunction)
+    check('member', 'T v = {42, 42, 42}', idsMember)
+    check('function', 'void f(T v = {42, 42, 42})', idsFunction)
+    check('member', 'T v = {42, 42, 42,}', idsMember)
+    check('function', 'void f(T v = {42, 42, 42,})', idsFunction)
+    # TODO: designator-list
+
+
+def test_attributes():
+    return  # TODO
+    # style: C++
+    check('member', '[[]] int f', {1: 'f__i', 2: '1f'})
+    check('member', '[ [ ] ] int f', {1: 'f__i', 2: '1f'},
+          # this will fail when the proper grammar is implemented
+          output='[[ ]] int f')
+    check('member', '[[a]] int f', {1: 'f__i', 2: '1f'})
+    # style: GNU
+    check('member', '__attribute__(()) int f', {1: 'f__i', 2: '1f'})
+    check('member', '__attribute__((a)) int f', {1: 'f__i', 2: '1f'})
+    check('member', '__attribute__((a, b)) int f', {1: 'f__i', 2: '1f'})
+    # style: user-defined id
+    check('member', 'id_attr int f', {1: 'f__i', 2: '1f'})
+    # style: user-defined paren
+    check('member', 'paren_attr() int f', {1: 'f__i', 2: '1f'})
+    check('member', 'paren_attr(a) int f', {1: 'f__i', 2: '1f'})
+    check('member', 'paren_attr("") int f', {1: 'f__i', 2: '1f'})
+    check('member', 'paren_attr(()[{}][]{}) int f', {1: 'f__i', 2: '1f'})
+    with pytest.raises(DefinitionError):
+        parse('member', 'paren_attr(() int f')
+    with pytest.raises(DefinitionError):
+        parse('member', 'paren_attr([) int f')
+    with pytest.raises(DefinitionError):
+        parse('member', 'paren_attr({) int f')
+    with pytest.raises(DefinitionError):
+        parse('member', 'paren_attr([)]) int f')
+    with pytest.raises(DefinitionError):
+        parse('member', 'paren_attr((])) int f')
+    with pytest.raises(DefinitionError):
+        parse('member', 'paren_attr({]}) int f')
+
+    # position: decl specs
+    check('function', 'static inline __attribute__(()) void f()',
+          {1: 'f', 2: '1fv'},
+          output='__attribute__(()) static inline void f()')
+    check('function', '[[attr1]] [[attr2]] void f()',
+          {1: 'f', 2: '1fv'},
+          output='[[attr1]] [[attr2]] void f()')
+    # position: declarator
+    check('member', 'int *[[attr]] i', {1: 'i__iP', 2: '1i'})
+    check('member', 'int *const [[attr]] volatile i', {1: 'i__iPVC', 2: '1i'},
+          output='int *[[attr]] volatile const i')
+    check('member', 'int &[[attr]] i', {1: 'i__iR', 2: '1i'})
+    check('member', 'int *[[attr]] *i', {1: 'i__iPP', 2: '1i'})
+
+
+def test_xref_parsing():
+    return  # TODO
+    def check(target):
+        class Config:
+            cpp_id_attributes = ["id_attr"]
+            cpp_paren_attributes = ["paren_attr"]
+        parser = DefinitionParser(target, None, Config())
+        ast, isShorthand = parser.parse_xref_object()
+        parser.assert_end()
+    check('f')
+    check('f()')
+    check('void f()')
+    check('T f()')
+
+
+# def test_print():
+#     # used for getting all the ids out for checking
+#     for a in ids:
+#         print(a)
+#     raise DefinitionError("")
+
+
+def filter_warnings(warning, file):
+    lines = warning.getvalue().split("\n");
+    res = [l for l in lines if "domain-c" in l and "{}.rst".format(file) in l and
+           "WARNING: document isn't included in any toctree" not in l]
+    print("Filtered warnings for file '{}':".format(file))
+    for w in res:
+        print(w)
+    return res
+
+
+@pytest.mark.sphinx(testroot='domain-c', confoverrides={'nitpicky': True})
+def test_build_domain_c(app, status, warning):
+    app.builder.build_all()
+    ws = filter_warnings(warning, "index")
+    assert len(ws) == 0
 
 
 def test_cfunction(app):
     text = (".. c:function:: PyObject* "
             "PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems)")
     doctree = restructuredtext.parse(app, text)
-    assert_node(doctree,
-                (addnodes.index,
-                 [desc, ([desc_signature, ([desc_type, ([pending_xref, "PyObject"],
-                                                        "* ")],
-                                           [desc_name, "PyType_GenericAlloc"],
-                                           [desc_parameterlist, (desc_parameter,
-                                                                 desc_parameter)])],
-                         desc_content)]))
     assert_node(doctree[1], addnodes.desc, desctype="function",
                 domain="c", objtype="function", noindex=False)
-    assert_node(doctree[1][0][2][0],
-                [desc_parameter, ([pending_xref, "PyTypeObject"],
-                                  [nodes.emphasis, "\xa0*type"])])
-    assert_node(doctree[1][0][2][1],
-                [desc_parameter, ([pending_xref, "Py_ssize_t"],
-                                  [nodes.emphasis, "\xa0nitems"])])
 
     domain = app.env.get_domain('c')
     entry = domain.objects.get('PyType_GenericAlloc')
-    assert entry == ('index', 'c-pytype-genericalloc', 'function')
+    assert entry == ('index', 'c.PyType_GenericAlloc', 'function')
 
 
 def test_cmember(app):
     text = ".. c:member:: PyObject* PyTypeObject.tp_bases"
     doctree = restructuredtext.parse(app, text)
-    assert_node(doctree,
-                (addnodes.index,
-                 [desc, ([desc_signature, ([desc_type, ([pending_xref, "PyObject"],
-                                                        "* ")],
-                                           [desc_name, "PyTypeObject.tp_bases"])],
-                         desc_content)]))
     assert_node(doctree[1], addnodes.desc, desctype="member",
                 domain="c", objtype="member", noindex=False)
 
     domain = app.env.get_domain('c')
     entry = domain.objects.get('PyTypeObject.tp_bases')
-    assert entry == ('index', 'c-pytypeobject-tp-bases', 'member')
+    assert entry == ('index', 'c.PyTypeObject.tp_bases', 'member')
 
 
 def test_cvar(app):
     text = ".. c:var:: PyObject* PyClass_Type"
     doctree = restructuredtext.parse(app, text)
-    assert_node(doctree,
-                (addnodes.index,
-                 [desc, ([desc_signature, ([desc_type, ([pending_xref, "PyObject"],
-                                                        "* ")],
-                                           [desc_name, "PyClass_Type"])],
-                         desc_content)]))
     assert_node(doctree[1], addnodes.desc, desctype="var",
                 domain="c", objtype="var", noindex=False)
 
     domain = app.env.get_domain('c')
     entry = domain.objects.get('PyClass_Type')
-    assert entry == ('index', 'c-pyclass-type', 'var')
+    assert entry == ('index', 'c.PyClass_Type', 'var')