Merge branch '1.7'

CHANGES

@@ -93,6 +93,9 @@ Bugs fixed
 * #4769: autodoc loses the first staticmethod parameter
 * #4790: autosummary: too wide two column tables in PDF builds
 * #4795: Latex customization via ``_templates/longtable.tex_t`` is broken
+* #4789: imgconverter: confused by convert.exe of Windows
+* #4783: On windows, Sphinx crashed when drives of srcdir and outdir are
+  different
 
 Testing
 --------

doc/builders.rst

@@ -155,36 +155,30 @@ The builder's "name" must be given to the **-b** command-line option of
    configuration values that customize the output of this builder, see the
    chapter :ref:`latex-options` for details.
 
-   .. note::
+   The produced LaTeX file uses several LaTeX packages that may not be present
-
+   in a "minimal" TeX distribution installation.  For example, on Ubuntu, the
-      The produced LaTeX file uses several LaTeX packages that may not be
+   following packages need to be installed for successful PDF builds:
-      present in a "minimal" TeX distribution installation. For example, on
-      Ubuntu, the following packages need to be installed for successful PDF
-      builds:
 
    * texlive-latex-recommended
    * texlive-fonts-recommended
    * texlive-latex-extra
-      * latexmk (for ``make latexpdf``)
+   * latexmk (for ``make latexpdf`` on GNU/Linux and MacOS X)
-
+   * latex-xcolor (old Ubuntu)
-      Sphinx will use ``xcolor.sty`` if present: recent Ubuntu distributions
+   * texlive-luatex, texlive-xetex (see :confval:`latex_engine`)
-      have ``xcolor.sty`` included in latex-recommended, earlier ones have it
-      in latex-xcolor. Unicode engines will need texlive-luatex or
-      texlive-xetex.
 
    The testing of Sphinx LaTeX is done on Ubuntu trusty with the above
    mentioned packages, which are from a TeXLive 2013 snapshot dated
    February 2014.
 
    .. versionchanged:: 1.6
-         Formerly, testing had been done for some years on Ubuntu precise
+      Formerly, testing had been done on Ubuntu precise (TeXLive 2009).
-         (based on TeXLive 2009).
+
-      .. versionchanged:: 1.6
+   .. note::
-         Use of ``latexmk`` for ``make latexpdf`` on GNU/Linux and Mac OS X
+
+      Since 1.6, ``make latexpdf`` uses ``latexmk`` (not on Windows).  This
+      makes sure the needed number of runs is automatically executed to get
+      the cross-references, bookmarks, indices, and tables of contents right.
   
-      Since 1.6, ``make latexpdf`` (or
-      ``make -C "<builddir>/latex"`` after a ``sphinx-build`` run) uses
-      ``latexmk`` (not on Windows).
       One can pass to ``latexmk`` options via the ``LATEXMKOPTS``
       Makefile variable. For example:
 
@@ -192,13 +186,22 @@ The builder's "name" must be given to the **-b** command-line option of
 
          make latexpdf LATEXMKOPTS="-silent"
 
-      reduces console output to a minimum. Also, if ``latexmk`` version is
+      reduces console output to a minimum.
-      4.52b or higher (Jan 17) and ``xelatex`` is the :confval:`latex_engine`,
-      then ``LATEXMKOPTS="-xelatex"`` will speed up PDF builds.
 
-      To pass options directly to the
+      Also, if ``latexmk`` version is 4.52b or higher (Jan 17)
-      ``(pdf|xe|lua)latex`` executable, use variable ``LATEXOPTS`` (for example
+      ``LATEXMKOPTS="-xelatex"`` will speed up PDF builds via XeLateX in case
-      ``LATEXOPTS="--interaction=nonstopmode"``).
+      of numerous graphics inclusions.
+
+      .. code-block:: console
+
+         make latexpdf LATEXMKOPTS="-xelatex"
+
+      To pass options directly to the ``(pdf|xe|lua)latex`` executable, use
+      variable ``LATEXOPTS``.
+
+      .. code-block:: console
+
+         make latexpdf LATEXOPTS="--interaction=nonstopmode"
 
    .. autoattribute:: name
 

doc/intl.rst

@@ -207,7 +207,7 @@ easy to fetch and push translations.
 
       $ pip install transifex-client
 
-   .. seealso:: `Transifex Client v0.8 — Transifex documentation`_
+   .. seealso:: `Transifex Client documentation`_
 
 
 #. Create your transifex_ account and create new project for your document
@@ -305,7 +305,7 @@ Contributing to Sphinx reference translation
 The recommended way for new contributors to translate Sphinx reference
 is to join the translation team on Transifex.
 
-There is `sphinx translation page`_ for Sphinx-1.3 documentation.
+There is `sphinx translation page`_ for Sphinx (master) documentation.
 
 1. Login to transifex_ service.
 2. Go to `sphinx translation page`_.
@@ -325,5 +325,5 @@ There is `sphinx translation page`_ for Sphinx-1.3 documentation.
 .. _`transifex-client`: https://pypi.python.org/pypi/transifex-client
 .. _`sphinx-intl`: https://pypi.python.org/pypi/sphinx-intl
 .. _Transifex: https://www.transifex.com/
-.. _`sphinx translation page`: https://www.transifex.com/sphinx-doc/sphinx-doc-1_3/
+.. _`sphinx translation page`: https://www.transifex.com/sphinx-doc/sphinx-doc/
-.. _`Transifex Client v0.8 — Transifex documentation`: https://docs.transifex.com/client/introduction/
+.. _`Transifex Client documentation`: http://docs.transifex.com/developer/client/

doc/latex.rst

@@ -8,8 +8,8 @@ LaTeX customization
 .. module:: latex
    :synopsis: LaTeX specifics.
 
-The *latex* target does not benefit from pre-prepared themes like the
+For details of the LaTeX/PDF builder command line invocation, refer to
-*html* target does (see :doc:`theming`).
+:py:class:`~sphinx.builders.latex.LaTeXBuilder`.
 
 .. raw:: latex
 
@@ -34,8 +34,10 @@ The *latex* target does not benefit from pre-prepared themes like the
 Basic customization
 -------------------
 
-It is achieved via usage of the
+The *latex* target does not benefit from prepared themes.
-:ref:`latex-options` as described in :doc:`config`. For example::
+
+Basic customization is obtained via usage of the :ref:`latex-options`. For
+example::
 
    # inside conf.py
    latex_engine = 'xelatex'
@@ -69,7 +71,7 @@ repertory, and get LaTeX to import it at run time::
    # or, if the \ProvidesPackage LaTeX macro is used in a file mystyle.sty
    'preamble': r'\usepackage{mystyle}',
 
-It is needed to set appropriately :confval:`latex_additional_files`, for
+It is then needed to set appropriately :confval:`latex_additional_files`, for
 example::
 
    latex_additional_files = ["mystyle.sty"]
@@ -79,11 +81,14 @@ example::
 The LaTeX style file options
 ----------------------------
 
+Additional customization is possible via LaTeX options of the Sphinx style
+file.
+
 The sphinxsetup interface
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The ``'sphinxsetup'`` key of :confval:`latex_elements` provides a convenient
-interface to the package options of the Sphinx style file::
+interface::
 
    latex_elements = {
        'sphinxsetup': 'key1=value1, key2=value2, ...',
@@ -103,10 +108,11 @@ inside the document preamble, like this::
 
 .. versionadded:: 1.5
 
-It is possible to insert further uses of the ``\sphinxsetup`` LaTeX macro
+.. hint::
-directly into the body of the document, via the help of the :rst:dir:`raw`
+
-directive. This is what is done for this documentation, for local styling
+   It is possible to insert further uses of the ``\sphinxsetup`` LaTeX macro
-of this chapter in the PDF output::
+   directly into the body of the document, via the help of the :rst:dir:`raw`
+   directive.  Here is how this present chapter in PDF is styled::
 
      .. raw:: latex
 
@@ -125,18 +131,16 @@ of this chapter in the PDF output::
               cautionBorderColor={named}{Cyan},
               cautionBgColor={named}{LightCyan}}
 
-at the start of the chapter and::
+   at the start of the chapter and::
 
      .. raw:: latex
 
         \endgroup
 
-at its end.
+   at its end.
 
-.. note::
+   The colors used in the above are provided by the ``svgnames`` option of the
-
+   "xcolor" package::
-   The colors above are made available via the ``svgnames`` option of
-   the "xcolor" package::
 
       latex_elements = {
           'passoptionstopackages': r'\PassOptionsToPackage{svgnames}{xcolor}',
@@ -465,7 +469,6 @@ Miscellany
   .. versionchanged:: 1.5
      formerly, use of *fncychap* with other styles than ``Bjarne`` was
      dysfunctional.
-- check file :file:`sphinx.sty` for more...
 
 .. hint::
 

sphinx/application.py

@@ -43,7 +43,7 @@ from sphinx.util.build_phase import BuildPhase
 from sphinx.util.console import bold  # type: ignore
 from sphinx.util.docutils import directive_helper
 from sphinx.util.i18n import find_catalog_source_files
-from sphinx.util.osutil import abspath, ensuredir
+from sphinx.util.osutil import abspath, ensuredir, relpath
 from sphinx.util.tags import Tags
 
 if False:
@@ -351,7 +351,7 @@ class Sphinx(object):
             if self.statuscode == 0 and self.builder.epilog:
                 logger.info('')
                 logger.info(self.builder.epilog % {
-                    'outdir': path.relpath(self.outdir),
+                    'outdir': relpath(self.outdir),
                     'project': self.config.project
                 })
         except Exception as err:

sphinx/builders/__init__.py

@@ -23,7 +23,7 @@ from sphinx.util import i18n, import_object, logging, status_iterator
 from sphinx.util.build_phase import BuildPhase
 from sphinx.util.console import bold  # type: ignore
 from sphinx.util.i18n import find_catalog
-from sphinx.util.osutil import SEP, ensuredir, relative_uri
+from sphinx.util.osutil import SEP, ensuredir, relative_uri, relpath
 from sphinx.util.parallel import ParallelTasks, SerialTasks, make_chunks, \
     parallel_available
 
@@ -242,7 +242,7 @@ class Builder(object):
 
         def cat2relpath(cat):
             # type: (CatalogInfo) -> unicode
-            return path.relpath(cat.mo_path, self.env.srcdir).replace(path.sep, SEP)
+            return relpath(cat.mo_path, self.env.srcdir).replace(path.sep, SEP)
 
         logger.info(bold(__('building [mo]: ')) + message)
         for catalog in status_iterator(catalogs, __('writing output... '), "darkgreen",

sphinx/builders/gettext.py

@@ -27,7 +27,7 @@ from sphinx.util import split_index_msg, logging, status_iterator
 from sphinx.util.console import bold  # type: ignore
 from sphinx.util.i18n import find_catalog
 from sphinx.util.nodes import extract_messages, traverse_translatable_index
-from sphinx.util.osutil import safe_relpath, ensuredir, canon_path
+from sphinx.util.osutil import relpath, ensuredir, canon_path
 from sphinx.util.tags import Tags
 
 if False:
@@ -286,8 +286,7 @@ class MessageCatalogBuilder(I18nBuilder):
                 if self.config.gettext_location:
                     # generate "#: file1:line1\n#: file2:line2 ..."
                     output.write("#: %s\n" % "\n#: ".join(  # type: ignore
-                        "%s:%s" % (canon_path(
+                        "%s:%s" % (canon_path(relpath(source, self.outdir)), line)
-                            safe_relpath(source, self.outdir)), line)
                         for source, line, _ in positions))
                 if self.config.gettext_uuid:
                     # generate "# uuid1\n# uuid2\n ..."

sphinx/environment/__init__.py

@@ -38,7 +38,7 @@ from sphinx.util.docutils import sphinx_domains, WarningStream
 from sphinx.util.i18n import find_catalog_files
 from sphinx.util.matching import compile_matchers
 from sphinx.util.nodes import is_translatable
-from sphinx.util.osutil import SEP, ensuredir
+from sphinx.util.osutil import SEP, ensuredir, relpath
 from sphinx.util.websupport import is_commentable
 
 if False:
@@ -354,7 +354,7 @@ class BuildEnvironment(object):
         *filename* should be absolute or relative to the source directory.
         """
         if filename.startswith(self.srcdir):
-            filename = os.path.relpath(filename, self.srcdir)
+            filename = relpath(filename, self.srcdir)
         for suffix in self.config.source_suffix:
             if filename.endswith(suffix):
                 return filename[:-len(suffix)]

sphinx/ext/autosummary/__init__.py

@@ -91,7 +91,7 @@ if False:
 logger = logging.getLogger(__name__)
 
 
-periods_re = re.compile('\.(?:\s+)')
+periods_re = re.compile(r'\.(?:\s+)')
 
 
 # -- autosummary_toc node ------------------------------------------------------

sphinx/ext/doctest.py

@@ -30,7 +30,7 @@ from sphinx.locale import __
 from sphinx.util import force_decode, logging
 from sphinx.util.console import bold  # type: ignore
 from sphinx.util.nodes import set_source_info
-from sphinx.util.osutil import fs_encoding
+from sphinx.util.osutil import fs_encoding, relpath
 
 if False:
     # For type annotation
@@ -372,7 +372,7 @@ Doctest summary
         """Try to get the file which actually contains the doctest, not the
         filename of the document it's included in."""
         try:
-            filename = path.relpath(node.source, self.env.srcdir)\
+            filename = relpath(node.source, self.env.srcdir)\
                 .rsplit(':docstring of ', maxsplit=1)[0]
         except Exception:
             filename = self.env.doc2path(docname, base=None)

sphinx/ext/imgconverter.py

@@ -8,6 +8,7 @@
     :copyright: Copyright 2007-2018 by the Sphinx team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
+import locale
 import subprocess
 
 from sphinx.errors import ExtensionError
@@ -38,17 +39,29 @@ class ImagemagickConverter(ImageConverter):
         try:
             args = [self.config.image_converter, '-version']
             logger.debug('Invoking %r ...', args)
-            ret = subprocess.call(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
+            p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-            if ret == 0:
-                return True
-            else:
-                return False
         except (OSError, IOError):
             logger.warning(__('convert command %r cannot be run.'
                               'check the image_converter setting'),
                            self.config.image_converter)
             return False
 
+        try:
+            stdout, stderr = p.communicate()
+        except (OSError, IOError) as err:
+            if err.errno not in (EPIPE, EINVAL):
+                raise
+            stdout, stderr = p.stdout.read(), p.stderr.read()
+            p.wait()
+        if p.returncode != 0:
+            encoding = locale.getpreferredencoding()
+            logger.warning(__('convert exited with error:\n'
+                              '[stderr]\n%s\n[stdout]\n%s'),
+                           stderr.decode(encoding), stdout.decode(encoding))
+            return False
+
+        return True
+
     def convert(self, _from, _to):
         # type: (unicode, unicode) -> bool
         """Converts the image to expected one."""
@@ -61,7 +74,7 @@ class ImagemagickConverter(ImageConverter):
                     self.config.image_converter_args +
                     [_from, _to])
             logger.debug('Invoking %r ...', args)
-            p = subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
+            p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
         except OSError as err:
             if err.errno != ENOENT:  # No such file or directory
                 raise

sphinx/testing/util.py

@@ -23,6 +23,7 @@ from sphinx.builders.latex import LaTeXBuilder
 from sphinx.ext.autodoc import AutoDirective
 from sphinx.pycode import ModuleAnalyzer
 from sphinx.testing.path import path
+from sphinx.util.osutil import relpath
 
 if False:
     # For type annotation
@@ -201,7 +202,7 @@ def find_files(root, suffix=None):
         dirpath = path(dirpath)
         for f in [f for f in files if not suffix or f.endswith(suffix)]:  # type: ignore
             fpath = dirpath / f
-            yield os.path.relpath(fpath, root)
+            yield relpath(fpath, root)
 
 
 def strip_escseq(text):

sphinx/util/i18n.py

@@ -23,7 +23,7 @@ from babel.messages.pofile import read_po
 from sphinx.errors import SphinxError
 from sphinx.locale import __
 from sphinx.util import logging
-from sphinx.util.osutil import SEP, walk
+from sphinx.util.osutil import SEP, relpath, walk
 
 logger = logging.getLogger(__name__)
 
@@ -97,7 +97,7 @@ def find_catalog_files(docname, srcdir, locale_dirs, lang, compaction):
     domain = find_catalog(docname, compaction)
     files = [gettext.find(domain, path.join(srcdir, dir_), [lang])  # type: ignore
              for dir_ in locale_dirs]
-    files = [path.relpath(f, srcdir) for f in files if f]  # type: ignore
+    files = [relpath(f, srcdir) for f in files if f]  # type: ignore
     return files  # type: ignore
 
 
@@ -138,7 +138,7 @@ def find_catalog_source_files(locale_dirs, locale, domains=None, gettext_compact
             filenames = [f for f in filenames if f.endswith('.po')]
             for filename in filenames:
                 base = path.splitext(filename)[0]
-                domain = path.relpath(path.join(dirpath, base), base_dir)
+                domain = relpath(path.join(dirpath, base), base_dir)
                 if gettext_compact and path.sep in domain:
                     domain = path.split(domain)[0]
                 domain = domain.replace(path.sep, SEP)

sphinx/util/osutil.py

@@ -210,14 +210,21 @@ def ustrftime(format, *args):
             return r.encode().decode('unicode-escape')
 
 
-def safe_relpath(path, start=None):
+def relpath(path, start=os.curdir):
     # type: (unicode, unicode) -> unicode
+    """Return a relative filepath to *path* either from the current directory or
+    from an optional *start* directory.
+
+    This is an alternative of ``os.path.relpath()``.  This returns original path
+    if *path* and *start* are on different drives (for Windows platform).
+    """
     try:
         return os.path.relpath(path, start)
     except ValueError:
         return path
 
 
+safe_relpath = relpath  # for compatibility
 fs_encoding = sys.getfilesystemencoding() or sys.getdefaultencoding()  # type: unicode