IntenseWebs/sphinx
3
0
Fork 0
mirror of https://github.com/sphinx-doc/sphinx.git synced 2025-02-25 18:55:22 -06:00
Code Issues Packages Projects Releases Wiki Activity

Fix #6629: linkcheck: Handle rate-limiting

Browse Source 
Follow the Retry-After header if present, otherwise use an exponential
back-off.
This commit is contained in:
François Freitag
2020-10-04 21:40:14 +02:00
parent c848d4dd37
commit 6b90a63f08
6 changed files with 285 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
doc/usage/builders/index.rst
View File

@@ -442,6 +442,10 @@ name is ``rinoh``. Refer to the `rinohtype manual`_ for details.
Since Sphinx-1.5, the linkcheck builder comes to use requests module.
.. versionchanged:: 3.4
The linkcheck builder retries links when servers apply rate limits.
.. module:: sphinx.builders.xml
.. class:: XMLBuilder

17
doc/usage/configuration.rst
View File

@@ -2525,6 +2525,23 @@ Options for the linkcheck builder
.. versionadded:: 2.3
.. confval:: linkcheck_rate_limit_timeout
The ``linkcheck`` builder may issue a large number of requests to the same
site over a short period of time. This setting controls the builder behavior
when servers indicate that requests are rate-limited.
If a server indicates when to retry (using the `Retry-After`_ header),
``linkcheck`` always follows the server indication.
Otherwise, ``linkcheck`` waits for a minute before to retry and keeps
doubling the wait time between attempts until it succeeds or exceeds the
``linkcheck_rate_limit_timeout``. By default, the timeout is 5 minutes.
.. _Retry-After: https://tools.ietf.org/html/rfc7231#section-7.1.3
.. versionadded:: 3.4
Options for the XML builder
---------------------------