Commit Graph

6710 Commits

Author SHA1 Message Date
Maxim Dounin
0a683fdd93 Cache: introduced min_free cache clearing.
Clearing cache based on free space left on a file system is
expected to allow better disk utilization in some cases, notably
when disk space might be also used for something other than nginx
cache (including nginx own temporary files) and while loading
cache (when cache size might be inaccurate for a while, effectively
disabling max_size cache clearing).

Based on a patch by Adam Bambuch.
2020-06-22 18:03:00 +03:00
Maxim Dounin
6bb4336196 Too large st_blocks values are now ignored (ticket #157).
With XFS, using "allocsize=64m" mount option results in large preallocation
being reported in the st_blocks as returned by fstat() till the file is
closed.  This in turn results in incorrect cache size calculations and
wrong clearing based on max_size.

To avoid too aggressive cache clearing on such volumes, st_blocks values
which result in sizes larger than st_size and eight blocks (an arbitrary
limit) are no longer trusted, and we use st_size instead.

The ngx_de_fs_size() counterpart is intentionally not modified, as
it is used on closed files and hence not affected by this problem.
2020-06-22 18:02:59 +03:00
Maxim Dounin
cd69bf51ca Large block sizes on Linux are now ignored (ticket #1168).
NFS on Linux is known to report wsize as a block size (in both f_bsize
and f_frsize, both in statfs() and statvfs()).  On the other hand,
typical file system block sizes on Linux (ext2/ext3/ext4, XFS) are limited
to pagesize.  (With FAT, block sizes can be at least up to 512k in
extreme cases, but this doesn't really matter, see below.)
To avoid too aggressive cache clearing on NFS volumes on Linux, block
sizes larger than pagesize are now ignored.

Note that it is safe to ignore large block sizes.  Since 3899:e7cd13b7f759
(1.0.1) cache size is calculated based on fstat() st_blocks, and rounding
to file system block size is preserved mostly for Windows.

Note well that on other OSes valid block sizes seen are at least up
to 65536.  In particular, UFS on FreeBSD is known to work well with block
and fragment sizes set to 65536.
2020-06-22 18:02:58 +03:00
Roman Arutyunyan
7547581bbc OCSP: fixed use-after-free on error.
When validating second and further certificates, ssl callback could be called
twice to report the error.  After the first call client connection is
terminated and its memory is released.  Prior to the second call and in it
released connection memory is accessed.

Errors triggering this behavior:
- failure to create the request
- failure to start resolving OCSP responder name
- failure to start connecting to the OCSP responder

The fix is to rearrange the code to eliminate the second call.
2020-06-15 20:17:16 +03:00
Quantum
2afc050bd0 Correctly flush request body to uwsgi with SSL.
The flush flag was not set when forwarding the request body to the uwsgi
server. When using uwsgi_pass suwsgi://..., this causes the uwsgi server
to wait indefinitely for the request body and eventually time out due to
SSL buffering.

This is essentially the same change as 4009:3183165283cc, which was made
to ngx_http_proxy_module.c.

This will fix the uwsgi bug https://github.com/unbit/uwsgi/issues/1490.
2020-06-15 17:35:26 -04:00
Vladimir Homutov
58d1412f0d Stream: fixed processing of zero length UDP packets (ticket #1982). 2020-06-08 11:40:34 +03:00
Maxim Dounin
2d4f04bba0 SSL: added verify callback to ngx_ssl_trusted_certificate().
This ensures that certificate verification is properly logged to debug
log during upstream server certificate verification.  This should help
with debugging various certificate issues.
2020-06-03 19:11:32 +03:00
Ruslan Ermilov
9c3ac44de2 Fixed SIGQUIT not removing listening UNIX sockets (closes #753).
Listening UNIX sockets were not removed on graceful shutdown, preventing
the next runs.  The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
2020-06-01 22:31:23 +03:00
Ruslan Ermilov
da370de990 Fixed removing of listening UNIX sockets when "changing binary".
When changing binary, sending a SIGTERM to the new binary's master process
should not remove inherited UNIX sockets unless the old binary's master
process has exited.
2020-06-01 20:19:27 +03:00
Gena Makhomed
59a0ceb994 Contrib: vim syntax, update core and 3rd party module directives. 2020-05-26 19:17:11 +03:00
Maxim Dounin
b89a5d5998 Version bump. 2020-05-26 22:03:00 +03:00
Maxim Dounin
bab7ab0477 release-1.19.0 tag 2020-05-26 18:00:20 +03:00
Maxim Dounin
87c8507f20 nginx-1.19.0-RELEASE 2020-05-26 18:00:20 +03:00
Maxim Dounin
f457127cc4 Updated OpenSSL used for win32 builds. 2020-05-25 22:10:37 +03:00
Maxim Dounin
4056ce4f4e HTTP/2: invalid connection preface logging (ticket #1981).
Previously, invalid connection preface errors were only logged at debug
level, providing no visible feedback, in particular, when a plain text
HTTP/2 listening socket is erroneously used for HTTP/1.x connections.
Now these are explicitly logged at the info level, much like other
client-related errors.
2020-05-25 18:33:42 +03:00
Sergey Kandaurov
9b87626b0b Fixed format specifiers. 2020-05-23 15:53:08 +03:00
Roman Arutyunyan
5727f9a1e0 OCSP: certificate status cache.
When enabled, certificate status is stored in cache and is used to validate
the certificate in future requests.

New directive ssl_ocsp_cache is added to configure the cache.
2020-05-22 17:25:27 +03:00
Roman Arutyunyan
60438ae395 SSL: client certificate validation with OCSP (ticket #1534).
OCSP validation for client certificates is enabled by the "ssl_ocsp" directive.
OCSP responder can be optionally specified by "ssl_ocsp_responder".

When session is reused, peer chain is not available for validation.
If the verified chain contains certificates from the peer chain not available
at the server, validation will fail.
2020-05-22 17:30:12 +03:00
Roman Arutyunyan
aa94ee82f6 OCSP stapling: iterate over all responder addresses.
Previously only the first responder address was used per each stapling update.
Now, in case of a network or parsing error, next address is used.

This also fixes the issue with unsupported responder address families
(ticket #1330).
2020-05-22 20:35:05 +03:00
Roman Arutyunyan
abdb9aebc6 OCSP stapling: keep extra chain in the staple object. 2020-05-17 14:24:35 +03:00
Roman Arutyunyan
3f2ac979eb OCSP stapling: moved response verification to a separate function. 2020-05-06 21:44:14 +03:00
Jinhua Tan
b47c1f35e2 Upstream: jump out of loop after matching the status code. 2020-05-13 22:02:47 +08:00
Sergey Kandaurov
41ecd45a5b Variables: fixed buffer over-read when evaluating "$arg_". 2020-05-08 19:19:16 +03:00
Ruslan Ermilov
ee9c61b89b gRPC: WINDOW_UPDATE after END_STREAM handling (ticket #1797).
As per https://tools.ietf.org/html/rfc7540#section-6.9,
WINDOW_UPDATE received after a frame with the END_STREAM flag
should be handled and not treated as an error.
2020-04-23 15:10:26 +03:00
Ruslan Ermilov
4c8abb84e3 gRPC: RST_STREAM(NO_ERROR) handling (ticket #1792).
As per https://tools.ietf.org/html/rfc7540#section-8.1,

: A server can send a complete response prior to the client
: sending an entire request if the response does not depend on
: any portion of the request that has not been sent and
: received.  When this is true, a server MAY request that the
: client abort transmission of a request without error by
: sending a RST_STREAM with an error code of NO_ERROR after
: sending a complete response (i.e., a frame with the
: END_STREAM flag).  Clients MUST NOT discard responses as a
: result of receiving such a RST_STREAM, though clients can
: always discard responses at their discretion for other
: reasons.

Previously, RST_STREAM(NO_ERROR) received from upstream after
a frame with the END_STREAM flag was incorrectly treated as an
error.  Now, a single RST_STREAM(NO_ERROR) is properly handled.

This fixes problems observed with modern grpc-c [1], as well
as with the Go gRPC module.

[1] https://github.com/grpc/grpc/pull/1661
2020-04-23 15:10:24 +03:00
Ruslan Ermilov
8c0a49472c Version bump. 2020-04-23 15:10:21 +03:00
Maxim Dounin
3ba88365b5 release-1.17.10 tag 2020-04-14 17:19:26 +03:00
Maxim Dounin
cd15be3a29 nginx-1.17.10-RELEASE 2020-04-14 17:19:26 +03:00
Maxim Dounin
01efecb795 Updated OpenSSL used for win32 builds. 2020-04-14 15:15:16 +03:00
Ruslan Ermilov
b82c08f610 The new auth_delay directive for delaying unauthorized requests.
The request processing is delayed by a timer.  Since nginx updates
internal time once at the start of each event loop iteration, this
normally ensures constant time delay, adding a mitigation from
time-based attacks.

A notable exception to this is the case when there are no additional
events before the timer expires.  To ensure constant-time processing
in this case as well, we trigger an additional event loop iteration
by posting a dummy event for the next event loop iteration.
2020-04-08 01:02:17 +03:00
Ruslan Ermilov
65ae8b3152 Auth basic: explicitly zero out password buffer. 2020-03-13 02:12:10 +03:00
Ruslan Ermilov
1688f575c2 Version bump. 2020-03-16 12:41:41 +03:00
Maxim Dounin
585f08ed0b release-1.17.9 tag 2020-03-03 18:04:21 +03:00
Maxim Dounin
ea3a6c2057 nginx-1.17.9-RELEASE 2020-03-03 18:04:21 +03:00
Maxim Dounin
7bff824878 Updated PCRE used for win32 builds. 2020-03-03 18:03:28 +03:00
Roman Arutyunyan
76ac67b36f Simplified subrequest finalization.
Now it looks similar to what it was before background subrequests were
introduced in 9552758a786e.
2020-02-28 19:54:13 +03:00
Dmitry Volyntsev
3733c6fd70 Fixed premature background subrequest finalization.
When "aio" or "aio threads" is used while processing the response body of an
in-memory background subrequest, the subrequest could be finalized with an aio
operation still in progress.  Upon aio completion either parent request is
woken or the old r->write_event_handler is called again.  The latter may result
in request errors.  In either case post_subrequest handler is never called with
the full response body, which is typically expected when using in-memory
subrequests.

Currently in nginx background subrequests are created by the upstream module
and the mirror module.  The issue does not manifest itself with these
subrequests because they are header-only.  But it can manifest itself with
third-party modules which create in-memory background subrequests.
2020-03-02 20:07:36 +03:00
Maxim Dounin
6ba18bc35e Added default overwrite in error_page 494.
We used to have default error_page overwrite for 495, 496, and 497, so
a configuration like

    error_page 495 /error;

will result in error 400, much like without any error_page configured.

The 494 status code was introduced later (in 3848:de59ad6bf557, nginx 0.9.4),
and relevant changes to ngx_http_core_error_page() were missed, resulting
in inconsistent behaviour of "error_page 494" - with error_page configured
it results in 494 being returned instead of 400.

Reported by Frank Liu,
http://mailman.nginx.org/pipermail/nginx/2020-February/058957.html.
2020-02-28 17:21:18 +03:00
Roman Arutyunyan
ba27037a49 Mp4: fixed possible chunk offset overflow.
In "co64" atom chunk start offset is a 64-bit unsigned integer.  When trimming
the "mdat" atom, chunk offsets are casted to off_t values which are typically
64-bit signed integers.  A specially crafted mp4 file with huge chunk offsets
may lead to off_t overflow and result in negative trim boundaries.

The consequences of the overflow are:
- Incorrect Content-Length header value in the response.
- Negative left boundary of the response file buffer holding the trimmed "mdat".
  This leads to pread()/sendfile() errors followed by closing the client
  connection.

On rare systems where off_t is a 32-bit integer, this scenario is also feasible
with the "stco" atom.

The fix is to add checks which make sure data chunks referenced by each track
are within the mp4 file boundaries.  Additionally a few more checks are added to
ensure mp4 file consistency and log errors.
2020-02-26 15:10:46 +03:00
Sergey Kandaurov
f909a7dc33 Disabled connection reuse while in SSL handshake.
During SSL handshake, the connection could be reused in the OCSP stapling
callback, if configured, which subsequently leads to a segmentation fault.
2020-02-27 19:03:21 +03:00
Maxim Dounin
37984f0be1 Disabled duplicate "Host" headers (ticket #1724).
Duplicate "Host" headers were allowed in nginx 0.7.0 (revision b9de93d804ea)
as a workaround for some broken Motorola phones which used to generate
requests with two "Host" headers[1].  It is believed that this workaround
is no longer relevant.

[1] http://mailman.nginx.org/pipermail/nginx-ru/2008-May/017845.html
2020-02-20 16:51:07 +03:00
Maxim Dounin
b4d6b70d7f Removed "Transfer-Encoding: identity" support.
The "identity" transfer coding has been removed in RFC 7230.  It is
believed that it is not used in real life, and at the same time it
provides a potential attack vector.
2020-02-20 16:19:34 +03:00
Maxim Dounin
e64d798edb Disabled multiple Transfer-Encoding headers.
We anyway do not support more than one transfer encoding, so accepting
requests with multiple Transfer-Encoding headers doesn't make sense.
Further, we do not handle multiple headers, and ignore anything but
the first header.

Reported by Filippo Valsorda.
2020-02-20 16:19:29 +03:00
Vladimir Homutov
de5a054b33 Made ngx_http_get_forwarded_addr_internal() non-recursive. 2020-02-11 13:22:44 +03:00
Sergey Kandaurov
16168dcb01 HTTP/2: fixed socket leak with an incomplete HEADERS frame.
A connection could get stuck without timers if a client has partially sent
the HEADERS frame such that it was split on the individual header boundary.
In this case, it cannot be processed without the rest of the HEADERS frame.

The fix is to call ngx_http_v2_state_headers_save() in this case.  Normally,
it would be called from the ngx_http_v2_state_header_block() handler on the
next iteration, when there is not enough data to continue processing.  This
isn't the case if recv_buffer became empty and there's no more data to read.
2020-02-05 16:29:23 +03:00
Sergey Kandaurov
b7ea950a12 Version bump. 2020-02-05 16:29:14 +03:00
Maxim Dounin
9e07862d6e release-1.17.8 tag 2020-01-21 16:39:42 +03:00
Maxim Dounin
369ef46cc3 nginx-1.17.8-RELEASE 2020-01-21 16:39:41 +03:00
Vladimir Homutov
0e3b3b5735 gRPC: variables support in the "grpc_pass" directive. 2020-01-17 12:13:02 +03:00
Daniil Bondarev
60f648f035 HTTP/2: removed ngx_debug_point() call.
With the recent change to prevent frames flood in d4448892a294,
nginx will finalize the connection with NGX_HTTP_V2_INTERNAL_ERROR
whenever flood is detected, causing nginx aborting or stopping if
the debug_points directive is used in nginx config.
2020-01-14 14:20:08 +03:00