Handling of pseudo-headers is refactored to be more in line with
HTTP/3 implementation, that is, a request line is now constructed
as soon as pseudo-headers are followed by regular fields. Here
this plugs a missing handling for absent mandatory or out-of-order
pseudo-headers.
Such requests are now rejected immediately as malformed.
Closes: https://github.com/nginx/nginx/issues/1526
With prerequisites similar to 696a7f1b9, it was possible to gain 1-byte
overread on invalid UTF-8 sequences. The reason is ngx_utf8_decode()
stops advancing the pointer position on the first encountered invalid
byte. The fix is to adjust the advanced pointer up to the whole saved
sequence in this case. Note that this may result in different output
compared to complete invalid UTF-8 sequences, which we can disregard
at this point.
Reported by Han Yan of Xiaomi and p4p3r of CYBERONE.
If the last variant was given an explicit percentage to reach 100%, it
did not cover hash values on the range boundary or due to accumulating
rounding error. Now this corresponds to the configuration with "*".
If the first response line was split across reads and it didn't appear
a status line, the portion already processed was lost. The change
introduces a new field for proper backtracking on status line fallback.
Previously, it was possible to start parsing headers with a wrong
parsing state after status line was not recognized, as a fallback
used in the scgi and uwsgi modules.
Reported by Leo Lin.
This restores a long-standing optimization when the entire request
body is empty and r->request_body_in_file_only is set, used to avoid
writing an empty file as initially introduced in 4c7f51136 (0.4.4).
The previous condition never worked with chunked body filter, where
rb->bufs holds at least the final chunk; in length body filter, it is
used to indicate the last received buffer since 2a7092138 (1.21.2).
The fix is to additionally check if it is the only empty buffer.
Found with UndefinedBehaviorSanitizer (pointer-overflow)
The SSL_R_RECORD_LAYER_FAILURE ("record layer failure") errors are
reported by OpenSSL 3.2 or newer as the last record layer error for
various low level read errors. Further, a976e6b9e (1.23.4) caused
to always log them at the "crit" level. For example, the following
errors are observed on OpenSSL 3.2.0 - 4.0:
SSL_read() failed (SSL: error:0A000119:SSL routines::decryption failed
or bad record mac error:0A000139:SSL routines::record layer failure)
SSL_read() failed (SSL: error:1C800066:Provider routines::cipher operation
failed error:0A000119:SSL routines::decryption failed or bad record mac
error:0A000139:SSL routines::record layer failure)
SSL_read() failed (SSL: error:0A00010B:SSL routines::wrong version number
error:0A000139:SSL routines::record layer failure)
These errors are now logged at the "info" level.
Closes: https://github.com/nginx/nginx/issues/961
Co-authored-by: Smeet23 <smeetagrawal2003@gmail.com>
Although uninitialized values aren't used in practice due to the
nature of the OpenSSL code flow, this violates the API contract.
Reported by lukefr09 on GitHub.
X509_get_issuer_name() and X509_get_subject_name() were changed to return
a const value. Since it is passed to functions with a non const argument
in older versions, the const modifier is conditionally compiled as needed.
ASN1_INTEGER was made opaque. ASN1_STRING accessors are used to preserve
the behaviour. ASN1_STRING_get0_data() compat shim is provided for OpenSSL
< 1.1.0 where it does not exist.
Previously, it was not properly cleared retaining length as part of
authenticating with CRAM-MD5 and APOP methods that expect to receive
password in auth response. This resulted in null pointer dereference
and worker process crash in subsequent auth attempts with CRAM-MD5.
Reported by Arkadi Vainbrand.
When configured, it enables Multipath TCP support on a listen socket.
As of now it works on Linux starting with Linux 5.6 and glibc 2.32,
where it is enabled with an IPPROTO_MPTCP socket(2) protocol.
To avoid EADDRINUSE errors in bind() and listen() when transitioning
between sockets with different protocols, SO_REUSEPORT is set on both
sockets. See f7f1607bf for potential implications.
Based on previous work by Maxime Dourov and Anthony Doeraene.
It uses a bloom filter to limit sending Stateless Reset packets no more
than once per second in average for the given address. This allows to
address resource asymmetry from precomputed packets, as well as to limit
potential Stateless Reset exchange.
Passing requests to SCGI uses a recalculated size of a request body
as per changes made in d60b8d10f (1.3.9) to support CONTENT_LENGTH
with chunked body requests. This, however, is not compatible with
unbuffered mode introduced later in 7ec559df5 (1.7.11), where such
an approach may not always represent complete request body.
The fix is to use r->headers_in.content_length_n representing either
original Content-Length, if any, or a recalculated value from request
body filters, such as chunked body filter.
Reported by Mufeed VH.
For instance, the last_buf flag is used in the http proxy module when
creating HTTP/2 requests to indicate the output is closed. The flag
is inherited in ngx_output_chain() to a destination buffer when reading
the buffered request body. Then it is used in the proxy output filter
to mark the last HTTP/2 DATA frame with END_STREAM.
The problem happens when reusing the destination buffer, such as to
re-read the buffered request body on next upstream, because this buffer
may contain a dirty last_buf value, which breaks sending HTTP/2 request
body in multiple output filter calls.
The flush and last_in_chain flags are cleared for consistency.
If a buffered request body wasn't fully sent, such as on early upstream
response or limited by flow control, unsent buffers could remain in the
input or busy chains when switching to the next upstream server. This
resulted either in the invalid request sent or a stalled connection.
The fix is to reset chains similar to ngx_http_upstream_reinit().
Now all GitHub .md files will reside in the docs directory.
While expicitly listing all files might be better for clarity,
this eliminates the need to touch "zip" target every such time.
This includes a recently added SUPPORT.md in 367113670.
A total response length with multiple ranges can be larger than the
source response size due to multipart boundary headers. This change
extends max ranges limit imposed in c2c3e3105 (1.1.2) by accounting
boundary headers. Notably, this covers suspicious requests with a
lot of small ranges that have an increased processing overhead and
are susceptible to range based amplification attacks.
The limit disables ranges as long as a total response length comes
close to the source size, additionally penalizing small size ranges
on a large source size where a processing overhead prevails, while
leaving a room for more ranges on a small source size, such that it
should not affect well-behaving applications. The limit can be
altered with the "max_ranges" directive.
Closes#988 on GitHub.
Chunked transfer encoding, since originally introduced in HTTP/1.1
in RFC 2068, is specified to use CRLF as the only line terminator.
Although tolerant applications may recognize a single LF, formally
this covers the start line and fields, and doesn't apply to chunks.
Strict chunked parsing is reaffirmed as intentional in RFC errata
ID 7633, notably "because it does not have to retain backwards
compatibility with 1.0 parsers".
A general RFC 2616 recommendation to tolerate deviations whenever
interpreted unambiguously doesn't apply here, because chunked body
is used to determine HTTP message framing; a relaxed parsing may
cause various security problems due to a broken delimitation.
For instance, this is possible when receiving chunked body from
intermediates that blindly parse chunk-ext or a trailer section
until CRLF, and pass it further without re-coding.
If request URI was shorter than location prefix, as after replacement
with try_files, location length was used to copy the remaining URI part
leading to buffer overread.
The fix is to replace full request URI in this case. In the following
configuration, request "/123" is changed to "/" when sent to backend.
location /1234 {
try_files /123 =404;
proxy_pass http://127.0.0.1:8080/;
}
Closes#983 on GitHub.
This allows to process a port subcomponent and save it in r->port
in a unified way, similar to r->headers_in.server. For HTTP/1.x
request line in the absolute form, r->host_end now includes a port
subcomponent, which is also consistent with HTTP/2 and HTTP/3.
Validation is rewritten to follow RFC 3986 host syntax, based on
ngx_http_parse_request_line(). The following is now rejected:
- the rest of gen-delims "#", "?", "@", "[", "]"
- other unwise delims <">, "<", ">", "\", "^", "`', "{", "|", "}"
- IP literals with a trailing dot, missing closing bracket, or pct-encoded
- a port subcomponent with invalid values
- characters in upper half
In addition to moving memcpy() under the length condition in 15bf6d8cc,
which addressed a reported UB due to string function conventions, this
is repeated for advancing an input buffer, to make the resulting code
more clean and readable.
Additionally, although considered harmless for both string functions and
additive operators, as previously discussed in GitHub PR 866, this fixes
the main source of annoying sanitizer reports in the module.
Prodded by UndefinedBehaviorSanitizer (pointer-overflow).
The function interface is changed to follow a common approach
to other functions used to setup SSL_CTX, with an exception of
"ngx_conf_t *cf" since it is not bound to nginx configuration.
This is required to report and propagate SSL_CTX_set_ex_data()
errors, as reminded by Coverity (CID 1668589).
Variables contain the IANA name of the signature scheme[1] used to sign
the TLS handshake.
Variables are only meaningful when using OpenSSL 3.5 and above, with older
versions they are empty. Moreover, since this data isn't stored in a
serialized session, variables are only available for new sessions.
[1] https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml
Requested by willmafh.
BoringSSL/AWS-LC provide two callbacks for each compression algorithm,
which may be used to compress and decompress certificates in runtime.
This change implements compression support with zlib, as enabled with
the ssl_certificate_compression directive. Compressed certificates
are stored in certificate exdata and reused in subsequent connections.
Notably, AWS-LC saves an X509 pointer in SSL connection, which allows
to use it from SSL_get_certificate() for caching purpose. In contrast,
BoringSSL reconstructs X509 on-the-fly, though given that it doesn't
support multiple certificates, always replacing previously configured
certificates, we use the last configured one from ssl->certs, instead.
In rare cases, it was possible to get into this error state on reload
with improperly updated file timestamps for certificate and key pairs.
The fix is to retry on X509_R_KEY_VALUES_MISMATCH, similar to 5d5d9adcc.
Additionally, loading SSL certificate is updated to avoid certificates
discarded on retry to appear in ssl->certs and in extra chain.
The change introduces an SNI based virtual server selection during
early ClientHello processing. The callback is available since
OpenSSL 1.1.1; for older OpenSSL versions, the previous behaviour
is kept.
Using the ClientHello callback sets a reasonable processing order
for the "server_name" TLS extension. Notably, session resumption
decision now happens after applying server configuration chosen by
SNI, useful with enabled verification of client certificates, which
brings consistency with BoringSSL behaviour. The change supersedes
and reverts a fix made in 46b9f5d38 for TLSv1.3 resumed sessions.
In addition, since the callback is invoked prior to the protocol
version negotiation, this makes it possible to set "ssl_protocols"
on a per-virtual server basis.
To keep the $ssl_server_name variable working with TLSv1.2 resumed
sessions, as previously fixed in fd97b2a80, a limited server name
callback is preserved in order to acknowledge the extension.
Note that to allow third-party modules to properly chain the call to
ngx_ssl_client_hello_callback(), the servername callback function is
passed through exdata.
This was broken in 7468a10b6 (1.29.0), resulting in a missing diagnostics
and SSL error queue not cleared for SSL handshakes rejected by SNI, seen
as "ignoring stale global SSL error" alerts, for instance, when doing SSL
shutdown of a long standing connection after rejecting another one by SNI.
The fix is to move the qc->error check after c->ssl->handshake_rejected is
handled first, to make the error queue cleared. Although not practicably
visible as needed, this is accompanied by clearing the error queue under
the qc->error case as well, to be on the safe side.
As an implementation note, due to the way of handling invalid transport
parameters for OpenSSL 3.5 and above, which leaves a passed pointer not
advanced on error, SSL_get_error() may return either SSL_ERROR_WANT_READ
or SSL_ERROR_WANT_WRITE depending on a library. To cope with that, both
qc->error and c->ssl->handshake_rejected checks were moved out of
"sslerr != SSL_ERROR_WANT_READ".
Also, this reconstructs a missing "SSL_do_handshake() failed" diagnostics
for the qc->error case, replacing using ngx_ssl_connection_error() with
ngx_connection_error(). It is made this way to avoid logging at the crit
log level because qc->error set is expected to have an empty error queue.
Reported and tested by Vladimir Homutov.
They might be reused in a session if an SMTP client proceeded
unauthenticated after previous invalid authentication attempts.
This could confuse an authentication server when passing stale
credentials along with "Auth-Method: none".
The condition to send the "Auth-Salt" header is similarly refined.
The ssl_certificate_compression directive allows to send compressed
server certificates. In OpenSSL, they are pre-compressed on startup.
To simplify configuration, the SSL_OP_NO_TX_CERTIFICATE_COMPRESSION
option is automatically cleared if certificates were pre-compressed.
SSL_CTX_compress_certs() may return an error in legitimate cases,
e.g., when none of compression algorithms is available or if the
resulting compressed size is larger than the original one, thus it
is silently ignored.
Certificate compression is supported in Chrome with brotli only,
in Safari with zlib only, and in Firefox with all listed algorithms.
It is supported since Ubuntu 24.10, which has OpenSSL with enabled
zlib and zstd support.
The actual list of algorithms supported in OpenSSL depends on how
the library was configured; it can be brotli, zlib, zstd as listed
in RFC 8879.
Certificate compression is supported since OpenSSL 3.2, it is enabled
automatically as negotiated in a TLSv1.3 handshake.
Using certificate compression and decompression in runtime may be
suboptimal in terms of CPU and memory consumption in certain typical
scenarios, hence it is disabled by default on both server and client
sides. It can be enabled with ssl_conf_command and similar directives
in upstream as appropriate, for example:
ssl_conf_command Options RxCertificateCompression;
ssl_conf_command Options TxCertificateCompression;
Compressing server certificates requires additional support, this is
addressed separately.
The function contains mostly HTTP/1.x specific request processing,
which has no use in other protocols. After the previous change in
HTTP/2, it can now be hidden.
This is an API change.
Previously, it misused the Host header processing resulting in
400 (Bad Request) errors for a valid request that contains both
":authority" and Host headers with the same value, treating it
after 37984f0be as if client sent more than one Host header.
Such an overly strict handling violates RFC 9113.
The fix is to process ":authority" as a distinct header, similarly
to processing an authority component in the HTTP/1.x request line.
This allows to disambiguate and compare Host and ":authority"
values after all headers were processed.
With this change, the ngx_http_process_request_header() function
can no longer be used here, certain parts were inlined similar to
the HTTP/3 module.
To provide compatibility for misconfigurations that use $http_host
to return the value of the ":authority" header, the Host header,
if missing, is now reconstructed from ":authority".
This might happen for Huffman encoded string literals as the result
of length expansion. Notably, the maximum length of string literals
is already limited with the "large_client_header_buffers" directive,
so this was only possible with nonsensically large configured limits.
The kevent udata field was changed from intptr_t to "void *",
similar to other BSDs and Darwin.
The NGX_KQUEUE_UDATA_T macro is adjusted to reflect that change,
fixing -Werror=int-conversion errors.
The NGX_KQUEUE_UDATA_T macro is used to compensate the incompatible
kqueue() API in NetBSD, it doesn't really belong to feature tests.
The change limits the macro visibility to the kqueue event module.
Moving from autotests also simplifies testing a particular NetBSD
version as seen in a subsequent change.
The kevent udata field is special in that we maintain compatibility
with NetBSD versions that predate using the "void *" type.
The fix is to cast to intermediate uintptr_t that is casted back to
"void *" where appropriate.
Prior to OpenSSL 3.0, OPENSSL_VERSION_NUMBER used the following format:
MNNFFPPS: major minor fix patch status
Where the status nibble (S) has 0+ for development and f for release.
The format was changed in OpenSSL 3.0.0, where it is always zero:
MNN00PP0: major minor patch
Disabling the build dependency feature is safe assuming that
nginx/Windows release zip is always built from a clean tree.
This allows to speed up total build time by around 40%.
As it may not be suitable in general, the option resides here
and not in configure.
In OpenSSL 3.5.0, the "quic_transport_parameters" extension set
internally by the QUIC API is cleared on the SSL context switch,
which disables sending QUIC transport parameters if switching to
a different server block on SNI. See the initial report in [1].
This is fixed post OpenSSL 3.5.0 [2]. The fix is anticipated in
OpenSSL 3.5.1, which has not been released yet. When building
with OpenSSL 3.5, OpenSSL compat layer is now used by default.
The OpenSSL 3.5 QUIC API support can be switched back using
--with-cc-opt='-DNGX_QUIC_OPENSSL_API=1'.
[1] https://github.com/nginx/nginx/issues/711
[2] https://github.com/openssl/openssl/commit/45bd3c3798
The gRPC module context has connection specific state, which can be lost
after request reinitialization when it comes to processing early hints.
The fix is to do only a portion of u->reinit_request() implementation
required after processing early hints, now inlined in modules.
Now NGX_HTTP_UPSTREAM_EARLY_HINTS is returned from u->process_header()
for early hints. When reading a cached response, this code is mapped
to NGX_HTTP_UPSTREAM_INVALID_HEADER to indicate invalid header format.
Sergey Kandaurov
