A predicate location name starts with '$' and is interpreted as a
string evaluated at runtime, and its value is used to route the
request into the location. A predicate location matches if the
string evaluates to a non-empty value not equal to "0". At each
nesting level, the first matching predicate location in the
configuration order is chosen. Predicate locations are matched
after prefix and regex locations. They are not matched if an exact
or a regex location matched, or if the longest matching prefix location
has the "^~" modifier. Predicate locations can be nested inside
other location types and can contain nested locations of their own.
Example:
map $arg_foo $pred {
bar 1;
qux 1;
}
location $pred {
root html;
}
Previously, when the worker file descriptor table was full, a listen
connection might be disabled and a timeout might be set for re-enabling
it. If during this period the worker received the NGX_CMD_QUIT signal,
the accept connection would be recycled, but the timer would remain set.
On timer expiration, an attempt to accept a new connection would result
in "accept4() failed (9: Bad file descriptor)" alert because c->fd was
reset to -1.
Fix is to delete accept timer when closing listen connections.
If a worker process' file descriptor table is full, recvmsg() is not
able to read out the channel fd sent from the master. This results in
different behavior on different platforms.
On Linux, recvmsg() succeeds, returns the payload and drops the fd.
On BSD systems, the first recvmsg() call fails with EMSGSIZE/EMFILE,
but the next call succeeds and returns the payload without the fd.
On Solaris, recvmsg() fails with EMFILE and drops both the payload
and the fd.
Previously, all of these paths resulted in fatal errors. The worker
channel was closed and the worker was no longer capable of receiving
messages from the master. In particular, the NGX_CMD_QUIT command
could not be delivered anymore. As a result, the worker never exited
after nginx reload and kept running and accepting new connections
forever until killed by a signal.
Fix is to consider those errors non-fatal. An additional check is
added to the NGX_CMD_CLOSE_CHANNEL handler to make sure the worker
has received a file descriptor for this entry before. Also, the
MSG_CTRUNC check is removed since a proper ancillary data size
validation is now implemented.
The directive accepts predicates. When enabled, request body is read
immediately after the headers and before choosing a location.
Body read settings from the server block are used in this case.
Example:
http {
map $http_content_type $is_json {
application/json 1;
}
server {
listen 8000;
client_body_early_read $is_json;
client_max_body_size 256;
client_body_buffer_size 256;
return 200 $request_body;
}
}
HTTP/2 and gRPC requests are now always sent with the ":authority"
pseudo-header, while the "Host" header is no longer passed.
Where a host value is set explicitly with "proxy_set_header Host"
or "grpc_set_header Host", that value is conveyed in ":authority".
As per RFC 9113, Section 8.3.1, clients and intermediaries MUST
generate an ":authority" pseudo-header to convey authority information.
For HTTP/1.1, HTTP/2 and gRPC, if the explicit value is empty, then
the value of $proxy_host is used instead. As per RFC 9110, Sections
4.2.1 and 4.2.2, the "http" and "https" URI schemes MUST contain a
non-empty host identifier. The change enforces this rule by
introducing a default.
Closes: https://github.com/nginx/nginx/issues/1426
As per RFC 9000, Section 4.5:
Once a final size for a stream is known, it cannot change. If a
RESET_STREAM or STREAM frame is received indicating a change in the
final size for the stream, an endpoint SHOULD respond with an error of
type FINAL_SIZE_ERROR. A receiver SHOULD treat receipt of data at or
beyond the final size as an error of type FINAL_SIZE_ERROR, even after
a stream is closed.
Previously, the stream final size was validated only after other checks
that could return earlier and skip validation: the receive state check
and, for STREAM frames, the check for data below the current receive
offset. As a result, a STREAM or RESET_STREAM frame carrying a final
size error was silently ignored once the receiving part of the stream
had reached a state in which the frame would otherwise be discarded, or
when a STREAM frame did not advance the receive offset. Now the final
size is validated first, so a FINAL_SIZE_ERROR is generated in these
cases as well.
Previously, stream flow control was not applied to the RESET_STREAM
final_size, which allowed a client to exceed it. The excess had to be
within the connection flow control limits anyway.
Reported by Tony Wang.
Previously, if a subrequest was posted twice, it could be finalized in
both calls, excessively reducing r->main->count and potentially leading
to a use-after-free.
The fix is to avoid posting a request if it's already posted. Also,
as a hardening measure, r->write_event_handler is now reset to a no-op
handler during active subrequest finalization.
The problem manifests itself in ngx_http_ssi_filter_module during
unbuffered proxying. If a subrequest is created for an SSI include
statement while the main request has some data postponed by another
include, this subrequest becomes double-posted when the main request
data is flushed. The first post comes from ngx_http_subrequest() and
the second one comes from ngx_http_postpone_filter(). In case of a
quick subrequest finalization, the above mentioned problem happens.
Reported by P4P3R-HAK.
If the script result turned out to be shorter than its predicted
length, the result string contained uninitialized bytes at the end.
The fix is to cut the result string by its actual size.
The following locations returned trailing garbage to the client with
URI "/1234abcd".
map $uri $foo {
~^/(?<bar>[0-9]).*$ $bar;
}
location ~(?<bar>[0-9]*)[a-z]*$ {
return 200 $1:$foo;
}
location ~(?<bar>[0-9]*)[a-z]*$ {
set $qux $1:$foo;
return 200 $qux;
}
As per RFC 2817 Section 5.2:
Like any other pipelined HTTP/1.1 request, data to be tunneled may be
sent immediately after the blank line. The usual caveats also apply:
data may be discarded if the eventual response is negative, and the
connection may be reset with no response if more than one TCP segment
is outstanding.
As per RFC 9110, Section 9.3.6:
A CONNECT request message does not have content.
Also, as per Section 8.6:
A user agent SHOULD NOT send a Content-Length header field when the
request message does not contain content and the method semantics do
not anticipate such data.
The fix includes the socket option level (IPPROTO_IPV6) in the feature
test and the macro (NGX_HAVE_IPV6_DONTFRAG) in the
ngx_configure_listening_sockets() function.
Reported by Eric Fortis.
Creating a control/encoder/decoder stream while another such stream
already exists, is not allowed. Also, closing such a stream results
in connection closure with NGX_HTTP_V3_ERR_CLOSED_CRITICAL_STREAM.
However, since stream creation and connection closure are asynchronous,
there could be a window where two control/encoder/decoder streams
could coexist within a single cycle iteration. This could result
in reusing parsing context, such as encoder insert buffer.
The change adds a mask for all standard client uni streams ever created.
This allows to check if a stream of this type was created before.
While here, mandatory stream validation now also uses this mask.
Previously, it was allocated from the encoder stream pool. This could
lead to use-after-free if the stream was closed and another encoder
stream was opened.
Reported by Trung Nguyen (@everping) of CyStack.
When the rewrite replacement string had no variables, but had
overlapping captures, the length of the allocated buffer could be
smaller than the replacement string. This could happen either
when the "redirect" parameter is specified, or when arguments are
present in the replacement string.
The following configurations resulted in heap buffer overflow when
using URI "/++++++++++++++++++++++++++++++":
location / {
rewrite ^/((.*))$ http://127.0.0.1:8080/$1$2 redirect;
return 200 foo;
}
location / {
rewrite ^/((.*))$ http://127.0.0.1:8080/?$1$2;
return 200 foo;
}
Reported by Mufeed VH of Winfunc Research.
Following 2046b45aa0, this change introduces better control of memory
allocation flags for escaped values. Notably:
- The e->is_args flag is now explicitly reset on rewrite start.
If the flag was set prior to rewrite start, then buffer overflow
could happen before 2046b45aa0.
- The le->is_args flag value is now copied from e->is_args when
calculating complex value length for "if" and "set" directives.
If e->is_args was set, but le->is_args was not, then buffer overflow
could happen before 2046b45aa0.
Previously, when ngx_handle_read_event() or ngx_handle_write_event()
returned an error while handling an SMTP, POP3 or IMAP session, the
released session memory could be accessed after handling the error.
Previously, when these fields were larger than ~2M, the number of bytes
allocated for the field length was insufficient for such a large number.
The deficit is 1 byte up until ~4M, 2 bytes for sizes above, and grows
bigger with even larger fields.
Currently, nginx does not have modules which allow to exploit this
overflow with reasonably large Content-Type and Location. The reason is
other response fields make up for this deficit. For example, the Date
header value contains the characters compressed well by Huffman
encoding, which frees up spare bytes in the header buffer.
Reported by Leo Lin.
Previously, if proxy_set_body was used with HTTP/2, and body size exceeded
16M, then an overflow happened in the 24-bit DATA frame size, which resulted
in sending unframed bytes, potentially allowing for an injection.
Also, DATA frame size could exceed NGX_HTTP_V2_DEFAULT_FRAME_SIZE (16K) and
available send window.
Reported by Mufeed VH of Winfunc Research.
The following code resulted in incorrect escaping of $1 and possible
segfault:
location / {
rewrite ^(.*) /new?c=1;
set $myvar $1;
return 200 $myvar;
}
If there were arguments in a rewrite's replacement string, the is_args flag
was set and incorrectly never cleared. This resulted in escaping applied
to any captures evaluated afterwards in set or if. Additionally buffer was
allocated by ngx_http_script_complex_value_code() without escaping expected,
thus this also resulted in buffer overrun and possible segfault.
A similar issue was fixed in 74d939974d.
Reported by Leo Lin.
Previously, when a client migrated to a new address, new QUIC streams
received this address before validation. This allowed an attacker to
create QUIC streams with a spoofed address.
Reported by Rodrigo Laneth.
Previously, when a client SSL connection was terminated (typically due to a
timeout) while resolving an OCSP responder, the OCSP context was freed, but
the resolve context was not. This resulted in use-after-free on resolve
completion.
Reported by Leo Lin.
Notably, "auth_delay" now delays the response for both 401 and 407.
Also, in the "satisfy any" mode, the next access/auth attempt is made
for 401, 403 and 407.
Previously, when an HTTP/2 request had no body or an explicit body
was set by proxy_set_body, the request consisted of only one buffer,
which had no b->last_buf flag set. This prevented ctx->output_closed
from being set after processing this buffer. Consequently,
u->keepalive might not be set to store the connection in the
keepalive cache.
Previously, the encoder stream allocated each new inserted field in the
connection pool. This memory was not freed until the end of the connection.
Now a special insert buffer is used for all inserts.
As per RFC 9113 and RFC 9114, any message containing such headers MUST be
treated as malformed.
As per RFC 9110, Section 7.6.1, the following headers are considered
connection-specific:
- Connection
- Proxy-Connection
- Keep-Alive
- TE
- Transfer-Encoding
- Upgrade
The only exception is the TE header field, which MAY be present in a
request header, but it MUST NOT contain any value other than "trailers".
Now host name resolved from client address is validated to only contain
the characters specified in RFC 1034, Section 3.5. The validation allows
to avoid injections when using the resolved host name in auth_http and
smtp proxy.
Reported by Asim Viladi Oglu Manizada, Colin Warren,
Xiao Liu (Yunnan University), Yuan Tan (UC Riverside), and
Bird Liu (Lanzhou University).
Previously, when alias was used in a location with Dav COPY or MOVE
enabled, and the destination URI was shorter than the alias, integer
underflow could happen in ngx_http_map_uri_to_path(), which could
result in heap buffer overwrite, followed by a possible segfault.
With some implementations of memcpy(), the segfault could be avoided
and the overwrite could result in a change of the source or destination
file names to be outside of the location root.
Reported by Calif.io in collaboration with Claude and Anthropic Research.
Previously, a 32-bit overflow could happen while validating atom entries
count. This allowed processing of an invalid atom with entrires beyond
its boundaries with reads and writes outside of the allocated mp4 buffer.
Reported by Prabhav Srinath (sprabhav7).
Previously, data validation checks did not cover the cases when the output
contained empty buffers. Such buffers are considered illegal and produce
"zero size buf in output" alerts. The change rejects the mp4 files which
produce such alerts.
Also, the change fixes possible buffer overread and overwrite that could
happen while processing empty stco and co64 atoms, as reported by
Pavel Kohout (Aisle Research) and Tim Becker.
Previously, it was possible to obtain a stateless reset token for a
connection by routing its packet to a wrong worker. This allowed to
terminate the connection.
The fix is to bind stateless reset token to the worker number.
Previously, when proxy_cache and keepalive were both enabled with an
HTTP/2 upstream, the second request for a cached resource could fail with
"upstream sent frame for unknown stream" error followed by "cache file
contains invalid header".
This happened because ctx->id was set to 1 in the case when no upstream
connection exists (e.g. cache hit), making the stream id check fail when
the cached response contained frames from a different stream.
The fix is to set ctx->id to 0 when there is no upstream connection,
indicating that no real stream exists, and skip the stream id validation
in this case. Also, ctx->id = 1 is now set only for new connections,
not in the shared done label.
Closes: https://github.com/nginx/nginx/issues/1101
Previously, when connecting to a backend, if the read event handler was
called before the write event handler, and the received response triggered
a next upstream condition, then ngx_http_upstream_reinit() was not called
to clean up the old upstream context. This had multiple implications.
For all proxy modules, since the last upstream response was not cleaned up,
it was mixed with the next upstream response. This could result in ignoring
the second response status code, duplicate response headers or reporting
old upstream header errors.
With ngx_http_grpc_module and ngx_http_proxy_v2_module, ctx->connection
was left dangling since the object it referenced was allocated from the
last upstream connection pool, which was deleted when freeing last upstream.
This lead to use-after-free when trying to reuse this object for the next
upstream.
When connecting to a backend, the connection write event is triggered
first in most cases. However if a response arrives quickly enough, both
read and write events can be triggered together within the same event loop
iteration. In this case the read event handler is called first and the
write event handler is called after it.
SSL initialization for backend connections happens only in the write event
handler since SSL handshake starts with sending Client Hello. Previously,
if a backend sent a quick plain text response, it could be parsed by the
read event handler prior to starting SSL handshake on the connection.
The change adds protection against parsing such responses on SSL-enabled
connections.
The new directives add_header_inherit and add_trailer_inherit allow
to alter inheritance rules for the values specified in the add_header
and add_trailer directives in a convenient way.
The "merge" parameter enables appending the values from the previous level
to the current level values.
The "off" parameter cancels inheritance of the values from the previous
configuration level, similar to add_header "" (2194e75bb).
The "on" parameter (default) enables the standard inheritance behaviour,
which is to inherit values from the previous level only if there are no
directives on the current level.
The inheritance rules themselves are inherited in a standard way. Thus,
for example, "add_header_inherit merge;" specified at the top level will
be inherited in all nested levels recursively unless redefined below.
After f10bc5a763 the address was set to NULL only when local address was
not specified at all. In case complex value evaluated to an empty or
invalid string, local address remained unchanged. Currenrly this is not
a problem since the value is only set once. This change is a preparation
for being able to change the local address after initial setting.
The change allows modules to use the CONNECT method with HTTP/1.1 requests.
To do so, they need to set the "allow_connect" flag in the core server
configuration.
The $request_port variable contains the port passed by the client in the
request line (for HTTP/1.x) or ":authority" pseudo-header (for HTTP/2 and
HTTP/3). If the request line contains no host, or ":authority" is missing,
then $request_port is taken from the "Host" header, similar to the $host
variable.
The $is_request_port variable contains ":" if $request_port is non-empty,
and is empty otherwise.
Previously, when using HTTP/2 over SSL, an early hints HEADERS frame was
queued in SSL buffer, and might not be immediately flushed. This resulted
in a delay of early hints delivery until the main response was sent.
The fix is to set the flush flag for the early hints HEADERS frame buffer.
RFC 9114, Section 4.3.1. specifies a restriction for :authority and Host
coexistence in an HTTP/3 request:
: If both fields are present, they MUST contain the same value.
Previously, this restriction was correctly enforced only for portless
values. When Host contained a port, the request failed as if :authority
and Host were different, regardless of :authority presence.
This happens because the value of r->headers_in.server used for :authority
has port stripped. The fix is to use r->host_start / r->host_end instead.
The change implements processing upstream early hints response in
ngx_http_proxy_module and ngx_http_grpc_module. A new directive
"early_hints" enables sending early hints to the client. By default,
sending early hints is disabled.
Example:
map $http_sec_fetch_mode $early_hints {
navigate $http2$http3;
}
early_hints $early_hints;
proxy_pass http://example.com;
After fixing ngx_http_v3_encode_varlen_int() in 400eb1b628,
NGX_HTTP_V3_VARLEN_INT_LEN retained the old value of 4, which is
insufficient for the values over 1073741823 (1G - 1).
The NGX_HTTP_V3_VARLEN_INT_LEN macro is used in ngx_http_v3_uni.c to
format stream and frame types. Old buffer size is enough for formatting
this data. Also, the macro is used in ngx_http_v3_filter_module.c to
format output chunks and trailers. Considering output_buffers and
proxy_buffer_size are below 1G in all realistic scenarios, the old buffer
size is enough here as well.
RFC 9002, Section 6.1.1 defines packet reordering threshold as 3. Testing
shows that such low value leads to spurious packet losses followed by
congestion window collapse. The change implements dynamic packet threshold
detection based on in-flight packet range. Packet threshold is defined
as half the number of in-flight packets, with mininum value of 3.
Also, renamed ngx_quic_lost_threshold() to ngx_quic_time_threshold()
for better compliance with RFC 9002 terms.
Previosly the threshold was hardcoded at 10000. This value is too low for
high BDP networks. For example, if all frames are STREAM frames, and MTU
is 1500, the upper limit for congestion window would be roughly 15M
(10000 * 1500). With 100ms RTT it's just a 1.2Gbps network (15M * 10 * 8).
In reality, the limit is even lower because of other frame types. Also,
the number of frames that could be used simultaneously depends on the total
amount of data buffered in all server streams, and client flow control.
The change sets frame threshold based on max concurrent streams and stream
buffer size, the product of which is the maximum number of in-flight stream
data in all server streams at any moment. The value is divided by 2000 to
account for a typical MTU 1500 and the fact that not all frames are STREAM
frames.
If connection is network-limited, MTU probes have little chance of being
sent since congestion window is almost always full. As a result, PMTUD
may not be able to reach the real MTU and the connection may operate with
a reduced MTU. The solution is to ignore the congestion window. This may
lead to a temporary increase in in-flight count beyond congestion window.
As per RFC 9000, Section 14.4:
Loss of a QUIC packet that is carried in a PMTU probe is therefore
not a reliable indication of congestion and SHOULD NOT trigger a
congestion control reaction.
Previously, these functions operated on a per-level basis. This however
resulted in excessive logging of in_flight and will also led to extra
work detecting underutilized congestion window in the followup patches.
On some systems the value of ngx_current_msec is derived from monotonic
clock, for which the following is defined by POSIX:
For this clock, the value returned by clock_gettime() represents
the amount of time (in seconds and nanoseconds) since an unspecified
point in the past.
As as result, overflow protection is needed when comparing two ngx_msec_t.
The change adds such protection to the ngx_quic_detect_lost() function.
Since recovery_start field was initialized with ngx_current_msec, all
congestion events that happened within the same millisecond or cycle
iteration, were treated as in recovery mode.
Also, when handling persistent congestion, initializing recovery_start
with ngx_current_msec resulted in treating all sent packets as in recovery
mode, which violates RFC 9002, see example in Appendix B.8.
While here, also fixed recovery_start wrap protection. Previously it used
2 * max_idle_timeout time frame for all sent frames, which is not a
reliable protection since max_idle_timeout is unrelated to congestion
control. Now recovery_start <= now condition is enforced. Note that
recovery_start wrap is highly unlikely and can only occur on a
32-bit system if there are no congestion events for 24 days.
Previously, the expiration caused QUIC connection finalization even if
there are application-terminated streams finishing sending data. Such
finalization terminated these streams.
An easy way to trigger this is to request a large file from HTTP/3 over
a small MTU. In this case keepalive timeout expiration may abruptly
terminate the request stream.
Improved logging for simpler data extraction for plotting congestion
window graphs. In particular, added current milliseconds number from
ngx_current_msec.
While here, simplified logging text and removed irrelevant data.
The directive sets a timeout during which a keepalive connection will
not be closed by nginx for connection reuse or graceful shutdown.
The change allows clients that send multiple requests over the same
connection without delay or with a small delay between them, to avoid
receiving a TCP RST in response to one of them. This excludes network
issues and non-graceful shutdown. As a side-effect, it also addresses
the TCP reset problem described in RFC 9112, Section 9.6, when the last
sent HTTP response could be damaged by a followup TCP RST. It is important
for non-idempotent requests, which cannot be retried by client.
It is not recommended to set keepalive_min_timeout to large values as
this can introduce an additional delay during graceful shutdown and may
restrict nginx from effective connection reuse.
While trying to close a stream in ngx_quic_close_streams() by calling its
read event handler, the next stream saved prior to that could be destroyed
recursively. This caused a segfault while trying to access the next stream.
The way the next stream could be destroyed in HTTP/3 is the following.
A request stream read event handler ngx_http_request_handler() could
end up calling ngx_http_v3_send_cancel_stream() to report a cancelled
request stream in the decoder stream. If sending stream cancellation
decoder instruction fails for any reason, and the decoder stream is the
next in order after the request stream, the issue is triggered.
The fix is to postpone calling read event handlers for all streams being
closed to avoid closing a released stream.
Previously, such packets were treated as long header packets with unknown
version 0, and a version negotiation packet was sent in response. This
could be used to set up an infinite traffic reflect loop with another nginx
instance.
Now version negotiation packets are ignored. As per RFC 9000, Section 6.1:
An endpoint MUST NOT send a Version Negotiation packet in response to
receiving a Version Negotiation packet.
When client address is received, IPv6 address could be specified without
square brackets and without port, as well as both with the brackets and
port. The change allows IPv6 in square brackets and no port, which was
previously considered an error. This format conforms to RFC 3986.
The change also affects proxy_bind and friends.
When cropping stsc atom, it's assumed that chunk index is never 0.
Based on this assumption, start_chunk and end_chunk are calculated
by subtracting 1 from it. If chunk index is zero, start_chunk or
end_chunk may underflow, which will later trigger
"start/end time is out mp4 stco chunks" error. The change adds an
explicit check for zero chunk index to avoid underflow and report
a proper error.
Zero chunk index is explicitly banned in ISO/IEC 14496-12, 8.7.4
Sample To Chunk Box. It's also implicitly banned in QuickTime File
Format specification. Description of chunk offset table references
"Chunk 1" as the first table element.
Currently an error is triggered if any of the chunk runs in stsc are
unordered. This however does not include the final chunk run, which
ends with trak->chunks + 1. The previous chunk index can be larger
leading to a 32-bit overflow. This could allow to skip the validity
check "if (start_sample > n)". This could later lead to a large
trak->start_chunk/trak->end_chunk, which would be caught later in
ngx_http_mp4_update_stco_atom() or ngx_http_mp4_update_co64_atom().
While there are no implications of the validity check being avoided,
the change still adds a check to ensure the final chunk run is ordered,
to produce a meaningful error and avoid a potential integer overflow.
A specially crafted mp4 file with an empty run of chunks in the stsc atom
and a large value for samples per chunk for that run, combined with a
specially crafted request, allowed to store that large value in prev_samples
and later in trak->end_chunk_samples while in ngx_http_mp4_crop_stsc_data().
Later in ngx_http_mp4_update_stsz_atom() this could result in buffer
overread while calculating trak->end_chunk_samples_size.
Now the value of samples per chunk specified for an empty run is ignored.
Unordered chunks could result in trak->end_chunk smaller than trak->start_chunk
in ngx_http_mp4_crop_stsc_data(). Later in ngx_http_mp4_update_stco_atom()
this caused buffer overread while trying to calculate trak->end_offset.
Roman Arutyunyan