Input filter might free a buffer if there is no data in it, and in case
of first buffer (used for cache header and request header, aka p->buf_to_file)
this resulted in cache corruption. Buffer memory was reused to read upstream
response before headers were written to disk.
Fix is to avoid moving pointers in ngx_event_pipe_add_free_buf() to a buffer
start if we were asked to free a buffer used by p->buf_to_file.
This fixes occasional cache file corruption, usually resulted
in "cache file ... has md5 collision" alerts.
Reported by Anatoli Marinov.
With previous code raw buffer might be lost if p->input_filter() was called
on a buffer without any data and used ngx_event_pipe_add_free_buf() to
return it to the free list. This eventually might cause "all buffers busy"
problem, resulting in segmentation fault due to null pointer dereference in
ngx_event_pipe_write_chain_to_temp_file().
In ngx_event_pipe_add_free_buf() the buffer was added to the list start
due to pos == last, and then "p->free_raw_bufs = cl->next" in
ngx_event_pipe_read_upstream() dropped both chain links to the buffer
from the p->free_raw_bufs list.
Fix is to move "p->free_raw_bufs = cl->next" before calling the
p->input_filter().
If possible we now just extend already present file buffer in p->out chain
instead of keeping ngx_buf_t for each buffer we've flushed to disk. This
saves about 120 bytes of memory per buffer flushed to disk, and resolves
high CPU usage observed in edge cases (due to coalescing these buffers on
send).
1. In ngx_event_pipe_write_chain_to_temp_file() make sure to fully write
all shadow buffers up to last_shadow. With this change recycled buffers
cannot appear in p->out anymore. This also fixes segmentation faults
observed due to ngx_event_pipe_write_chain_to_temp() not freeing any
raw buffers while still returning NGX_OK.
2. In ngx_event_pipe_write_to_downstream() we now properly check for busy
size as a size of buffers, not a size of data in these buffers. This
fixes situations where all available buffers became busy (including
segmentation faults due to this).
3. The ngx_event_pipe_free_shadow_raw_buf() function is dropped. It's
incorrect and not needed.
If client closed connection in ngx_event_pipe_write_to_downstream(), buffers
in the "out" chain were lost. This caused cpu hog if all available buffers
were in the "out" chain. Fix is to call ngx_chain_update_chains() before
checking return code of output filter to avoid loosing buffers in the "out"
chain.
Note that this situation (all available buffers in the "out" chain) isn't
normal, it should be prevented by busy buffers limit. Though right now it
may happen with complex protocols like fastcgi. This should be addressed
separately.
As long as ngx_event_pipe() has more data read from upstream than specified
in p->length it's passed to input filter even if buffer isn't yet full. This
allows to process data with known length without relying on connection close
to signal data end.
By default p->length is set to -1 in upstream module, i.e. end of data is
indicated by connection close. To set it from per-protocol handlers upstream
input_filter_init() now called in buffered mode (as well as in
unbuffered mode).
The ngx_chain_update_chains() needs pool to free chain links used for buffers
with non-matching tags. Providing one helps to reduce memory consumption
for long-lived requests.
Setting read->eof to 0 seems to be just a typo. It appeared in
nginx-0.0.1-2003-10-28-18:45:41 import (r164), while identical code in
ngx_recv.c introduced in the same import do actually set read->eof to 1.
Failure to set read->eof to 1 results in EOF not being generally detectable
from connection flags. On the other hand, kqueue won't report any read
events on such a connection since we use EV_CLEAR. This resulted in read
timeouts if such connection was cached and used for another request.
*) Feature: the "restrict_host_names" directive was canceled.
*) Feature: the --with-cpu-opt=ppc64 configuration parameter.
*) Bugfix: on some condition the proxied connection with a client was
terminated prematurely.
Thanks to Vladimir Shutoff.
*) Bugfix: the "X-Accel-Limit-Rate" header line was not taken into
account if the request was redirected using the "X-Accel-Redirect"
header line.
*) Bugfix: the "post_action" directive ran only after a successful
completion of a request.
*) Bugfix: the proxied response body generated by the "post_action"
directive was transferred to a client.
*) Feature: the new 444 code of the "return" directive to close
connection.
*) Feature: the "so_keepalive" directive in IMAP/POP3 proxy.
*) Bugfix: if there are unclosed connection nginx now calls abort()
only on gracefull quit and active "debug_points" directive.
*) Feature: the IMAP/POP3 proxy supports STARTTLS and STLS.
*) Bugfix: the IMAP/POP3 proxy did not work with the select, poll, and
/dev/poll methods.
*) Bugfix: in SSI handling.
*) Bugfix: now Solaris sendfilev() is not used to transfer the client
request body to FastCGI-server via the unix domain socket.
*) Bugfix: the "auth_basic" directive did not disable the
authorization; the bug had appeared in 0.3.11.
*) Security: if nginx was built with the ngx_http_realip_module and the
"satisfy_any on" directive was used, then access and authorization
directives did not work. The ngx_http_realip_module was not built
and is not built by default.
*) Change: the "$time_gmt" variable name was changed to "$time_local".
*) Change: the "proxy_header_buffer_size" and
"fastcgi_header_buffer_size" directives was renamed to the
"proxy_buffer_size" and "fastcgi_buffer_size" directives.
*) Feature: the ngx_http_memcached_module.
*) Feature: the "proxy_buffering" directive.
*) Bugfix: the changes in accept mutex handling when the "rtsig" method
was used; the bug had appeared in 0.3.0.
*) Bugfix: if the client sent the "Transfer-Encoding: chunked" header
line, then nginx returns the 411 error.
*) Bugfix: if the "auth_basic" directive was inherited from the http
level, then the realm in the "WWW-Authenticate" header line was
without the "Basic realm" text.
*) Bugfix: if the "combined" format was explicitly specified in the
"access_log" directive, then the empty lines was written to the log;
the bug had appeared in 0.3.8.
*) Bugfix: nginx did not run on the sparc platform under any OS except
Solaris.
*) Bugfix: now it is not necessary to place space between the quoted
string and closing bracket in the "if" directive.
*) Change: the "valid_referers" directive and the "$invalid_referer"
variable were moved to the new ngx_http_referer_module from the
ngx_http_rewrite_module.
*) Change: the "$apache_bytes_sent" variable name was changed to
"$body_bytes_sent".
*) Feature: the "$sent_http_..." variables.
*) Feature: the "if" directive supports the "=" and "!=" operations.
*) Feature: the "proxy_pass" directive supports the HTTPS protocol.
*) Feature: the "proxy_set_body" directive.
*) Feature: the "post_action" directive.
*) Feature: the ngx_http_empty_gif_module.
*) Feature: the "worker_cpu_affinity" directive for Linux.
*) Bugfix: the "rewrite" directive did not unescape URI part in
redirect, now it is unescaped except the %00-%25 and %7F-%FF
characters.
*) Bugfix: nginx could not be built by the icc 9.0 compiler.
*) Bugfix: if the SSI was enabled for zero size static file, then the
chunked response was encoded incorrectly.
*) Feature: the "limit_rate" directive is supported in in proxy and
FastCGI mode.
*) Feature: the "X-Accel-Limit-Rate" response header line is supported
in proxy and FastCGI mode.
*) Feature: the "break" directive.
*) Feature: the "log_not_found" directive.
*) Bugfix: the response status code was not changed when request was
redirected by the ""X-Accel-Redirect" header line.
*) Bugfix: the variables set by the "set" directive could not be used
in SSI.
*) Bugfix: the segmentation fault may occurred if the SSI page has more
than one remote subrequest.
*) Bugfix: nginx treated the backend response as invalid if the status
line in the header was transferred in two packets; the bug had
appeared in 0.1.29.
*) Feature: the "ssi_types" directive.
*) Feature: the "autoindex_exact_size" directive.
*) Bugfix: the ngx_http_autoindex_module did not support the long file
names in UTF-8.
*) Feature: the IMAP/POP3 proxy.
*) Feature: the "working_directory" directive.
*) Feature: the "port_in_redirect" directive.
*) Bugfix: the segmentation fault was occurred if the backend response
header was in several packets; the bug had appeared in 0.1.29.
*) Bugfix: if more than 10 servers were configured or some server did
not use the "listen" directive, then the segmentation fault was
occurred on the start.
*) Bugfix: the segmentation fault might occur if the response was
bigger than the temporary file.
*) Bugfix: nginx returned the 400 response on requests like
"GET http://www.domain.com/uri HTTP/1.0"; the bug had appeared in
0.1.28.
*) Feature: the ngx_http_ssi_module supports "include virtual" command.
*) Feature: the ngx_http_ssi_module supports the condition command like
'if expr="$NAME"' and "else" and "endif" commands. Only one nested
level is supported.
*) Feature: the ngx_http_ssi_module supports the DATE_LOCAL and
DATE_GMT variables and "config timefmt" command.
*) Feature: the "ssi_ignore_recycled_buffers" directive.
*) Bugfix: the "echo" command did not show the default value for the
empty QUERY_STRING variable.
*) Change: the ngx_http_proxy_module was rewritten.
*) Feature: the "proxy_redirect", "proxy_pass_request_headers",
"proxy_pass_request_body", and "proxy_method" directives.
*) Feature: the "proxy_set_header" directive. The "proxy_x_var" was
canceled and must be replaced with the proxy_set_header directive.
*) Change: the "proxy_preserve_host" is canceled and must be replaced
with the "proxy_set_header Host $host" and the "proxy_redirect off"
directives, the "proxy_set_header Host $host:$proxy_port" directive
and the appropriate proxy_redirect directives.
*) Change: the "proxy_set_x_real_ip" is canceled and must be replaced
with the "proxy_set_header X-Real-IP $remote_addr" directive.
*) Change: the "proxy_add_x_forwarded_for" is canceled and must be
replaced with
the "proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for"
directive.
*) Change: the "proxy_set_x_url" is canceled and must be replaced with
the "proxy_set_header X-URL http://$host:$server_port$request_uri"
directive.
*) Feature: the "fastcgi_param" directive.
*) Change: the "fastcgi_root", "fastcgi_set_var" and "fastcgi_params"
directive are canceled and must be replaced with the fastcgi_param
directives.
*) Feature: the "index" directive can use the variables.
*) Feature: the "index" directive can be used at http and server levels.
*) Change: the last index only in the "index" directive can be absolute.
*) Feature: the "rewrite" directive can use the variables.
*) Feature: the "internal" directive.
*) Feature: the CONTENT_LENGTH, CONTENT_TYPE, REMOTE_PORT, SERVER_ADDR,
SERVER_PORT, SERVER_PROTOCOL, DOCUMENT_ROOT, SERVER_NAME,
REQUEST_METHOD, REQUEST_URI, and REMOTE_USER variables.
*) Change: nginx now passes the invalid lines in a client request
headers or a backend response header.
*) Bugfix: if the backend did not transfer response for a long time and
the "send_timeout" was less than "proxy_read_timeout", then nginx
returned the 408 response.
*) Bugfix: the segmentation fault was occurred if the backend sent an
invalid line in response header; the bug had appeared in 0.1.26.
*) Bugfix: the segmentation fault may occurred in FastCGI fault
tolerance configuration.
*) Bugfix: the "expires" directive did not remove the previous
"Expires" and "Cache-Control" headers.
*) Bugfix: nginx did not take into account trailing dot in "Host"
header line.
*) Bugfix: the ngx_http_auth_module did not work under Linux.
*) Bugfix: the rewrite directive worked incorrectly, if the arguments
were in a request.
*) Bugfix: nginx could not be built on MacOS X.
*) Bugfix: nginx did run on Linux parisc.
*) Feature: nginx now does not start under FreeBSD if the sysctl
kern.ipc.somaxconn value is too big.
*) Bugfix: if a request was internally redirected by the
ngx_http_index_module module to the ngx_http_proxy_module or
ngx_http_fastcgi_module modules, then the index file was not closed
after request completion.
*) Feature: the "proxy_pass" can be used in location with regular
expression.
*) Feature: the ngx_http_rewrite_filter_module module supports the
condition like "if ($HTTP_USER_AGENT ~ MSIE)".
*) Bugfix: nginx started too slow if the large number of addresses and
text values were used in the "geo" directive.
*) Change: a variable name must be declared as "$name" in the "geo"
directive. The previous variant without "$" is still supported, but
will be removed soon.
*) Feature: the "%{VARIABLE}v" logging parameter.
*) Feature: the "set $name value" directive.
*) Bugfix: gcc 4.0 compatibility.
*) Feature: the --with-openssl-opt=OPTIONS autoconfiguration directive.
*) Bugfix: if the response were transferred by chunks, then on the HEAD
request the final chunk was issued.
*) Bugfix: the "Connection: keep-alive" header were issued, even if the
keepalive_timeout directive forbade the keep-alive use.
*) Bugfix: the errors in the ngx_http_fastcgi_module caused the
segmentation faults.
*) Bugfix: the compressed response encrypted by SSL may not transferred
complete.
*) Bugfix: the TCP-specific TCP_NODELAY, TCP_NOPSUH, and TCP_CORK
options, are not used for the unix domain sockets.
*) Feature: the rewrite directive supports the arguments rewriting.
*) Bugfix: the response code 400 was returned for the POST request with
the "Content-Length: 0" header; the bug had appeared in 0.1.14.
*) Feature: the autoconfiguration directives:
--http-client-body-temp-path=PATH, --http-proxy-temp-path=PATH, and
--http-fastcgi-temp-path=PATH
*) Change: the directory name for the temporary files with the client
request body is specified by directive client_body_temp_path, by
default it is <prefix>/client_body_temp.
*) Feature: the ngx_http_fastcgi_module and the directives:
fastcgi_pass, fastcgi_root, fastcgi_index, fastcgi_params,
fastcgi_connect_timeout, fastcgi_send_timeout, fastcgi_read_timeout,
fastcgi_send_lowat, fastcgi_header_buffer_size, fastcgi_buffers,
fastcgi_busy_buffers_size, fastcgi_temp_path,
fastcgi_max_temp_file_size, fastcgi_temp_file_write_size,
fastcgi_next_upstream, and fastcgi_x_powered_by.
*) Bugfix: the "[alert] zero size buf" error; the bug had appeared in
0.1.3.
*) Change: the URI must be specified after the host name in the
proxy_pass directive.
*) Change: the %3F symbol in the URI was considered as the argument
string start.
*) Feature: the unix domain sockets support in the
ngx_http_proxy_module.
*) Feature: the ssl_engine and ssl_ciphers directives.
Thanks to Sergey Skvortsov for SSL-accelerator.
*) Feature: the worker_priority directive.
*) Change: both tcp_nopush and tcp_nodelay directives affect the
transferred response.
*) Bugfix: nginx did not call initgroups().
Thanks to Andrew Sitnikov and Andrei Nigmatulin.
*) Change: now the ngx_http_autoindex_module shows the file size in the
bytes.
*) Bugfix: the ngx_http_autoindex_module returned the 500 error if the
broken symlink was in a directory.
*) Bugfix: the files bigger than 4G could not be transferred using
sendfile.
*) Bugfix: if the backend was resolved to several backends and there
was an error while the response waiting then process may got caught
in an endless loop.
*) Bugfix: the worker process may exit with the "unknown cycle" message
when the /dev/poll method was used.
*) Bugfix: "close() channel failed" errors.
*) Bugfix: the autodetection of the "nobody" and "nogroup" groups.
*) Bugfix: the send_lowat directive did not work on Linux.
*) Bugfix: the segmentation fault occurred if there was no events
section in configuration.
*) Bugfix: nginx could not be built on OpenBSD.
*) Bugfix: the double slashes in "://" in the URI were converted to
":/".
*) Bugfix: the proxied request was sent without arguments if the
request contains "//", "/./", "/../" or "%XX".
*) Bugfix: the large compressed responses may be transferred not
completely.
*) Bugfix: the files bigger than 2G was not transferred on Linux that
does not support sendfile64().
*) Bugfix: while the build configuration on Linux the
--with-poll_module parameter was required; the bug had appeared in
0.1.8.
*) Bugfix: in the ngx_http_autoindex_module if the long file names were
in the listing.
*) Feature: the "^~" modifier in the location directive.
*) Feature: the proxy_max_temp_file_size directive.
*) Bugfix: on Solaris and Linux there may be too many "recvmsg()
returned not enough data" alerts.
*) Bugfix: there were the "writev() failed (22: Invalid argument)"
errors on Solaris in proxy mode without sendfile. On other platforms
that do not support sendfile at all the process got caught in an
endless loop.
*) Bugfix: segmentation fault on Solaris in proxy mode and using
sendfile.
*) Bugfix: segmentation fault on Solaris.
*) Bugfix: on-line upgrade did not work on Linux.
*) Bugfix: the ngx_http_autoindex_module module did not escape the
spaces, the quotes, and the percent signs in the directory listing.
*) Change: the decrease of the copy operations.
*) Feature: the userid_p3p directive.
*) Feature: the ngx_http_autoindex_module and the autoindex directive.
*) Feature: the proxy_set_x_url directive.
*) Bugfix: proxy module may get caught in an endless loop when sendfile
is not used.
*) Feature: the gzip_types directive.
*) Feature: the tcp_nodelay directive.
*) Feature: the send_lowat directive is working not only on OSes that
support kqueue NOTE_LOWAT, but also on OSes that support SO_SNDLOWAT.
*) Feature: the setproctitle() emulation for Linux and Solaris.
*) Bugfix: the "Location" header rewrite bug fixed while the proxying.
*) Bugfix: the ngx_http_chunked_module module may get caught in an
endless loop.
*) Bugfix: the /dev/poll module bugs fixed.
*) Bugfix: the responses were corrupted when the temporary files were
used while the proxying.
*) Bugfix: the unescaped requests were passed to the backend.
*) Bugfix: while the build configuration on Linux 2.4 the
--with-poll_module parameter was required.
