nginx_process_request_and_response_body

Posted on Edited on In ,

Overview

nginx provides flexible way when processing header and body to meet different network env to achieve better performance, say nginx can send the request immediately after get the whole request headers, or send the request to upstream after get the entire body, when cache request body, we can cache it in memory if memory is large enough or cache it to a file.
Also we can cache the response body to a file or send it immediately to client, all are configurable!!!

Body processing

request

request header buffer

Before talking about request body, let’s say process header buffer first, here are directives related to request header

1
2
3
Syntax:	client_header_buffer_size size;
Default: client_header_buffer_size 1k;
Context: http, server

Sets buffer size for reading client request header. For most requests, a buffer of 1K bytes is enough. However, if a request includes long cookies, or comes from a WAP client, it may not fit into 1K. If a request line or a request header field does not fit into this buffer then larger buffers, configured by the large_client_header_buffers directive, are allocated.

1
2
3
Syntax:	large_client_header_buffers number size;
Default: large_client_header_buffers 4 8k;
Context: http, server

Sets the maximum number and size of buffers used for reading large client request header. A request line cannot exceed the size of one buffer, or the 414 (Request-URI Too Large) error is returned to the client. A request header field(one header) cannot exceed the size of one buffer as well, or the 400 (Bad Request) error is returned to the client. Buffers are allocated only on demand that means it's allocated only when 1K is not enough to hold request headers or request line.

request body

1
2
3
4
# It's clear to call client_request_buffering_on to align with others.
Syntax:	proxy_request_buffering on | off;
Default: proxy_request_buffering on;
Context: http, server, location

When buffering is enabled, the entire request body is read from the client before sending the request to a proxied server, send request(headers, then body) happens only when we receive the entire request body.

When buffering is disabled, the request body is sent to the proxied server immediately as it is received(request headers). In this case, the request cannot be passed to the next server if nginx already started sending the request body.

As mentioned above when buffering is enabled, in some case request body is large, how can we cache it? nginx first uses memory, then disk for body caching, but if CPS is high and with large request body, it’s not a good way to buffer them all as it could use up disk or memory.

1
2
3
Syntax:	client_body_buffer_size size;
Default: client_body_buffer_size 8k|16k;
Context: http, server, location

Sets buffer size for reading client request body. In case the request body is larger than the buffer(part of unused header buffer + this buffer), the whole body or only its part is written to a temporary file(each request has only one temporary file), By default, buffer size is equal to two memory pages. This is 8K on x86, other 32-bit platforms, and x86-64. It is usually 16K on other 64-bit platforms.

Note: temporary file is unlink() after nginx open it, that means you CAN NOT see it even worker is writing data to it, temporary file is deleted after get response header from upstream that means we send out entire request body

1
2
3
4
5
6
7
# show fd of temporary file
$lsof -p worker_pid | grep 'deleted'

# from error.log file
2020/12/17 15:06:40 [warn] 3856#0: *1 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000001

# temporary is removed automatically from disk get response header from upstream.
1
2
3
Syntax:	client_body_temp_path path [level1 [level2 [level3]]];
Default: client_body_temp_path client_body_temp;
Context: http, server, location

Defines a directory for storing temporary files holding client request bodies. Up to three-level subdirectory hierarchy can be used under the specified directory. For example, in the following configuration

1
client_body_temp_path /spool/nginx/client_temp 1 2;

a path to a temporary file might look like this:

1
/spool/nginx/client_temp/7/45/00000123457
1
2
3
Syntax:	client_body_in_file_only on | clean | off;
Default: client_body_in_file_only off;
Context: http, server, location

Determines whether nginx should save the entire client request body(no matter what size it is) into a file, it's same file with proxy_request_buffering directive. This directive can be used during debugging, or when using the $request_body_file variable.

When set to the value on, temporary files are not removed after request processing.

The value clean will cause the temporary files left after request processing to be removed.

no buffering request body conf

1
2
3
proxy_request_buffering off;
proxy_pass_request_body on;
proxy_http_version 1.1; # reason for this http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering
1
2
3
Syntax:	client_max_body_size size;
Default: client_max_body_size 1m;
Context: http, server, location

Sets the maximum allowed size of the client request body, specified in the “Content-Length” request header field. If the size in a request exceeds the configured value, the 413 (Request Entity Too Large) error is returned to the client. Please be aware that browsers cannot correctly display this error. Setting size to 0 disables checking of client request body size.

response

There are several directives used for control response, should we buffer it when necessary.

1
2
3
4
# write to temporary file only when client is slow than upstream
Syntax:	proxy_buffering on | off;
Default: proxy_buffering on;
Context: http, server, location

When buffering is enabled, nginx receives a response from the proxied server as soon as possible, saving it into the buffers set by the proxy_buffer_size and proxy_buffers directives. If the whole response does not fit into memory, a part of it can be saved to a temporary file on the disk. when buffering is enabled, nginx tries to receive enough body as much as possible, but if client is ready to write, we still send body to it during receiving, that means temporary file may be not used if client is faster than upstream. buffering NOT mean sending to client only when receive entire response, it’s different with buffering for request.

When buffering is disabled, the response is passed to a client synchronously, immediately as it is received. nginx will not try to read the whole response from the proxied server.The maximum size of the data that nginx can receive from the server at a time is set by the proxy_buffer_size directive.

1
2
3
Syntax:	proxy_buffers number size;
Default: proxy_buffers 8 4k|8k;
Context: http, server, location

Sets the number and size of the buffers used for reading a response from the proxied server, for a single connection. By default, the buffer size is equal to one memory page. This is either 4K or 8K, depending on a platform.

1
2
3
Syntax:	proxy_max_temp_file_size size;
Default: proxy_max_temp_file_size 1024m;
Context: http, server, location

When buffering of responses from the proxied server is enabled, and the whole response does not fit into the buffers set by the proxy_buffer_size and proxy_buffers directives, a part of the response can be saved to a temporary file. This directive sets the maximum size of the temporary file. the temporary file is deleted automatically when request is finalize in ngx_http_upstream_finalize_request().

The zero value disables buffering of responses to temporary files.

1
2
3
Syntax:	proxy_temp_path path [level1 [level2 [level3]]];
Default: proxy_temp_path proxy_temp;
Context: http, server, location

Defines a directory for storing temporary files with data received from proxied servers. Up to three-level subdirectory hierarchy can be used underneath the specified directory. For example, in the following configuration

1
proxy_temp_path /spool/nginx/proxy_temp 1 2;

a temporary file might look like this:

1
/spool/nginx/proxy_temp/7/45/00000123457

non buffering response

1
2
3
proxy_buffering off
# receive buffer used for header and body at a time recv(fd, size)
proxy_buffer_size size;

buffering response

1
2
3
4
5
6
7
8
9
proxy_buffering on;

# below two used for saving response in memory
proxy_buffer_size size;
proxy_buffers number size;

# if memory is used up, save it to disk
proxy_temp_path path;
proxy_max_temp_file_size size;

request header

1
2
3
4
// build r->headers_in from request header
static void ngx_http_process_request_headers(ngx_event_t *rev);
//build header sent to upstream(backend), create a temp buffer to hold all header copied from r->headers_in
static ngx_int_t ngx_http_proxy_create_request(ngx_http_request_t *r);

response header

1
2
3
4
5
6
7
8
9
// build header sent to client(set r->header_out) when parsed all response headers
static ngx_int_t ngx_http_upstream_process_headers(ngx_http_request_t *r, ngx_http_upstream_t *u);
// send response header to client and cleanup request body temp file(it will call header filter later on)
static void ngx_http_upstream_send_response(ngx_http_request_t *r, ngx_http_upstream_t *u);

// header filter to add/update/remove(manipulate r->header_out) response header
static ngx_int_t ngx_http_headers_filter(ngx_http_request_t *r);
// last header filter based on r->header_out create a temp buffer to hold all header copied from r->header_out, then send it out
static ngx_int_t ngx_http_header_filter(ngx_http_request_t *r)

Body

request body

1
2
3
4
// client is ready to read, read request and send it to upstream
static void ngx_http_read_client_request_body_handler(ngx_http_request_t *r);
// upstream is ready to write, read request body and send it to upstream
static void ngx_http_upstream_send_request_handler(ngx_http_request_t *r, ngx_http_upstream_t *u);

response Body

1
2
3
4
5
6
7
8
9
10
// downstream is ready to write
static void ngx_http_upstream_process_downstream(ngx_http_request_t *r);
// upstream is ready to read
static void ngx_http_upstream_process_upstream(ngx_http_request_t *r, ngx_http_upstream_t *u);

// sending response(non buffered mode)
// downstream is ready to write(read response and send it to client)
static void ngx_http_upstream_process_non_buffered_downstream(ngx_http_request_t *r);
// upstream is ready to read(read response and send it to client)
static void ngx_http_upstream_process_non_buffered_upstream(ngx_http_request_t *r, ngx_http_upstream_t *u);