NginX times out 502 [bad gateway] using curl [solved]

[maboroshi]

I am running in to an issue where my NginX servers script execution is timing out with 502 bad gateway if requesting a large(r) file over http socks / tor or with longer timed http requests.

Does anyone know how to prevent NginX from timing out on requests such as those? I had similar issues when trying to execute a tiered encryption with swordfish or making an os system call.

Server is uwsgi/nginx/web2py

Any advice is greatly appreciated.

*cheers

Edit * Error Log: upstream prematurely closed connection while reading response header from upstream, client

Posting config info


nginx.conf

Code: Select all

user www-data;
worker_processes 2;
pid /var/run/nginx.pid;

events {
	worker_connections 2048;
	use epoll;
	multi_accept on;
}

http {
	ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

	open_file_cache max=1000 inactive=20s;
	open_file_cache_valid 20s;
	open_file_cache_min_uses 2;
	open_file_cache_errors on;

	client_body_buffer_size 10K;
	client_header_buffer_size 1k;
	client_max_body_size 1000M;
	large_client_header_buffers 2 1k;

	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;

	keepalive_requests 100;
	keepalive_timeout 22;
	reset_timedout_connection on;
	client_body_timeout 22;
	client_header_timeout 22;
	send_timeout 18;
		
        server_tokens off; 
	server_name_in_redirect off;

	include /etc/nginx/mime.types;
	default_type application/octet-stream;
	
	##
	# Logging Settings
	##

	access_log /var/log/nginx/access.log;
	error_log /var/log/nginx/error.log;

	##
	# Gzip Settings
	##

	gzip on;
	gzip_disable "msie6";

	gzip_min_length 1600;
	gzip_proxied expired no-cache no-store private auth;
	
	gzip_vary on;
	gzip_proxied any;
	gzip_comp_level 6;
	gzip_buffers 16 8k;
	gzip_http_version 1.1;
	gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

	##
	# Virtual Host Configs
	##

	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;
}

sites-available/web2py

Code: Select all

server {
        listen          80;
        server_name     $hostname;

        location ~* ^/(\w+)/static/ {
            root /home/www-data/web2py/applications/;
		}

        location / {
        uwsgi_pass      unix:///tmp/web2py.socket;
	    uwsgi_read_timeout 3600;
	    uwsgi_send_timeout 3600;
            include         uwsgi_params;
            uwsgi_param     UWSGI_SCHEME $scheme;
        }
}

server {
        listen 443 default_server ssl;
        server_name     $hostname;
        ssl_certificate         /etc/nginx/ssl/web2py.crt;
        ssl_certificate_key     /etc/nginx/ssl/web2py.key;
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 10m;
        ssl_ciphers ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA;
        ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        keepalive_timeout    70;
        location / {
            uwsgi_pass      unix:///tmp/web2py.socket;
            include         uwsgi_params;
            uwsgi_param     UWSGI_SCHEME $scheme;
        }
}

server {
        listen 443 ssl;
        server_name		cryaboutcrypt.ninja www.cryaboutcrypt.ninja;
        ssl_certificate         /etc/nginx/ssl/cryaboutcrypt.crt;
        ssl_certificate_key     /etc/nginx/ssl/cryaboutcrypt.key;
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 10m;
        ssl_ciphers ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA;
        ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        keepalive_timeout    70;
        location / {
            uwsgi_pass      unix:///tmp/web2py.socket;
            include         uwsgi_params;
            uwsgi_param     UWSGI_SCHEME $scheme;
        }
}

uwsgi.conf

Code: Select all

[uwsgi]

socket = /tmp/web2py.socket
pythonpath = /home/www-data/web2py/
mount = /=wsgihandler:application
cheaper-algo = spare
cheaper = 2
cheaper-initial = 5
workers = 10
cheaper-step = 1
master = true
harakiri = 30
reload-mercy = 8
cpu-affinity = 2
stats = /tmp/stats.socket
max-requests = 4000
limit-as = 2048
reload-on-as = 512
reload-on-rss = 512
uid = www-data
gid = www-data
cron = 0 0 -1 -1 -1 python /home/www-data/web2py/web2py.py -Q -S welcome -M -R scripts/sessions2trash.py -A -o
no-orphans = true

The request from curl is made through SSL to an http file server. Smaller files (less then 2 mb) will download appropriately. After the 2mb or so mark it fails with the error message listed above. I would say every time at the 2/2.5 mb download it drops.

Any advice is more than welcome

[bad_brain]

hm, try if something changes when increasing those values to lets say 120s:

Code: Select all

   open_file_cache max=1000 inactive=20s;
   open_file_cache_valid 20s;

for the sake of performance and resource usage I also recommend to adjust those settings (assuming the keep alives work similar to the apache ones):

Code: Select all

   keepalive_requests 100;
   keepalive_timeout 22;

30 and 5 might be good values to start, simply see what happens then.

[maboroshi]

I tried changing those values as well as a few others you had suggested it may be that I need to upgrade to the nginx 1.7 branch. There seems to have been a bug (or something along those lines ) in the version I am running. I am not sure this will solve it but hopefully.

I will attempt to do this first on a local system just to be sure.

Thanks bb

[bad_brain]

no problem buddy, taking a day off to relax, then I can help you if needed...

[maboroshi]

This has been solved with the suggested values from bad_brain and by updating nginx.

*cheers

Mabo!