Django + Guncorn + nginx: 111: Connection refused; bad gateway 502

Question

For some reason, it does not work. Do you see any issues in this code? Thank you for your review in advance!

Error.log:

2019/11/10 18:02:02 [error] 8761#8761: *1 connect() to unix:/run/gunicorn.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: demid.com, request: "GET / HTTP/1.1", upstream: "http://unix:/run/gunicorn.sock:/", host: "127.0.0.1"

Terminal:

demid@demid-Aspire-7736:~$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
demid@demid-Aspire-7736:~$ systemctl daemon-reload
demid@demid-Aspire-7736:~$ systemctl restart gunicorn.socket gunicorn.service nginx.service; systemctl status gunicorn.socket gunicorn.service nginx.service
Failed to dump process list, ignoring: No such file or directory
● gunicorn.socket - gunicorn socket
   Loaded: loaded (/etc/systemd/system/gunicorn.socket; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-11-10 17:59:39 EET; 288ms ago
   Listen: /run/gunicorn.sock (Stream)
   CGroup: /system.slice/gunicorn.socket

lapkr. 10 17:59:39 demid-Aspire-7736 systemd[1]: Listening on gunicorn socket.

● gunicorn.service - gunicorn daemon
   Loaded: loaded (/etc/systemd/system/gunicorn.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-11-10 17:59:39 EET; 213ms ago
 Main PID: 8756 (gunicorn)
    Tasks: 1 (limit: 4669)
   CGroup: /system.slice/gunicorn.service
           └─8756 /home/demid/myprojectdir/myprojectenv/bin/python3 /home/demid/myprojectdir/myprojectenv/bin/gunicorn

lapkr. 10 17:59:39 demid-Aspire-7736 systemd[1]: Started gunicorn daemon.

● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-11-10 17:59:39 EET; 70ms ago
     Docs: man:nginx(8)
Process: 8757 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, 
  Process: 8759 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 8758 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 8760 (nginx)
    Tasks: 3 (limit: 4669)
   CGroup: /system.slice/nginx.service
           ├─8760 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           ├─8761 nginx: worker process
           └─8762 nginx: worker process

lapkr. 10 17:59:39 demid-Aspire-7736 systemd[1]: Starting A high performance web server and a reverse proxy server...
lapkr. 10 17:59:39 demid-Aspire-7736 systemd[1]: Started A high performance web server and a reverse proxy server.

/etc/nginx/conf.d/demid.com.conf

server {
    listen         80 default_server;
    listen         [::]:80 default_server;
    server_name    demid.com;
    location = /favicon.ico {access_log off; log_not_found off;}
    location /static/ {
root /demid/myprojectdir;
    }

    location / {
include proxy_params;
#proxy_set_header Host $http_host;
#    proxy_set_header X-Real-IP $remote_addr;
#    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#    proxy_set_header X-Forwarded-Proto $scheme;
    #proxy_pass http://unix:/home/sammy/myproject/myproject.sock;
proxy_pass http://unix:/run/gunicorn.sock;
}
}

/etc/systemd/system/gunicorn.service

[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After = network.target

[Service]
User=root
Group=www-data
WorkingDirectory=/home/demid/myprojectdir
ExecStart=/home/demid/myprojectdir/myprojectenv/bin/gunicorn \
--access-logfile - \
--workers 3 \
--bind unix:/run/gunicorn.sock \
myproject.wsgi:application

[Install]
WantedBy=multi-user.target

/etc/nginx/nginx.conf

worker_processes auto;
worker_rlimit_nofile 50000;

events {
    worker_connections  1024;
    use epoll;
    multi_accept on;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    keepalive_timeout  65;
    keepalive_requests 256;
    reset_timedout_connection on;

    gzip  on;
    gzip_vary on;
    gzip_proxied any;
    gzip_min_length 1000;
    gzip_types text/plain text/xml text/css text/javascript application/x-javascript application/json application/xml application/xml+rss image/png image/gif image/jpeg image/jpg;

    open_file_cache max=50000 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;

    client_max_body_size 512m;

    server_tokens off;

    include /etc/nginx/conf.d/*.conf;
}

Answer 1

To summarize - nginx was returning 111: Connection refused after trying to connect to gunicorn through a UNIX socket located at /run/gunicorn.sock.

At first we made sure this wasn't a permission issue:

# ls -l /run/gunicorn.sock
srw-rw-rw- 1 root root 0 lapkr 10 20:34 /run/gunicorn.sock

The rw-rw-rw- means that every user can read and write into the file.

(Side note - these permissions are too broad, better practice might be to make sure that nginx workers are run by the same user as gunicorn.service and grant RW permissions only to this user, but this answer is focused mainly on making the connection work, and this really isn't my domain anyway.)

The next action was to list processes listening on the socket:

# netstat --protocol=unix -nlp | grep 'gunicorn'

(Instead of gunicorn you can also grep full socket path, but here it was enough).

Output was empty, gunicorn wasn't listening.

After looking at gunicorn service logs:

# systemctl status gunicorn.service
# journalctl -u gunicorn.service

we found that the socket binding failed with message:

ModuleNotFoundError: No module named 'myproject.wsgi' 

This was caused by using wrong path to Python application for socket binding in /etc/systemd/system/gunicorn.service.

The file contains two params that affect this - WorkingDirectory and ExecStart.

The WorkingDirectory must point to a directory that contains the main gunicorn config file - settings.py.

The ExecStart must contain the main command to run the gunicorn virtualenv binary. You need to run it with the option --bind /run/gunicorn.sock myproject.wsgi:application, where /run/gunicorn.sock is socket filepath and myproject.wsgi:application is relative path to Python module entrypoint (dirs are separated by dots, extension .py is ommitted, module name is followed by colon and a callable application variable).