uWSGI with NetBox

[markkuleinio]

uWSGI is new for me, but thinking about replacing gunicorn with it. Any comments for my attempt to replicate the stock gunicorn setup as uWSGI?

/opt/netbox/venv/bin/pip install uwsgi
(and add uwsgi in local_requirements.txt)

/opt/netbox/netbox/local/uwsgi.ini:

[uwsgi]
strict = true
master = true
enable-threads = true
vacuum = true
single-interpreter = true
die-on-term = true
auto-procname = true
procname-append = , %o

socket = 127.0.0.1:8001
chdir = /opt/netbox/netbox
wsgi-file = netbox/wsgi.py
processes = 5
threads = 2
harakiri = 120
thunder-lock = true
# Increase the buffer size (default = 4096 bytes) if large requests are needed:
# buffer-size = 16384

# Enable if stats are desired:
#stats = 127.0.0.1:9001
#stats-http = true
#memory-report = true

# Only log internal messages and errors
disable-logging = true
log-4xx = true
log-5xx = true

/etc/systemd/system/netbox.service:

[Unit]
Description=NetBox uWSGI Service
Documentation=https://docs.netbox.dev/
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
NotifyAccess=all
Restart=always
RestartSec=30
ExecStart=/opt/netbox/venv/bin/uwsgi --ini /opt/netbox/netbox/local/uwsgi.ini
ExecReload=kill -HUP $MAINPID
User=netbox
Group=netbox

[Install]
WantedBy=multi-user.target

/etc/nginx/sites-enabled/netbox (I only have port 80 as I'm behind AWS ALB that does TLS offloading):

server {
    listen 80;
    server_name netbox.mydomain.com;

    client_max_body_size 25m;

    uwsgi_read_timeout 120;

    location /static/ {
        alias /opt/netbox/netbox/static/;
    }

    location / {
        include uwsgi_params;
        uwsgi_pass 127.0.0.1:8001;
    }

    # Used by netbox-plugin-auth-saml2
    location /sso/ {
        rewrite (.*) /api/plugins$1 break;
        include uwsgi_params;
        uwsgi_pass 127.0.0.1:8001;
    }
}

With this setup it seems to work:

• Client IP is relayed correctly for API token IP checks (even though Nginx is behind a load balancer's NAT)
• URLs in API responses are correct (https is included even though Nginx only gets http)
• Device links in racks work fine (I think sometimes they had problems if the headers weren't correctly set)
• netbox-plugin-auth-saml2 works (I still like it, I have SSO button in login banner, haven't used python-social-auth)
• sudo systemctl reload netbox causes a graceful reload of uWSGI
  • Note that starting a NetBox worker still takes a few seconds (not related to uWSGI), so the next few requests to NetBox will experience some delay after restarting/reloading it
• Large requests: If you use NetBox API with for example very long filters, you may need to increase the buffer-size value

Questions in mind:

• Is something missing from the uWSGI configuration?
• Is there something in the systemd config that needs to be set differently?

Anything else?

Tagging @tyler-8 here as he happened to mention using uWSGI 🙂

[markkuleinio]

I have now run a couple of NetBox 3.7.x instances with these uWSGI+systemd configurations a few days successfully, looks good. Obviously processes and threads should be customized according to the local needs, but those values above seem to work just fine for a small VM (2 vCPU + 4 GB RAM) in the cloud.

Note that I didn't add any max-requests configuration there, I'll keep monitoring the memory footprint of the workers. (The original reason for using max_requests in NetBox' gunicorn default configuration was fighting some memory leaks in some part of the application stack.)

I'll still keep updating the configs above if I find something to adjust.

Feel free to tell your own experiences about using uWSGI with NetBox.

[markkuleinio]

Hint, when the statistics are enabled in the config above:

/opt/netbox/venv/bin/pip install uwsgitop
/opt/netbox/venv/bin/uwsgitop http://localhost:9001/

--> lets you monitor the worker processes in real-time (use q to quit)

[markkuleinio]

Number of requests served by all the workers (processes = 5, threads = 3 in this setup):

$ curl http://localhost:9001 -s | jq .workers[].requests | (sum=0; while read -r line; do ((sum += line)); done; echo $sum)
346393

Memory usage of all uWSGI processes combined for seven days (this NetBox 3.7.1 instance was started just about exactly 7 days ago), after those ~350k requests:

[jeremystretch]

Thanks for sharing this @markkuleinio! Given that there's still been no response to the gunicorn bug mentioned in #14691, we should consider switching to uWSGI for NetBox's reference implementation.

[markkuleinio]

Maybe someone is willing to also provide memory (or other relevant) data about using gunicorn without max_requests, as that's the primary reason for getting the 502s with the latest gunicorn, if I have understood correctly

[markkuleinio]

Those who are willing/needing to capture the uwsgi protocol traffic (like sudo tcpdump -i lo port 8001 -w capture.pcap), I have also written a quick skeleton for a Wireshark dissector:

https://github.com/markkuleinio/wireshark-uwsgi-dissector

[tyler-8]

Here's my uwsgi config, I'm using this Ansible role to manage it: https://github.com/lae/ansible-role-netbox

[uwsgi]
master=true
protocol=uwsgi
uid=netbox
gid=netbox
processes=8
module=netbox.wsgi
virtualenv=/srv/netbox/current/venv-py3
chdir=/srv/netbox/current/netbox
static-map=/static=/srv/netbox/current/netbox/static
logger=file:logfile=/srv/netbox/shared/application.log,maxsize=50000000
req-logger=file:logfile=/srv/netbox/shared/requests.log,maxsize=50000000
env=prometheus_multiproc_dir=/run/netbox_metrics
; Ensures the app successfully loads, otherwise uwsgi will stay running with no active app to serve
need-app=True
; Only run if all uwsgi parameters are valid uwsgi parameters
strict=True
; If you have large query parameters or domain cookies increase this above the default
buffer-size=4096
harakiri=60
max-requests=5000
max-worker-lifetime=43200
max-worker-lifetime-delta=60
worker-reload-mercy=60
single-interpreter=True
log-date=[%%Y-%%m-%%d %%H:%%M:%%S]

[tyler-8]

nginx speaks "uwsgi" protocol, so you can use this in your nginx config instead of pointing to 127.0.0.1:8000

   location / {
       include uwsgi_params;
       uwsgi_pass unix:///var/run/netbox_uwsgi.sock;
       uwsgi_param X-Forwarded-Host $server_name;
       uwsgi_param X-Real-IP $remote_addr;
       uwsgi_param X-Forwarded-Proto $scheme;

   }

You can setup systemd socket with the below systemd config. uwsgi will automatically detect the socket and use it

# filename: netbox.socket
# should match service file: 'netbox.service' in this case
[Unit]
Description=Socket for NetBox IPAM/DCIM Tool

[Socket]
ListenStream=/var/run/netbox_uwsgi.sock
SocketUser=netbox
SocketGroup=netbox

[Install]
WantedBy=sockets.target

[markkuleinio]

Thanks! I specifically wanted to use uwsgi over TCP socket so that I can capture that easily whenever more information is needed. AFAIK unix socket capturing is somewhat more complicated. But unix sockets are said to be a bit more efficient I think.

[DanSheps]

Shouldn't the "WantedBy" be "netbox.service"?

[markkuleinio]

uWSGI process memory usage after running NetBox for two weeks and ~750k requests:

[markkuleinio]

FYI there is some current uWSGI+Sentry issue (I see NetBox has some Sentry integration available, don't know about that):

getsentry/sentry-python#2699

[markkuleinio]

One more memory graph for this NetBox 3.7.1 for seven days:

Before system reboot uWSGI memory usage was above 2.7 GB, after reboot and ~170k requests in three days it is currently about 2.63 GB.

[jeremystretch]

Thanks for sharing this data @markkuleinio. I think there's a case to be made for switching NetBox's standard WSGI solution from gunicorn to uWSGI.

[markkuleinio]

About installing uWSGI with pip: uWSGI is only available in source distribution (hopefully I'm using the correct term, there are no wheels, just the sources, see https://pypi.org/project/uWSGI/#files), but in my setups on Debian Linux 12 I have been able to install (compile) uWSGI successfully (using local_requirements.txt as described above) without special packages. However, in my specific EC2 instances there may be some customizations under the hood.

[candlerb]

What URL is your monitoring system hitting? Can you show the full output of curl -v <url> ?

[MalfuncEddie]

xymon
I've written a scritp that does a curl if it is hitting it I will update this thread.

[MalfuncEddie]

I've updated to v4.0.3 and all my memory/connection issue are gone in both uwsgi and gunicorn. No idea what fix did this. So I can no longer reproduce the error.

[jonaagenilsen]

I've updated to v4.0.3 and all my memory/connection issue are gone in both uwsgi and gunicorn. No idea what fix did this. So I can no longer reproduce the error.

Same here. Running gunicorn @ v3.7.8 with tweaks in reference of https://www.tinfoilcipher.co.uk/2019/11/11/netbox-502-bad-gateway/ .. still get 502 errors (less though).

Cloned the server, applied v4.0.3 and I no longer get that error :)

[jonaagenilsen]

I've updated to v4.0.3 and all my memory/connection issue are gone in both uwsgi and gunicorn. No idea what fix did this. So I can no longer reproduce the error.

Same here. Running gunicorn @ v3.7.8 with tweaks in reference of https://www.tinfoilcipher.co.uk/2019/11/11/netbox-502-bad-gateway/ .. still get 502 errors (less though).

Cloned the server, applied v4.0.3 and I no longer get that error :)

Spoke too soon..

[markkuleinio]

Adding this piece of information. These don't work with pyuwsgi (tested with 2.0.28):

auto-procname = true
procname-append = , %o

With pyuwsgi:

32520 ?     Ss   0:01 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/uwsgi --ini /opt/netbox/uwsgi.ini
32579 ?     Sl   0:00 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/uwsgi --ini /opt/netbox/uwsgi.ini
32581 ?     Sl   0:00 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/uwsgi --ini /opt/netbox/uwsgi.ini
32583 ?     Sl   0:00 /opt/netbox/venv/bin/python3 /opt/netbox/venv/bin/uwsgi --ini /opt/netbox/uwsgi.ini

With uwsgi:

33468 ?     Ss   0:01 uWSGI master, /opt/netbox/uwsgi.ini
33524 ?     Sl   0:00 uWSGI worker 1, /opt/netbox/uwsgi.ini
33526 ?     Sl   0:00 uWSGI worker 2, /opt/netbox/uwsgi.ini
33528 ?     Sl   0:00 uWSGI worker 3, /opt/netbox/uwsgi.ini