Deploying Uwsgi to host Dash apps

When I set up this site, I wanted to host web apps, not just static pages, and my initial goal was a Shiny Server. While somewhat challenging, the good news was that once it is working, just dropping a new directory into /srv/shiny-server/ and the new app springs to life.

I started making dash apps earlier this year. The various .ini files, systemctl services, etc. was considerably more complicated than following the instructions for a shiny server. I found some excellent blog posts that helped me configure my various files but I can never seem to rediscover the posts that helped me the most, so I’m going to walk through my setup here in hopes it helps others. I use nginx so I assume you have a basic working nginx install for a static web server. If you do not, there are plenty of tutorials on that, such as this one from Digital Ocean.

I have done this most recently on Ubuntu 20.04 LTS. I deployed uwsgi in emperor mode with each of my dash apps as a vassal, this makes it easier to deploy additional dash apps.

Install UWSGI

It looks like I installed uwsgi with:

sudo apt-get install build-essential python3-dev
sudo pip3 install uwsgi

Configure the Emperor

The first step is configuring the emperor via an ini file. Mine very straigthforward, in /etc/uwsgi/emperor.ini

[uwsgi]
emperor = /etc/uwsgi/vassals
uid = www-data
gid = www-data
#plugins = logfile
logger = file:/var/log/uwsgi/emperor.log

You can probably test this via

sudo uwsgi --master --enable-threads --ini /etc/uwsgi/emperor.ini

Which hopefully runs and looks nominal. You’ll need to control-c to get out of this.

Set up your vassal(s)

Here is a vassal ini file for my dash dataframe live example.

[uwsgi]

socket = /srv/dash_dataframe_table/uwsgi.sock
chmod-socket=664
chdir = /srv/dash_dataframe_table/
binary-path = /usr/local/bin/uwsgi
module = example:server
uid = www-data
gid = www-data
processes = 1
threads = 1
master = true
logger = file:/var/log/uwsgi/dashtable.log
idle = 60
die-on-idle = true
cheap = true
reload-mercy=5
worker-reload-mercy=5

I also have a corresponding dataframe.socket text file in /etc/uwsgi/vassals that says simply has the text /srv/dash_dataframe_table/uwsgi.sock. This is for the emperor on demand functionality. For a simple demo app like this, it’ll shutdown / be idle until it gets a connection. That’s the cheap/lazy/die-on-idle settings you see above.

The app code itself needs to be in /srv/dash_dataframe_table. (The www-data group needs to have read/write to everything in that folder.) The line module = example:server tells it to look for the server object in the example.py file. It is important for dash apps that something in your source code say server = app.server so that the Flask server itself is exposed.

At this point you should be able to test run emperor on the command line with:

$ /usr/local/bin/uwsgi --master --enable-threads --ini /etc/uwsgi/emperor.ini --emperor-on-demand-extension .socket

which should also launch and show it’s waiting for a socket connection. Cancel out of this with ctrl-c.

Set up the Emperor as a service

You want a service in systemd that runs the emperor automatically. My /etc/systemd/system/emperor.uwsgi.service file looks like this:

[Unit]
Description=uWSGI Emperor
After=syslog.target

[Service]
ExecStart=/usr/local/bin/uwsgi --master --enable-threads --ini /etc/uwsgi/emperor.ini --emperor-on-demand-extension .socket
RuntimeDirectory=uwsgi
Restart=always
KillSignal=SIGQUIT
Type=notify
StandardError=syslog
NotifyAccess=all

[Install]
WantedBy=multi-user.target

You can now get the service running by sudo systemctl start emperor.uwsgi.service

Configure NGINX

You can add separate blocks like this to the nginx file; this hard codes a specific URL to your application:

    location /dash/table_example/ {
        include uwsgi_params;
        uwsgi_pass unix:///srv/dash_dataframe_table/uwsgi.sock;
    }

Test the file with ngnix -t and then restart the server with (most likely):

sudo systemctl restart nginx.service

In this example, the dash app at /srv/dash_dataframe_table is now at the url dash/table_example/

UPDATE: A better method (Originally updated: August 2022)

I have found a better system that doesn’t involved editing the nginx configuration every time.

location /dash/

{
include uwsgi_params;
    if ($request_uri ~* "/dash/([a-zA-Z_]+)/*") {
        uwsgi_pass unix:///srv/$1/uwsgi.sock;
    }

}

Now so long as the URL and the app directory match, I don’t have to ever edit the configuration of nginx. In this case, https://example.com/dash/cool_app/

will look for an socket in /srv/cool_app/uwsgi.sock.

The code above restricts the directory names and URLs to letters and _ but I think that’s a reasonable restriction. I could probably tweak the regex to allow for more options in the directory names, like dashes.

Now when I deploy a new app as vassal, I still create the .ini/.socket pair but then it just works without much fuss. I am hoping to create a script (shell or python) to help me deploy new apps even more easily.

Adding additional apps

Adding a new app involves:

It is important to note that the url_base_pathname of the actual dash application must be consistent with the nginx path.

For example, this is how the dataframe table example dash app is instantiated:

app = dash.Dash("example of dataframe",
                external_stylesheets=[dbc.themes.YETI],
                title="Example Data Frame",
                meta_tags=[
                    {
                        "name": "viewport",
                        "content": "width=device-width, initial-scale=1"
                    },
                ],
                url_base_pathname='/dash/table_example/')

January 2023 Update

Since I want the url and the folder name to always match now, I’m doing this, now with Pathlib

from pathlib import Path
parent_dir = Path().absolute().stem

such that I can set url_base_pathname=f'/dash/{parent_dir}/

Advanced INI files with magic variables

If you can get a consistent convention of naming, you can save yourself time by using magic variables in the .ini files. For example, %n is the file name of the .ini file minus the extension. So if you keep your directory structure and .ini file names consistent, than a template like the one below can be reused with minimal changes. I have only started to do this a little. It makes me think that I should always have my app in the same file name app.py then I would not even have to change the module line.

[uwsgi]

#customize this part
module = app:server
processes = 1
threads = 1

#leave these on if you want the app to shut itself down when idle.
idle = 60
die-on-idle = true
cheap = true


# below here should be the same every time using the file name to set the path and the logger name

reload-mercy=5
worker-reload-mercy=5
socket = /srv/%n/uwsgi.sock
chmod-socket=664
chdir = /srv/%n/
binary-path = /usr/local/bin/uwsgi
uid = www-data
gid = www-data

master = true
logger = file:/var/log/uwsgi/%n.log

Conclusion / Common pitfalls

Permissions can be an issue, I have set everything up such that the apps run as the user www-data, and the directories must all have www-data as their owner and group. I added my regular account to the www-data group so I can pull the repos normally to update the account.

I’ve written some simple aliases / bash scripts to pull and relaunch the app when I make changes. Something like:

#!/usr/bin/bash

cd /srv/$1
git pull
sudo touch /etc/uwsgi/vassals/$1.ini
sleep 2
tail  /var/log/uwsgi/$1.log

Will pull the repo, touch the ini file so the emperor restarts the app, and then waits and tails the log so I can see if everything worked ok. The $1 lets me pass in an app directory name on the command line, instead of needing a different script for every app.

I can then run this command over ssh to update the app deployment.