Process supervision

This page covers keeping the server running: the supervisord setup the .deb ships, what its pieces do, and what any other supervisor needs to get right.

What the .deb sets up

The package runs the server under supervisord. It ships no systemd unit.

File	Role
/etc/supervisor/conf.d/ricochet.conf	The supervisord program
/opt/ricochet/run.sh	The command supervisord runs
/etc/ricochet/env	Password, public IP and database overrides that run.sh reads
/opt/ricochet/health_check.sh	Health check against the ops listener
/var/log/ricochet/stdout.log, stderr.log	Logs, rotated by supervisord

The program definition, abridged:

/etc/supervisor/conf.d/ricochet.conf

[program:ricochet]
command=/opt/ricochet/run.sh
directory=/var/lib/ricochet
user=ricochet
autostart=true
autorestart=true
startsecs=10
startretries=3
stopsignal=TERM
stopwaitsecs=30
stopasgroup=true
killasgroup=true
stdout_logfile=/var/log/ricochet/stdout.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
stderr_logfile=/var/log/ricochet/stderr.log
stderr_logfile_maxbytes=50MB
stderr_logfile_backups=10
environment=HOME="/var/lib/ricochet"

• The server runs as the ricochet system user, in /var/lib/ricochet.
• supervisord restarts it whenever it exits. A process that dies within 10 seconds of starting counts as a failed start; after 3 in a row it is left in FATAL, which stops a restart loop while, for example, the database is down.
• On stop, supervisord sends SIGTERM and waits 30 seconds before SIGKILL. That leaves room for a graceful shutdown. See Upgrades & draining.
• Each log rotates at 50 MB and keeps 10 backups.

Both this file and /etc/ricochet/config.yaml are conffiles: an upgrade does not overwrite your edits.

run.sh

run.sh sources /etc/ricochet/env, exits with ERROR: DB_PASSWORD is not set if the password is missing, exports it as RICOCHET_PG_PASSWORD, and starts the server:

exec /opt/ricochet/ricochet_server \
    --production \
    $PORT_FLAG \
    --data-dir /var/lib/ricochet/sf_storage \
    --pg-host "${DB_HOST:-localhost}" \
    --pg-port "${DB_PORT:-5432}" \
    --pg-database "${DB_NAME:-ricochet}" \
    --pg-username "${DB_USER:-ricochet}" \
    --pg-sslmode "${DB_SSLMODE:-require}" \
    $EXTERNAL_ADDRS_FLAG

$PORT_FLAG is --port $LISTEN_PORT when LISTEN_PORT is set. $EXTERNAL_ADDRS_FLAG is --external-addrs /ip4/$EXTERNAL_IP/udp/<port>/udx when EXTERNAL_IP is set. The server also loads /etc/ricochet/config.yaml, under these flags. The full variable table is on Install.

Day-to-day commands

sudo supervisorctl start ricochet
sudo supervisorctl stop ricochet
sudo supervisorctl restart ricochet      # after editing /etc/ricochet/config.yaml or env
sudo supervisorctl status ricochet

sudo supervisorctl tail -f ricochet          # stdout
sudo supervisorctl tail -f ricochet stderr

The server reads its configuration only at startup. After changing config.yaml or env, restart it. After changing ricochet.conf, run sudo supervisorctl reread && sudo supervisorctl update first.

A process in FATAL stays down until you start it again: fix the cause (read stderr.log and stdout.log), then supervisorctl start ricochet.

Health checks

supervisord restarts a process that exits; it cannot tell a running process that cannot serve. Ask the server. /opt/ricochet/health_check.sh checks /healthz and /readyz and exits 0 (healthy), 1 (up but not ready) or 2 (not responding):

*/5 * * * * /opt/ricochet/health_check.sh >/dev/null || logger -t ricochet -p daemon.err "health check failed"

See Health & readiness.

Resource limits

The server has no memory setting; it bounds connections, requests in flight, the database pool and the mailbox cache, and memory follows. supervisord cannot cap memory itself. For a hard ceiling, put the cgroup around supervisord, for example MemoryMax= on the supervisor systemd unit, which covers every process it starts.

Using another supervisor

The repository ships only the supervisord setup. If you run the server under systemd, a container runtime or anything else, give it what run.sh and ricochet.conf give it:

• a dedicated, unprivileged user that can read the config and identity key and write the data directory;
• the database password in RICOCHET_PG_PASSWORD, not on the command line;
• a stable --data-dir (or identity_file, or RICOCHET_SEED_HEX), so the peer ID survives restarts;
• SIGTERM to stop, and a kill timeout longer than ops.drain_delay plus ops.shutdown_timeout;
• restart on exit, with a limit on rapid restarts;
• a liveness probe on /healthz and a readiness probe on /readyz.

The server logs to standard output (startup errors such as a bad config file go to standard error) and writes no log files of its own.

Troubleshooting

• FATAL right after installing the package: the server tried to start before you set DB_PASSWORD and created the database. Configure it, then supervisorctl start ricochet.
• ERROR: DB_PASSWORD is not set in stdout.log: set it in /etc/ricochet/env.
• Run it by hand to see the error: sudo -u ricochet /opt/ricochet/run.sh runs the same command in the foreground. Stop the supervised copy first, or the port is taken.
• Permission errors on the data directory: sudo chown -R ricochet:ricochet /var/lib/ricochet /var/log/ricochet.