tmux for Long-Running Background Tasks

I run a housing calculator at calgaryhomeestimate.com on the same Ubuntu server as this site. The backend is a FastAPI app with an ML valuation model that needs to be recalibrated periodically against fresh sold-listings data. That calibration pipeline — backfilling assessment records, running comps validation, generating subtype and postal multipliers — can take anywhere from 15 minutes to over an hour depending on dataset size.

For a long time I ran these jobs in the background with & and hoped for the best.

python3 backfill_calgary_assessment.py --match-only > /tmp/backfill.log 2>&1 &

This works until it doesn’t. Your SSH session drops, the terminal closes, your laptop sleeps, you accidentally hit Ctrl+C — and the job is gone. No output, no progress, no way to resume. You start over.

tmux solves this cleanly. Here’s how I actually use it.

The Core Concept

tmux is a terminal multiplexer. It runs a server process on the remote machine that hosts persistent sessions. You attach to a session from your SSH client, run commands, then detach — the session keeps running on the server whether you’re connected or not.

The lifecycle is:

SSH into the server
Create or attach to a named tmux session
Run your long job inside that session
Detach (Ctrl+B then D) — your job keeps running
Reconnect later and reattach to see progress

That’s it. No nohup. No disown. No hoping /tmp/myprocess.log has something useful in it.

Basic Commands You Actually Need

# Create a named session
tmux new -s calibration

# Detach from current session
# Ctrl+B, then D

# List all sessions
tmux ls

# Reattach to a session by name
tmux attach -t calibration

# Kill a session when done
tmux kill-session -t calibration

Name your sessions. tmux new -s s0 is useless two days later. tmux new -s backfill-2025-07 tells you exactly what’s in there.

Running the Housing Calculator Pipeline

The calibration pipeline for the housing calculator runs several sequential Python scripts. Before tmux, the run script looked like this:

# run_backfills_and_validate.sh
python3 backfill_calgary_assessment.py $MATCH_ONLY
python3 scripts/calibration/validate_comp_coverage.py --years 2
python3 scripts/calibration/report_city_assessment_coverage.py --years 2
python3 scripts/calibration/check_backfill_success.py --years 2

Each step takes 5–20 minutes. The whole thing takes 45+ minutes on a full run. If your SSH connection drops after step 2, you have no idea where it failed and no way to resume.

With tmux:

# From your laptop
ssh chowling@chowlingserver

# Create a persistent session for this calibration run
tmux new -s backfill-july

# Inside the session — run the whole pipeline
cd /var/www/Housing-Calculator
./run_backfills_and_validate.sh --years 2

# Detach (you can close your laptop now)
# Ctrl+B, D

# Hours later, reattach and see what happened
ssh chowling@chowlingserver
tmux attach -t backfill-july

The full terminal history is sitting there. You can scroll up, see every step, catch any errors.

Splitting the Screen for Multi-Service Starts

The housing calculator runs three services: a FastAPI backend, a Vite frontend, and ngrok. The manage_services.sh script starts them as background processes and logs to /tmp:

python3 -m mls_app.main > /tmp/housing_calc_backend.log 2>&1 &
npm run dev > /tmp/housing_calc_frontend.log 2>&1 &
ngrok http 3000 --log=stdout > /tmp/housing_calc_ngrok.log 2>&1 &

This works, but you’re blind. You have to tail -f three separate log files across three terminal windows. In tmux, you can watch all three at once in a single pane layout:

tmux new -s housing

# Split vertically
# Ctrl+B, %

# Split the right pane horizontally
# Ctrl+B, "

# Navigate between panes: Ctrl+B, arrow keys

# In pane 1: watch backend
tail -f /tmp/housing_calc_backend.log

# In pane 2: watch frontend
tail -f /tmp/housing_calc_frontend.log

# In pane 3: watch ngrok
tail -f /tmp/housing_calc_ngrok.log

Now you have a single window with live output from all three services. Detach, go do other work, reattach at any time.

Long Deploys and Rebuilds

The deploy script for the housing calculator does a git pull, npm run build (Vite build), systemctl restart housing-calculator, and a warmup call. On a low-spec server, npm run build alone can take several minutes.

# scripts/deploy_restart.sh — runs systemctl + nginx reload + curl warmup
# Takes 3-10 minutes total on a low-spec box

Running this interactively over SSH is asking for a dropped connection mid-deploy. Run it in tmux:

tmux new -s deploy
cd /var/www/Housing-Calculator
./scripts/deploy_restart.sh
# Ctrl+B, D — detach

If the deploy fails, the error is preserved in the session. You reattach and read it.

Practical Habits

One session per concern. Keep housing for the live services and logs. Use calibration for data pipeline runs. Use deploy for deploys. Don’t reuse a session for something new until you’ve confirmed the previous job is done.

Check before creating. Always run tmux ls when you SSH in. A half-finished calibration run from last week is sitting in a session you forgot about — don’t start a new one on top of it.

Use scroll mode. Inside tmux, Ctrl+B, [ enters copy/scroll mode. Use arrow keys or PageUp/PageDown to read output that scrolled off the screen. Press q to exit.

Name sessions with context. calibration-full-2025-07 beats cal. Six months from now you’ll thank yourself.

Log to file anyway. Even with tmux, pipe long jobs to a log file. The session can be killed. The log file can’t:

./run_backfills_and_validate.sh 2>&1 | tee /tmp/calibration-$(date +%Y%m%d).log

tee writes to both the terminal (so you see output live in tmux) and the log file (so you have a record even if the session dies).

When Background Processes Are Still Fine

tmux isn’t always the answer. The housing calculator’s manage_services.sh starts the backend, frontend, and ngrok as background processes, and that’s the right call there — those are persistent daemons managed by PID files and pgrep/pkill.

For actual long-lived production services, use systemctl. The housing calculator backend runs as a systemd service (housing-calculator.service). That handles restart-on-failure, boot persistence, and log management via journalctl. tmux is for human-attended jobs: data pipelines, builds, one-off scripts, anything where you want to watch output and confirm results.

Summary

Situation	Use
SSH background daemon	`systemctl`
Long script, want to detach	`tmux new -s name`
Multi-step pipeline	`tmux` + `tee` to log
Multi-service log monitoring	`tmux` split panes
Anything you can afford to lose	`cmd > log 2>&1 &`

The pattern is simple: if you need to be able to walk away and come back to a terminal, it goes in tmux. If it’s a permanent service, it goes in systemctl. The housing calculator uses both — systemd for the running app, tmux for the maintenance work around it.