Document updating the dashboard server

The README only covered the first install, so the admin page update
stalled at a step that wasn't written down: nginx serves its own copy
of public/ in /var/www, and without copying it a new page quietly falls
back to index.html.

"Updating the server" covers: backing up the database, pulling and
listing what changed, Python packages, copying the web files,
restarting, health checks, which deploy/ templates map to which
installed files, and redeploying collectors when they change. A
rollback section restores the database before starting the older API
(which refuses a newer database), using sqlite3 .restore rather than
cp because of the WAL file.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

README.md

@@ -173,3 +173,88 @@ Collector tests (PowerShell 7 + Pester; see [collectors/tests](collectors/tests/
 
 Back up the SQLite file with `sqlite3 backup-dashboard.db ".backup copy.db"`. Don't use a plain `cp`
 while the service is running.
+
+### Updating the server
+
+After changes are merged into `main`, on the dashboard server:
+
+1. **Back up the database.** It takes seconds, and an update may upgrade the database's structure
+   when the API restarts, which only this backup can reverse. Needs the `sqlite3` package:
+   ```bash
+   sudo -u backupdash sqlite3 /var/lib/backup-dashboard/backup-dashboard.db ".backup /var/lib/backup-dashboard/before-update.db"
+   ```
+2. **Pull, and see what changed** (the list tells you whether steps 7 and 8 apply):
+   ```bash
+   sudo git -C /opt/backup-dashboard pull
+   ```
+   ```bash
+   git -C /opt/backup-dashboard diff --stat ORIG_HEAD HEAD
+   ```
+3. **Python packages.** Does nothing unless `backend/requirements.txt` changed:
+   ```bash
+   sudo /opt/backup-dashboard/venv/bin/pip install -q -r /opt/backup-dashboard/backend/requirements.txt
+   ```
+4. **Web files.** nginx serves its own copy in `/var/www`, so a pull alone doesn't change what
+   browsers get. If this step is skipped, new pages quietly show the dashboard instead (nginx
+   falls back to `index.html` for files it doesn't have):
+   ```bash
+   sudo rsync -a /opt/backup-dashboard/public/ /var/www/backup-dashboard/public/
+   ```
+   Browsers pick up the new files on their next page load.
+5. **Restart the API**, which loads the new code and settings and upgrades the database if needed:
+   ```bash
+   sudo systemctl restart backup-dashboard-api
+   ```
+6. **Check it's healthy.** The first should print `{"status":"ok"}`; the log shouldn't show errors:
+   ```bash
+   curl -s http://127.0.0.1:8000/api/v1/health
+   ```
+   ```bash
+   journalctl -u backup-dashboard-api -n 20 --no-pager
+   ```
+7. **If files in `deploy/` changed:** those are templates whose installed copies live elsewhere,
+   so a pull doesn't update them. Compare each changed one with its installed copy (e.g.
+   `diff /opt/backup-dashboard/deploy/oauth2-proxy.service /etc/systemd/system/oauth2-proxy.service`)
+   and bring the changes across:
+
+   | Changed in the repo | Installed copy | After updating it |
+   |---|---|---|
+   | `deploy/backup-dashboard.env` | `/etc/backup-dashboard/backup-dashboard.env` | Add any new settings, keeping your values; restart the API |
+   | `deploy/backup-dashboard-api.service` | `/etc/systemd/system/backup-dashboard-api.service` | `sudo systemctl daemon-reload`, restart the API |
+   | `deploy/nginx.conf` | Your nginx site config | Keep your server name and certificate paths; `sudo nginx -t && sudo systemctl reload nginx` |
+   | `deploy/oauth2-proxy.cfg` | `/etc/oauth2-proxy/backup-dashboard.cfg` | Keep your IDs and secrets; `sudo systemctl restart oauth2-proxy` |
+   | `deploy/oauth2-proxy.service` | `/etc/systemd/system/oauth2-proxy.service` | `sudo systemctl daemon-reload && sudo systemctl restart oauth2-proxy` |
+
+8. **If files in `collectors/` changed:** the backup servers keep running their installed copy.
+   On each affected server, copy over the whole `collectors` folder, re-run that collector's
+   installer, and do a dry run (see its README). The changes are already covered by the
+   collector tests (`./collectors/tests/Invoke-CollectorTests.ps1`) if you want to run them first.
+
+**If something goes wrong**, go back to the version from before step 2 (`ORIG_HEAD`). The order
+matters: an older API won't start on a database a newer one has upgraded, so restore the database
+before starting it.
+
+1. Stop the API:
+   ```bash
+   sudo systemctl stop backup-dashboard-api
+   ```
+2. Check whether the update upgraded the database: if these two numbers differ, it did.
+   ```bash
+   for db in backup-dashboard before-update; do sudo -u backupdash sqlite3 /var/lib/backup-dashboard/$db.db "PRAGMA user_version"; done
+   ```
+   If so, restore the backup from step 1 (`.restore`, not `cp`: it also deals with SQLite's `-wal`
+   file). Runs reported since the backup drop out of the history, since collectors have already
+   marked them as sent; backups themselves are unaffected.
+   ```bash
+   sudo -u backupdash sqlite3 /var/lib/backup-dashboard/backup-dashboard.db ".restore /var/lib/backup-dashboard/before-update.db"
+   ```
+3. Go back to the previous code and web files, then start the API:
+   ```bash
+   sudo git -C /opt/backup-dashboard checkout ORIG_HEAD
+   ```
+   ```bash
+   sudo rsync -a /opt/backup-dashboard/public/ /var/www/backup-dashboard/public/ && sudo systemctl start backup-dashboard-api
+   ```
+
+Once the problem is fixed, return to `main` with `sudo git -C /opt/backup-dashboard checkout main`
+and update as above.