Backup-Dashboard/README.md

# Backup-Dashboard

Proof of concept backup dashboard, intended to display backup jobs from multiple sources.

```
collectors ──POST /api/v1/ingest (API key)──▶ ┌──────────────┐
                                              │ FastAPI +    │
browser ──nginx (TLS)──▶ /api/* ─────────────▶│ SQLite       │
         │  └─ static files: public/          └──────────────┘
         └─ auth_request ─▶ oauth2-proxy ─▶ Microsoft Entra ID (Microsoft 365 sign-in)
```

- `public/`: static frontend (no build step). Add `?mock=1` to the URL to use built-in sample data.
- `backend/`: FastAPI API backed by a single SQLite file.
- `deploy/`: nginx site and systemd unit.
- `collectors/`: PowerShell scripts that report results from backup products ([Veeam B&R](collectors/veeam/README.md), [Azure Backup / MARS](collectors/mars/README.md), [Windows Server Backup](collectors/wsb/README.md), [Veeam Backup for M365](collectors/vb365/README.md)). Code they share (config, logging, posting, state, installing) is in `collectors/common/BackupDashboard.Collector.psm1`.

## Data model

- **Tenant** (company) → **Job** → **Run**.
- A job is keyed by `(tenant, source_system, job.external_id)`, so renaming a job at the source updates it instead of creating a duplicate.
- A run has `status` (`running` | `success` | `warning` | `failed`), `started_at` and `finished_at`.
  The dashboard's **Last Run** is the most recent `finished_at`. While a job is running it shows
  **Running**, and Last Run still shows when the previous run finished.
- Runs older than `BACKUP_DASH_RETENTION_DAYS` (default 90, minimum 30) are purged every 6 hours.

## Reporting results (collectors)

Each collector (a script on a Veeam server, a webhook relay, etc.) gets its own API key:

```bash
cd backend && python manage.py create-key veeam-acme
```

Then it POSTs one report per run. Send `running` when a run starts (optional) and the final status
when it ends. Reports with the same `started_at` (or `run.external_id`) update the same run, and
retries are safe.

```bash
curl -X POST https://backups.example.com/api/v1/ingest \
  -H "Authorization: Bearer bdk_..." -H "Content-Type: application/json" \
  -d '{
    "tenant": "Acme Corp",
    "source_system": "Veeam B&R",
    "job": { "external_id": "4f1c…", "name": "File Server – Nightly", "next_run_at": "2026-10-05T22:00:00-04:00" },
    "run": {
      "status": "success",
      "started_at": "2026-10-04T22:00:05-04:00",
      "finished_at": "2026-10-04T22:47:12-04:00",
      "message": "Job completed successfully.",
      "size_bytes": 128849018880,
      "details": { "repository": "REPO-01" }
    }
  }'
```

- All timestamps must include a timezone offset (`Get-Date -Format o` in PowerShell does this).
- `job.next_run_at` controls the **Overdue** flag. If you omit it, the stored value is kept. If you send `null`, it's cleared.
- `run` can be omitted to only register or update a job.
- Unknown fields are rejected (HTTP 422), so typos in collector scripts fail loudly.

## API (read)

| Endpoint | Returns |
|---|---|
| `GET /api/v1/jobs` | All jobs with current status, `running_since`, `last_run_at`, `next_run_at` |
| `GET /api/v1/jobs/{id}` | One job, plus the last finished run's duration, size, message and details |
| `GET /api/v1/jobs/{id}/runs?days=30` | Run history (no messages), newest first |
| `GET /api/v1/runs/{id}` | One run including message and details |
| `GET /api/v1/tenants` | Companies with jobs, for choosing a report |
| `GET /api/v1/reports/tenant?tenant=…&days=30` | One company's jobs and every run in the period (messages for warnings/failures only) |
| `GET /api/v1/reports/tenant.csv?tenant=…&days=30&tz=America/Edmonton` | The same as CSV, with full messages and times in `tz` |
| `GET /api/v1/health` | `{"status": "ok"}` |

The OpenAPI schema is at `/api/v1/openapi.json`.

## Reports

**Reports** in the dashboard's top bar (or a company's name in a job's detail) opens a backup report
for one company over the last 7, 30 or 90 days. The address carries the choice, e.g.
`/report.html?tenant=Alpha%20Construction&days=30`, so a report can be bookmarked or linked.

- **On the page:** a summary (jobs, runs, success rate, warnings, failures, overdue now); per job,
  its run counts, success rate, last result and last successful backup; a day-by-day strip; and
  every warning and failure with its message. **Print / Save as PDF** prints it light and without
  the page controls, whatever the screen's theme.
- **CSV:** every run in the period, with full messages and times in the browser's time zone. It
  opens directly in Excel. Values that a spreadsheet would treat as formulas are prefixed with `'`.
- **Success rate** is successful runs ÷ finished runs; runs still in progress don't count against it.
- **Days before a job was monitored** (before its earliest known run) show as *Not monitored
  yet*, not as missed backups.
- Periods are limited by retention (`BACKUP_DASH_RETENTION_DAYS`, 90 by default). Anyone who can
  sign in can view reports.

## Authentication

- **Dashboard users:** Microsoft 365 sign-in. nginx checks every request with
  [oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/), which signs users in with Microsoft
  Entra ID. Who may sign in is controlled in Entra (users or a group assigned to the app). Every
  signed-in user can see every tenant. Sessions last 12 hours; signing in again is usually silent.
  The dashboard shows who's signed in; **Sign out** also signs out of Microsoft in that browser,
  otherwise the next visit would sign straight back in.
- **Admins:** the sign-in names in `BACKUP_DASH_ADMINS` (in `/etc/backup-dashboard/backup-dashboard.env`,
  comma-separated) get an **Admin** link to the API keys page. Restart the API after changing it.
- **Collectors:** per-collector bearer keys, checked by the app. Only a SHA-256 hash is stored.
  `/api/v1/ingest` is the only path that doesn't need Microsoft sign-in.

### Managing API keys

On the **Admin** page (`/admin.html`), admins can:
- **Create** a key. It's shown once, with a copy button. Paste it when a collector's installer
  asks for the API key.
- **See** each key's status, when it was last used and for which company, and who created it.
  Active keys quiet for over 24 hours are flagged, since collectors report hourly.
- **Revoke** a key. The collectors using it are refused from their next run. This can't be undone,
  and names stay taken, so the history stays unambiguous.

The same is possible from the server with `manage.py` (`create-key`, `list-keys`, `revoke-key`),
recorded as created/revoked by `manage.py (<user>)`.

### Microsoft 365 sign-in setup

**In the Microsoft Entra admin center** (entra.microsoft.com), as an admin of your tenant:

1. *App registrations → New registration*. Name: `Backup Dashboard`. Supported account types:
   **Accounts in this organizational directory only** (single tenant). Redirect URI: platform
   **Web**, `https://<your dashboard host>/oauth2/callback`. Register.
2. On its *Overview*, note the **Application (client) ID** and **Directory (tenant) ID**.
3. *Certificates & secrets → New client secret*. Copy the secret's **Value** now (it's only shown
   once). Note its expiry date: sign-in stops working when it expires, so set a reminder to
   create a new one before then.
4. *Enterprise applications → Backup Dashboard → Properties*: set **Assignment required** to
   **Yes**. Then *Users and groups → Add user/group* and add the people (or a group) who should
   see the dashboard. Everyone else in the tenant is refused by Microsoft before reaching it.

No API permissions beyond the default `User.Read` are needed.

**On the dashboard server:**

1. Install oauth2-proxy 7.8 or newer from its
   [releases](https://github.com/oauth2-proxy/oauth2-proxy/releases) as `/usr/local/bin/oauth2-proxy`,
   and create its user: `sudo useradd --system --no-create-home oauth2-proxy`.
2. Install the config and the client secret, readable only by root and oauth2-proxy:
   ```bash
   sudo install -d -m 750 -g oauth2-proxy /etc/oauth2-proxy
   sudo install -m 640 -g oauth2-proxy deploy/oauth2-proxy.cfg /etc/oauth2-proxy/backup-dashboard.cfg
   sudo sh -c 'umask 027; printf %s "<client secret value>" > /etc/oauth2-proxy/client-secret' && sudo chgrp oauth2-proxy /etc/oauth2-proxy/client-secret
   ```
   In `/etc/oauth2-proxy/backup-dashboard.cfg`, fill in `<TENANT_ID>`, `<APPLICATION_CLIENT_ID>`,
   `<DASHBOARD_HOST>`, and `<COOKIE_SECRET>`, generated with:
   `openssl rand -base64 32 | tr -- '+/' '-_'`
3. Install `deploy/oauth2-proxy.service` into `/etc/systemd/system/`, then
   `sudo systemctl daemon-reload && sudo systemctl enable --now oauth2-proxy`.
   Check it with `journalctl -u oauth2-proxy -n 20`: it should say "OAuthProxy configured for
   Microsoft Entra ID".
4. Install the new `deploy/nginx.conf` (keeping your server name and certificate paths), then
   `sudo nginx -t && sudo systemctl reload nginx`. Basic auth and its htpasswd file are no longer
   used.

**Troubleshooting:** sign-in problems show in `journalctl -u oauth2-proxy`. Microsoft's own
errors (wrong redirect URI, user not assigned, expired secret) appear on its sign-in page with an
`AADSTS…` code.

## Development

```bash
cd backend
python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt -r requirements-dev.txt
pytest
python manage.py --create seed-demo            # 30 days of fake history in ./backup-dashboard.db
BACKUP_DASH_SERVE_STATIC=1 uvicorn --factory app.main:create_app --reload
# → http://localhost:8000
```

Collector tests (PowerShell 7 + Pester; see [collectors/tests](collectors/tests/README.md)):

```powershell
./collectors/tests/Invoke-CollectorTests.ps1
```

## Deployment (Linux, nginx)

1. Copy the repo to `/opt/backup-dashboard`, create a venv at `/opt/backup-dashboard/venv`, and run `pip install -r backend/requirements.txt`.
2. Copy `deploy/backup-dashboard.env` to `/etc/backup-dashboard/backup-dashboard.env`. Both the service and `manage.py` read their settings from it.
3. Create a `backupdash` system user, then install `deploy/backup-dashboard-api.service` into `/etc/systemd/system/` and enable it. The database is created on first start.
4. Copy `public/` to `/var/www/backup-dashboard/public`, install `deploy/nginx.conf`, get a certificate with certbot, and set up Microsoft 365 sign-in (see [Microsoft 365 sign-in setup](#microsoft-365-sign-in-setup)).
5. Run `manage.py` as the service user. It finds the database through the env file:
   ```bash
   cd /opt/backup-dashboard/backend
   sudo -u backupdash /opt/backup-dashboard/venv/bin/python manage.py create-key veeam-acme
   ```
   It prints the database path it's using and won't create a new database unless you pass `--create`.

Back up the SQLite file with `sqlite3 backup-dashboard.db ".backup copy.db"`. Don't use a plain `cp`
while the service is running.

### Updating the server

After changes are merged into `main`, on the dashboard server:

1. **Back up the database.** It takes seconds, and an update may upgrade the database's structure
   when the API restarts, which only this backup can reverse. Needs the `sqlite3` package:
   ```bash
   sudo -u backupdash sqlite3 /var/lib/backup-dashboard/backup-dashboard.db ".backup /var/lib/backup-dashboard/before-update.db"
   ```
2. **Pull, and see what changed** (the list tells you whether steps 7 and 8 apply):
   ```bash
   sudo git -C /opt/backup-dashboard pull
   ```
   ```bash
   git -C /opt/backup-dashboard diff --stat ORIG_HEAD HEAD
   ```
3. **Python packages.** Does nothing unless `backend/requirements.txt` changed:
   ```bash
   sudo /opt/backup-dashboard/venv/bin/pip install -q -r /opt/backup-dashboard/backend/requirements.txt
   ```
4. **Web files.** nginx serves its own copy in `/var/www`, so a pull alone doesn't change what
   browsers get. If this step is skipped, new pages quietly show the dashboard instead (nginx
   falls back to `index.html` for files it doesn't have):
   ```bash
   sudo rsync -a /opt/backup-dashboard/public/ /var/www/backup-dashboard/public/
   ```
   Browsers pick up the new files on their next page load.
5. **Restart the API**, which loads the new code and settings and upgrades the database if needed:
   ```bash
   sudo systemctl restart backup-dashboard-api
   ```
6. **Check it's healthy.** The first should print `{"status":"ok"}`; the log shouldn't show errors:
   ```bash
   curl -s http://127.0.0.1:8000/api/v1/health
   ```
   ```bash
   journalctl -u backup-dashboard-api -n 20 --no-pager
   ```
7. **If files in `deploy/` changed:** those are templates whose installed copies live elsewhere,
   so a pull doesn't update them. Compare each changed one with its installed copy (e.g.
   `diff /opt/backup-dashboard/deploy/oauth2-proxy.service /etc/systemd/system/oauth2-proxy.service`)
   and bring the changes across:

   | Changed in the repo | Installed copy | After updating it |
   |---|---|---|
   | `deploy/backup-dashboard.env` | `/etc/backup-dashboard/backup-dashboard.env` | Add any new settings, keeping your values; restart the API |
   | `deploy/backup-dashboard-api.service` | `/etc/systemd/system/backup-dashboard-api.service` | `sudo systemctl daemon-reload`, restart the API |
   | `deploy/nginx.conf` | Your nginx site config | Keep your server name and certificate paths; `sudo nginx -t && sudo systemctl reload nginx` |
   | `deploy/oauth2-proxy.cfg` | `/etc/oauth2-proxy/backup-dashboard.cfg` | Keep your IDs and secrets; `sudo systemctl restart oauth2-proxy` |
   | `deploy/oauth2-proxy.service` | `/etc/systemd/system/oauth2-proxy.service` | `sudo systemctl daemon-reload && sudo systemctl restart oauth2-proxy` |

8. **If files in `collectors/` changed:** the backup servers keep running their installed copy.
   On each affected server, copy over the whole `collectors` folder, re-run that collector's
   installer, and do a dry run (see its README). The changes are already covered by the
   collector tests (`./collectors/tests/Invoke-CollectorTests.ps1`) if you want to run them first.

**If something goes wrong**, go back to the version from before step 2 (`ORIG_HEAD`). The order
matters: an older API won't start on a database a newer one has upgraded, so restore the database
before starting it.

1. Stop the API:
   ```bash
   sudo systemctl stop backup-dashboard-api
   ```
2. Check whether the update upgraded the database: if these two numbers differ, it did.
   ```bash
   for db in backup-dashboard before-update; do sudo -u backupdash sqlite3 /var/lib/backup-dashboard/$db.db "PRAGMA user_version"; done
   ```
   If so, restore the backup from step 1 (`.restore`, not `cp`: it also deals with SQLite's `-wal`
   file). Runs reported since the backup drop out of the history, since collectors have already
   marked them as sent; backups themselves are unaffected.
   ```bash
   sudo -u backupdash sqlite3 /var/lib/backup-dashboard/backup-dashboard.db ".restore /var/lib/backup-dashboard/before-update.db"
   ```
3. Go back to the previous code and web files, then start the API:
   ```bash
   sudo git -C /opt/backup-dashboard checkout ORIG_HEAD
   ```
   ```bash
   sudo rsync -a /opt/backup-dashboard/public/ /var/www/backup-dashboard/public/ && sudo systemctl start backup-dashboard-api
   ```

Once the problem is fixed, return to `main` with `sudo git -C /opt/backup-dashboard checkout main`
and update as above.