kyleandClaude Opus 5.5
8d0e6dca71
The Veeam, MARS, WSB and VB365 collectors each carried identical copies of logging, ISO time conversion, property lookup, posting to the API, config loading, TLS setup, the sent-runs state file and the 401 check, and their installers repeated the same key prompt, locked-down config, file copy and scheduled-task registration. These now live in collectors/common/BackupDashboard.Collector.psm1: - Initialize-Collector / Test-RunSent / Send-JobReport / Test-AuthRejected / Complete-Collector for collector runs - Write-Log, Get-ErrorText, ConvertTo-IsoTime, Get-FirstValue - Install-CollectorTask for the shared half of each installer Collectors find the module next to themselves (installed) or in ..\common (repo); installers copy it into Program Files beside the collector. The diagnostic scripts stay standalone single files. No behavior change: a harness ran each collector against its mock (12 scenarios: dry run, two real runs against a stub API, state file, logs, exit codes, rejected key, en-GB, backfill) before and after, and all 84 outputs are identical. Installers, run with stand-ins for the Windows-only commands, write identical configs and tasks; the only differences are the module being copied and the summary line naming the shell's path. Co-Authored-By: Claude Opus 5.5 <[email protected]>
109 lines
5.4 KiB
Markdown
109 lines
5.4 KiB
Markdown
# Backup-Dashboard
|
||
|
||
Proof of concept backup dashboard, intended to display backup jobs from multiple sources.
|
||
|
||
```
|
||
collectors ──POST /api/v1/ingest (API key)──▶ ┌──────────────┐
|
||
│ FastAPI + │
|
||
browser ──nginx (basic auth, TLS)──▶ /api/* ─▶│ SQLite │
|
||
└─ static files: public/ └──────────────┘
|
||
```
|
||
|
||
- `public/`: static frontend (no build step). Add `?mock=1` to the URL to use built-in sample data.
|
||
- `backend/`: FastAPI API backed by a single SQLite file.
|
||
- `deploy/`: nginx site and systemd unit.
|
||
- `collectors/`: PowerShell scripts that report results from backup products ([Veeam B&R](collectors/veeam/README.md), [Azure Backup / MARS](collectors/mars/README.md), [Windows Server Backup](collectors/wsb/README.md), [Veeam Backup for M365](collectors/vb365/README.md)). Code they share (config, logging, posting, state, installing) is in `collectors/common/BackupDashboard.Collector.psm1`.
|
||
|
||
## Data model
|
||
|
||
- **Tenant** (company) → **Job** → **Run**.
|
||
- A job is keyed by `(tenant, source_system, job.external_id)`, so renaming a job at the source updates it instead of creating a duplicate.
|
||
- A run has `status` (`running` | `success` | `warning` | `failed`), `started_at` and `finished_at`.
|
||
The dashboard's **Last Run** is the most recent `finished_at`. While a job is running it shows
|
||
**Running**, and Last Run still shows when the previous run finished.
|
||
- Runs older than `BACKUP_DASH_RETENTION_DAYS` (default 90, minimum 30) are purged every 6 hours.
|
||
|
||
## Reporting results (collectors)
|
||
|
||
Each collector (a script on a Veeam server, a webhook relay, etc.) gets its own API key:
|
||
|
||
```bash
|
||
cd backend && python manage.py create-key veeam-acme
|
||
```
|
||
|
||
Then it POSTs one report per run. Send `running` when a run starts (optional) and the final status
|
||
when it ends. Reports with the same `started_at` (or `run.external_id`) update the same run, and
|
||
retries are safe.
|
||
|
||
```bash
|
||
curl -X POST https://backups.example.com/api/v1/ingest \
|
||
-H "Authorization: Bearer bdk_..." -H "Content-Type: application/json" \
|
||
-d '{
|
||
"tenant": "Acme Corp",
|
||
"source_system": "Veeam B&R",
|
||
"job": { "external_id": "4f1c…", "name": "File Server – Nightly", "next_run_at": "2026-10-05T22:00:00-04:00" },
|
||
"run": {
|
||
"status": "success",
|
||
"started_at": "2026-10-04T22:00:05-04:00",
|
||
"finished_at": "2026-10-04T22:47:12-04:00",
|
||
"message": "Job completed successfully.",
|
||
"size_bytes": 128849018880,
|
||
"details": { "repository": "REPO-01" }
|
||
}
|
||
}'
|
||
```
|
||
|
||
- All timestamps must include a timezone offset (`Get-Date -Format o` in PowerShell does this).
|
||
- `job.next_run_at` controls the **Overdue** flag. If you omit it, the stored value is kept. If you send `null`, it's cleared.
|
||
- `run` can be omitted to only register or update a job.
|
||
- Unknown fields are rejected (HTTP 422), so typos in collector scripts fail loudly.
|
||
|
||
## API (read)
|
||
|
||
| Endpoint | Returns |
|
||
|---|---|
|
||
| `GET /api/v1/jobs` | All jobs with current status, `running_since`, `last_run_at`, `next_run_at` |
|
||
| `GET /api/v1/jobs/{id}` | One job, plus the last finished run's duration, size, message and details |
|
||
| `GET /api/v1/jobs/{id}/runs?days=30` | Run history (no messages), newest first |
|
||
| `GET /api/v1/runs/{id}` | One run including message and details |
|
||
| `GET /api/v1/health` | `{"status": "ok"}` |
|
||
|
||
The OpenAPI schema is at `/api/v1/openapi.json`.
|
||
|
||
## Authentication
|
||
|
||
- **Dashboard users:** nginx basic auth over HTTPS (`deploy/nginx.conf`). Every user can see every tenant.
|
||
*Planned:* Microsoft 365 sign-in through [oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/)
|
||
with the Entra ID provider and nginx `auth_request`. That's an nginx-only change. The frontend
|
||
already tells the user to reload when a session expires (HTTP 401).
|
||
- **Collectors:** per-collector bearer keys, checked by the app. Only a SHA-256 hash is stored.
|
||
To revoke one, run `python manage.py revoke-key NAME`.
|
||
|
||
## Development
|
||
|
||
```bash
|
||
cd backend
|
||
python -m venv .venv && . .venv/bin/activate
|
||
pip install -r requirements.txt -r requirements-dev.txt
|
||
pytest
|
||
python manage.py --create seed-demo # 30 days of fake history in ./backup-dashboard.db
|
||
BACKUP_DASH_SERVE_STATIC=1 uvicorn --factory app.main:create_app --reload
|
||
# → http://localhost:8000
|
||
```
|
||
|
||
## Deployment (Linux, nginx)
|
||
|
||
1. Copy the repo to `/opt/backup-dashboard`, create a venv at `/opt/backup-dashboard/venv`, and run `pip install -r backend/requirements.txt`.
|
||
2. Copy `deploy/backup-dashboard.env` to `/etc/backup-dashboard/backup-dashboard.env`. Both the service and `manage.py` read their settings from it.
|
||
3. Create a `backupdash` system user, then install `deploy/backup-dashboard-api.service` into `/etc/systemd/system/` and enable it. The database is created on first start.
|
||
4. Copy `public/` to `/var/www/backup-dashboard/public`, install `deploy/nginx.conf`, get a certificate with certbot, and create users with `htpasswd -B`.
|
||
5. Run `manage.py` as the service user. It finds the database through the env file:
|
||
```bash
|
||
cd /opt/backup-dashboard/backend
|
||
sudo -u backupdash /opt/backup-dashboard/venv/bin/python manage.py create-key veeam-acme
|
||
```
|
||
It prints the database path it's using and won't create a new database unless you pass `--create`.
|
||
|
||
Back up the SQLite file with `sqlite3 backup-dashboard.db ".backup copy.db"`. Don't use a plain `cp`
|
||
while the service is running.
|