Compare commits
72
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2297187f88 | ||
|
|
abb684ff0d | ||
|
|
d4e821799c | ||
|
|
c5fe69a813 | ||
|
|
2fa9eea274 | ||
|
|
8d0e6dca71 | ||
|
|
118a7f978d | ||
|
|
3f407ba76e | ||
|
|
dd3e2fe885 | ||
|
|
51b97c4556 | ||
|
|
555e928e01 | ||
|
|
b625a8af65 | ||
|
|
a14b1b0dc9 | ||
|
|
5982fc9434 | ||
|
|
10181e014b | ||
|
|
b4b703ceee | ||
|
|
10bfc7a3aa | ||
|
|
960839ed76 | ||
|
|
f0b4d84c0b | ||
|
|
067b714678 | ||
|
|
3355a7d4bc | ||
|
|
e916afaae1 | ||
|
|
1f96d11a27 | ||
|
|
41c4fa2700 | ||
|
|
a84a35463b | ||
|
|
96b729bec8 | ||
|
|
ae574fa2b6 | ||
|
|
e50711b93b | ||
|
|
8fd17e3817 | ||
|
|
ba798284df | ||
|
|
9ae93cfbc2 | ||
|
|
986cf3be88 | ||
|
|
17ee303670 | ||
|
|
effafc5fa0 | ||
|
|
2318579baa | ||
|
|
1f02c1c84b | ||
|
|
3b1aee7016 | ||
|
|
d3ab05416d | ||
|
|
728b5771cc | ||
|
|
f55a44f2e6 | ||
|
|
b8452e1fbb | ||
|
|
d12e00f69e | ||
|
|
33c1613ef9 | ||
|
|
8e29b9cbc4 | ||
|
|
49bf9ef094 | ||
|
|
4d183da7bd | ||
|
|
25f7717ace | ||
|
|
be23e03c5f | ||
|
|
65c587da0d | ||
|
|
29f961f80e | ||
|
|
02789b1910 | ||
|
|
54a6c0b3d6 | ||
|
|
f11345bb0f | ||
|
|
7217b3e9a8 | ||
|
|
47ec7dc9a7 | ||
|
|
15ea9e0fec | ||
|
|
fe4bd6827c | ||
|
|
4e57eb1a46 | ||
|
|
670994b937 | ||
|
|
a7c3169606 | ||
|
|
938a0a2af0 | ||
|
|
f7e029db8a | ||
|
|
2df1a75fd5 | ||
|
|
4d8ee209a5 | ||
|
|
6c486fbaf3 | ||
|
|
fdc7deff16 | ||
|
|
0391807027 | ||
|
|
b0c560377b | ||
|
|
17346cc058 | ||
|
|
65eb3c88f3 | ||
|
|
996fe33a40 | ||
|
|
bf752a904f |
@@ -0,0 +1,8 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.venv/
|
||||
venv/
|
||||
.pytest_cache/
|
||||
*.db
|
||||
*.db-wal
|
||||
*.db-shm
|
||||
@@ -1,3 +1,281 @@
|
||||
# Backup-Dashboard
|
||||
|
||||
Proof of concept backup dashboard, intended to display backup jobs from multiple sources
|
||||
Proof of concept backup dashboard, intended to display backup jobs from multiple sources.
|
||||
|
||||
```
|
||||
collectors ──POST /api/v1/ingest (API key)──▶ ┌──────────────┐
|
||||
│ FastAPI + │
|
||||
browser ──nginx (TLS)──▶ /api/* ─────────────▶│ SQLite │
|
||||
│ └─ static files: public/ └──────────────┘
|
||||
└─ auth_request ─▶ oauth2-proxy ─▶ Microsoft Entra ID (Microsoft 365 sign-in)
|
||||
```
|
||||
|
||||
- `public/`: static frontend (no build step). Add `?mock=1` to the URL to use built-in sample data.
|
||||
- `backend/`: FastAPI API backed by a single SQLite file.
|
||||
- `deploy/`: nginx site and systemd unit.
|
||||
- `collectors/`: PowerShell scripts that report results from backup products ([Veeam B&R](collectors/veeam/README.md), [Azure Backup / MARS](collectors/mars/README.md), [Windows Server Backup](collectors/wsb/README.md), [Veeam Backup for M365](collectors/vb365/README.md)). Code they share (config, logging, posting, state, installing) is in `collectors/common/BackupDashboard.Collector.psm1`.
|
||||
|
||||
## Data model
|
||||
|
||||
- **Tenant** (company) → **Job** → **Run**.
|
||||
- A job is keyed by `(tenant, source_system, job.external_id)`, so renaming a job at the source updates it instead of creating a duplicate.
|
||||
- A run has `status` (`running` | `success` | `warning` | `failed`), `started_at` and `finished_at`.
|
||||
The dashboard's **Last Run** is the most recent `finished_at`. While a job is running it shows
|
||||
**Running**, and Last Run still shows when the previous run finished.
|
||||
- Runs older than `BACKUP_DASH_RETENTION_DAYS` (default 90, minimum 30) are purged every 6 hours.
|
||||
|
||||
## Reporting results (collectors)
|
||||
|
||||
Each collector (a script on a Veeam server, a webhook relay, etc.) gets its own API key:
|
||||
|
||||
```bash
|
||||
cd backend && python manage.py create-key veeam-acme
|
||||
```
|
||||
|
||||
Then it POSTs one report per run. Send `running` when a run starts (optional) and the final status
|
||||
when it ends. Reports with the same `started_at` (or `run.external_id`) update the same run, and
|
||||
retries are safe.
|
||||
|
||||
```bash
|
||||
curl -X POST https://backups.example.com/api/v1/ingest \
|
||||
-H "Authorization: Bearer bdk_..." -H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"tenant": "Acme Corp",
|
||||
"source_system": "Veeam B&R",
|
||||
"job": { "external_id": "4f1c…", "name": "File Server – Nightly", "next_run_at": "2026-10-05T22:00:00-04:00" },
|
||||
"run": {
|
||||
"status": "success",
|
||||
"started_at": "2026-10-04T22:00:05-04:00",
|
||||
"finished_at": "2026-10-04T22:47:12-04:00",
|
||||
"message": "Job completed successfully.",
|
||||
"size_bytes": 128849018880,
|
||||
"details": { "repository": "REPO-01" }
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
- All timestamps must include a timezone offset (`Get-Date -Format o` in PowerShell does this).
|
||||
- `job.next_run_at` controls the **Overdue** flag. If you omit it, the stored value is kept. If you send `null`, it's cleared.
|
||||
- `run` can be omitted to only register or update a job.
|
||||
- Unknown fields are rejected (HTTP 422), so typos in collector scripts fail loudly.
|
||||
|
||||
## API (read)
|
||||
|
||||
| Endpoint | Returns |
|
||||
|---|---|
|
||||
| `GET /api/v1/jobs` | All jobs with current status, `running_since`, `last_run_at`, `next_run_at` |
|
||||
| `GET /api/v1/jobs/{id}` | One job, plus the last finished run's duration, size, message and details |
|
||||
| `GET /api/v1/jobs/{id}/runs?days=30` | Run history (no messages), newest first |
|
||||
| `GET /api/v1/runs/{id}` | One run including message and details |
|
||||
| `GET /api/v1/tenants` | Companies with jobs, for choosing a report |
|
||||
| `GET /api/v1/reports/tenant?tenant=…&days=30` | One company's jobs and every run in the period (messages for warnings/failures only) |
|
||||
| `GET /api/v1/reports/tenant.csv?tenant=…&days=30&tz=America/Edmonton` | The same as CSV, with full messages and times in `tz` |
|
||||
| `GET /api/v1/health` | `{"status": "ok"}` |
|
||||
|
||||
The OpenAPI schema is at `/api/v1/openapi.json`.
|
||||
|
||||
## Reports
|
||||
|
||||
**Reports** in the dashboard's top bar (or a company's name in a job's detail) opens a backup report
|
||||
for one company over the last 7, 30 or 90 days. The address carries the choice, e.g.
|
||||
`/report.html?tenant=Alpha%20Construction&days=30`, so a report can be bookmarked or linked.
|
||||
|
||||
- **On the page:** a summary (jobs, runs, success rate, warnings, failures, overdue now); per job,
|
||||
its run counts, success rate, last result and last successful backup; a day-by-day strip; and
|
||||
every warning and failure with its message. **Print / Save as PDF** prints it light and without
|
||||
the page controls, whatever the screen's theme.
|
||||
- **CSV:** every run in the period, with full messages and times in the browser's time zone. It
|
||||
opens directly in Excel. Values that a spreadsheet would treat as formulas are prefixed with `'`.
|
||||
- **Success rate** is successful runs ÷ finished runs; runs still in progress don't count against it.
|
||||
- **Days before a job was monitored** (before its earliest known run) show as *Not monitored
|
||||
yet*, not as missed backups.
|
||||
- Periods are limited by retention (`BACKUP_DASH_RETENTION_DAYS`, 90 by default). Anyone who can
|
||||
sign in can view reports.
|
||||
|
||||
## Authentication
|
||||
|
||||
- **Dashboard users:** Microsoft 365 sign-in. nginx checks every request with
|
||||
[oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/), which signs users in with Microsoft
|
||||
Entra ID. Who may sign in is controlled in Entra (users or a group assigned to the app). Every
|
||||
signed-in user can see every tenant. Sessions last 12 hours; signing in again is usually silent.
|
||||
The dashboard shows who's signed in; **Sign out** also signs out of Microsoft in that browser,
|
||||
otherwise the next visit would sign straight back in.
|
||||
- **Admins:** the sign-in names in `BACKUP_DASH_ADMINS` (in `/etc/backup-dashboard/backup-dashboard.env`,
|
||||
comma-separated) get an **Admin** link to the API keys page. Restart the API after changing it.
|
||||
- **Collectors:** per-collector bearer keys, checked by the app. Only a SHA-256 hash is stored.
|
||||
`/api/v1/ingest` is the only path that doesn't need Microsoft sign-in.
|
||||
|
||||
### Managing API keys
|
||||
|
||||
On the **Admin** page (`/admin.html`), admins can:
|
||||
- **Create** a key. It's shown once, with a copy button. Paste it when a collector's installer
|
||||
asks for the API key.
|
||||
- **See** each key's status, when it was last used and for which company, and who created it.
|
||||
Active keys quiet for over 24 hours are flagged, since collectors report hourly.
|
||||
- **Revoke** a key. The collectors using it are refused from their next run. This can't be undone,
|
||||
and names stay taken, so the history stays unambiguous.
|
||||
|
||||
The same is possible from the server with `manage.py` (`create-key`, `list-keys`, `revoke-key`),
|
||||
recorded as created/revoked by `manage.py (<user>)`.
|
||||
|
||||
### Microsoft 365 sign-in setup
|
||||
|
||||
**In the Microsoft Entra admin center** (entra.microsoft.com), as an admin of your tenant:
|
||||
|
||||
1. *App registrations → New registration*. Name: `Backup Dashboard`. Supported account types:
|
||||
**Accounts in this organizational directory only** (single tenant). Redirect URI: platform
|
||||
**Web**, `https://<your dashboard host>/oauth2/callback`. Register.
|
||||
2. On its *Overview*, note the **Application (client) ID** and **Directory (tenant) ID**.
|
||||
3. *Certificates & secrets → New client secret*. Copy the secret's **Value** now (it's only shown
|
||||
once). Note its expiry date: sign-in stops working when it expires, so set a reminder to
|
||||
create a new one before then.
|
||||
4. *Enterprise applications → Backup Dashboard → Properties*: set **Assignment required** to
|
||||
**Yes**. Then *Users and groups → Add user/group* and add the people (or a group) who should
|
||||
see the dashboard. Everyone else in the tenant is refused by Microsoft before reaching it.
|
||||
|
||||
No API permissions beyond the default `User.Read` are needed.
|
||||
|
||||
**On the dashboard server:**
|
||||
|
||||
1. Install oauth2-proxy 7.8 or newer from its
|
||||
[releases](https://github.com/oauth2-proxy/oauth2-proxy/releases) as `/usr/local/bin/oauth2-proxy`,
|
||||
and create its user: `sudo useradd --system --no-create-home oauth2-proxy`.
|
||||
2. Install the config and the client secret, readable only by root and oauth2-proxy:
|
||||
```bash
|
||||
sudo install -d -m 750 -g oauth2-proxy /etc/oauth2-proxy
|
||||
sudo install -m 640 -g oauth2-proxy deploy/oauth2-proxy.cfg /etc/oauth2-proxy/backup-dashboard.cfg
|
||||
sudo sh -c 'umask 027; printf %s "<client secret value>" > /etc/oauth2-proxy/client-secret' && sudo chgrp oauth2-proxy /etc/oauth2-proxy/client-secret
|
||||
```
|
||||
In `/etc/oauth2-proxy/backup-dashboard.cfg`, fill in `<TENANT_ID>`, `<APPLICATION_CLIENT_ID>`,
|
||||
`<DASHBOARD_HOST>`, and `<COOKIE_SECRET>`, generated with:
|
||||
`openssl rand -base64 32 | tr -- '+/' '-_'`
|
||||
3. Install `deploy/oauth2-proxy.service` into `/etc/systemd/system/`, then
|
||||
`sudo systemctl daemon-reload && sudo systemctl enable --now oauth2-proxy`.
|
||||
Check it with `journalctl -u oauth2-proxy -n 20`: it should say "OAuthProxy configured for
|
||||
Microsoft Entra ID".
|
||||
4. Install the new `deploy/nginx.conf` (keeping your server name and certificate paths), then
|
||||
`sudo nginx -t && sudo systemctl reload nginx`. Basic auth and its htpasswd file are no longer
|
||||
used.
|
||||
|
||||
**Troubleshooting:** sign-in problems show in `journalctl -u oauth2-proxy`. Microsoft's own
|
||||
errors (wrong redirect URI, user not assigned, expired secret) appear on its sign-in page with an
|
||||
`AADSTS…` code.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
python -m venv .venv && . .venv/bin/activate
|
||||
pip install -r requirements.txt -r requirements-dev.txt
|
||||
pytest
|
||||
python manage.py --create seed-demo # 30 days of fake history in ./backup-dashboard.db
|
||||
BACKUP_DASH_SERVE_STATIC=1 uvicorn --factory app.main:create_app --reload
|
||||
# → http://localhost:8000
|
||||
```
|
||||
|
||||
Collector tests (PowerShell 7 + Pester; see [collectors/tests](collectors/tests/README.md)):
|
||||
|
||||
```powershell
|
||||
./collectors/tests/Invoke-CollectorTests.ps1
|
||||
```
|
||||
|
||||
## Deployment (Linux, nginx)
|
||||
|
||||
1. Copy the repo to `/opt/backup-dashboard`, create a venv at `/opt/backup-dashboard/venv`, and run `pip install -r backend/requirements.txt`.
|
||||
2. Copy `deploy/backup-dashboard.env` to `/etc/backup-dashboard/backup-dashboard.env`. Both the service and `manage.py` read their settings from it.
|
||||
3. Create a `backupdash` system user, then install `deploy/backup-dashboard-api.service` into `/etc/systemd/system/` and enable it. The database is created on first start.
|
||||
4. Copy `public/` to `/var/www/backup-dashboard/public`, install `deploy/nginx.conf`, get a certificate with certbot, and set up Microsoft 365 sign-in (see [Microsoft 365 sign-in setup](#microsoft-365-sign-in-setup)).
|
||||
5. Run `manage.py` as the service user. It finds the database through the env file:
|
||||
```bash
|
||||
cd /opt/backup-dashboard/backend
|
||||
sudo -u backupdash /opt/backup-dashboard/venv/bin/python manage.py create-key veeam-acme
|
||||
```
|
||||
It prints the database path it's using and won't create a new database unless you pass `--create`.
|
||||
|
||||
Back up the SQLite file with `sqlite3 backup-dashboard.db ".backup copy.db"`. Don't use a plain `cp`
|
||||
while the service is running.
|
||||
|
||||
### Updating the server
|
||||
|
||||
After changes are merged into `main`, on the dashboard server:
|
||||
|
||||
1. **Back up the database.** It takes seconds, and an update may upgrade the database's structure
|
||||
when the API restarts, which only this backup can reverse. Needs the `sqlite3` package:
|
||||
```bash
|
||||
sudo -u backupdash sqlite3 /var/lib/backup-dashboard/backup-dashboard.db ".backup /var/lib/backup-dashboard/before-update.db"
|
||||
```
|
||||
2. **Pull, and see what changed** (the list tells you whether steps 7 and 8 apply):
|
||||
```bash
|
||||
sudo git -C /opt/backup-dashboard pull
|
||||
```
|
||||
```bash
|
||||
git -C /opt/backup-dashboard diff --stat ORIG_HEAD HEAD
|
||||
```
|
||||
3. **Python packages.** Does nothing unless `backend/requirements.txt` changed:
|
||||
```bash
|
||||
sudo /opt/backup-dashboard/venv/bin/pip install -q -r /opt/backup-dashboard/backend/requirements.txt
|
||||
```
|
||||
4. **Web files.** nginx serves its own copy in `/var/www`, so a pull alone doesn't change what
|
||||
browsers get. If this step is skipped, new pages quietly show the dashboard instead (nginx
|
||||
falls back to `index.html` for files it doesn't have):
|
||||
```bash
|
||||
sudo rsync -a /opt/backup-dashboard/public/ /var/www/backup-dashboard/public/
|
||||
```
|
||||
Browsers pick up the new files on their next page load.
|
||||
5. **Restart the API**, which loads the new code and settings and upgrades the database if needed:
|
||||
```bash
|
||||
sudo systemctl restart backup-dashboard-api
|
||||
```
|
||||
6. **Check it's healthy.** The first should print `{"status":"ok"}`; the log shouldn't show errors:
|
||||
```bash
|
||||
curl -s http://127.0.0.1:8000/api/v1/health
|
||||
```
|
||||
```bash
|
||||
journalctl -u backup-dashboard-api -n 20 --no-pager
|
||||
```
|
||||
7. **If files in `deploy/` changed:** those are templates whose installed copies live elsewhere,
|
||||
so a pull doesn't update them. Compare each changed one with its installed copy (e.g.
|
||||
`diff /opt/backup-dashboard/deploy/oauth2-proxy.service /etc/systemd/system/oauth2-proxy.service`)
|
||||
and bring the changes across:
|
||||
|
||||
| Changed in the repo | Installed copy | After updating it |
|
||||
|---|---|---|
|
||||
| `deploy/backup-dashboard.env` | `/etc/backup-dashboard/backup-dashboard.env` | Add any new settings, keeping your values; restart the API |
|
||||
| `deploy/backup-dashboard-api.service` | `/etc/systemd/system/backup-dashboard-api.service` | `sudo systemctl daemon-reload`, restart the API |
|
||||
| `deploy/nginx.conf` | Your nginx site config | Keep your server name and certificate paths; `sudo nginx -t && sudo systemctl reload nginx` |
|
||||
| `deploy/oauth2-proxy.cfg` | `/etc/oauth2-proxy/backup-dashboard.cfg` | Keep your IDs and secrets; `sudo systemctl restart oauth2-proxy` |
|
||||
| `deploy/oauth2-proxy.service` | `/etc/systemd/system/oauth2-proxy.service` | `sudo systemctl daemon-reload && sudo systemctl restart oauth2-proxy` |
|
||||
|
||||
8. **If files in `collectors/` changed:** the backup servers keep running their installed copy.
|
||||
On each affected server, copy over the whole `collectors` folder, re-run that collector's
|
||||
installer, and do a dry run (see its README). The changes are already covered by the
|
||||
collector tests (`./collectors/tests/Invoke-CollectorTests.ps1`) if you want to run them first.
|
||||
|
||||
**If something goes wrong**, go back to the version from before step 2 (`ORIG_HEAD`). The order
|
||||
matters: an older API won't start on a database a newer one has upgraded, so restore the database
|
||||
before starting it.
|
||||
|
||||
1. Stop the API:
|
||||
```bash
|
||||
sudo systemctl stop backup-dashboard-api
|
||||
```
|
||||
2. Check whether the update upgraded the database: if these two numbers differ, it did.
|
||||
```bash
|
||||
for db in backup-dashboard before-update; do sudo -u backupdash sqlite3 /var/lib/backup-dashboard/$db.db "PRAGMA user_version"; done
|
||||
```
|
||||
If so, restore the backup from step 1 (`.restore`, not `cp`: it also deals with SQLite's `-wal`
|
||||
file). Runs reported since the backup drop out of the history, since collectors have already
|
||||
marked them as sent; backups themselves are unaffected.
|
||||
```bash
|
||||
sudo -u backupdash sqlite3 /var/lib/backup-dashboard/backup-dashboard.db ".restore /var/lib/backup-dashboard/before-update.db"
|
||||
```
|
||||
3. Go back to the previous code and web files, then start the API:
|
||||
```bash
|
||||
sudo git -C /opt/backup-dashboard checkout ORIG_HEAD
|
||||
```
|
||||
```bash
|
||||
sudo rsync -a /opt/backup-dashboard/public/ /var/www/backup-dashboard/public/ && sudo systemctl start backup-dashboard-api
|
||||
```
|
||||
|
||||
Once the problem is fixed, return to `main` with `sudo git -C /opt/backup-dashboard checkout main`
|
||||
and update as above.
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.venv/
|
||||
venv/
|
||||
.pytest_cache/
|
||||
*.db
|
||||
*.db-wal
|
||||
*.db-shm
|
||||
@@ -0,0 +1,44 @@
|
||||
"""Admin API: managing collector API keys. Only for users in BACKUP_DASH_ADMINS."""
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, StringConstraints
|
||||
|
||||
from . import db, keys
|
||||
from .auth import require_admin, require_page_request
|
||||
|
||||
log = logging.getLogger("backup_dashboard")
|
||||
|
||||
router = APIRouter(prefix="/api/v1/admin", dependencies=[Depends(require_admin)])
|
||||
|
||||
|
||||
class NewKey(BaseModel):
|
||||
name: Annotated[str, StringConstraints(strip_whitespace=True, pattern=f"^{keys.NAME_PATTERN}$")]
|
||||
|
||||
|
||||
@router.get("/keys")
|
||||
def list_keys(conn: sqlite3.Connection = Depends(db.get_db)):
|
||||
return {"keys": keys.list_keys(conn)}
|
||||
|
||||
|
||||
@router.post("/keys", status_code=201, dependencies=[Depends(require_page_request)])
|
||||
def create_key(body: NewKey, admin: str = Depends(require_admin), conn: sqlite3.Connection = Depends(db.get_db)):
|
||||
try:
|
||||
record, secret = keys.create_key(conn, body.name, created_by=admin)
|
||||
except keys.KeyNameTaken:
|
||||
raise HTTPException(409, f"A key named {body.name!r} already exists (revoked keys keep their names)") from None
|
||||
log.info("admin %s created API key %r", admin, body.name)
|
||||
# The key itself is returned this once; only its hash is stored
|
||||
return {"key": record, "secret": secret}
|
||||
|
||||
|
||||
@router.post("/keys/{key_id}/revoke", dependencies=[Depends(require_page_request)])
|
||||
def revoke_key(key_id: int, admin: str = Depends(require_admin), conn: sqlite3.Connection = Depends(db.get_db)):
|
||||
record = keys.revoke_key(conn, key_id, revoked_by=admin)
|
||||
if record is None:
|
||||
raise HTTPException(404, "Key not found")
|
||||
log.info("admin %s revoked API key %r", admin, record["name"])
|
||||
return {"key": record}
|
||||
@@ -0,0 +1,73 @@
|
||||
"""Authentication and authorisation.
|
||||
|
||||
Collectors (machines) authenticate with API keys, checked here. Dashboard *users* sign in with
|
||||
Microsoft 365 in nginx (via oauth2-proxy), which then tells the app who they are in
|
||||
X-Forwarded-* headers it sets itself (a browser can't supply them). The app only decides whether
|
||||
that user is an admin.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import secrets
|
||||
import sqlite3
|
||||
|
||||
from fastapi import Depends, Header, HTTPException, Request
|
||||
|
||||
from .db import get_db
|
||||
from .timeutil import to_db, utcnow
|
||||
|
||||
KEY_PREFIX = "bdk_"
|
||||
|
||||
|
||||
def generate_key() -> str:
|
||||
return KEY_PREFIX + secrets.token_urlsafe(32)
|
||||
|
||||
|
||||
def hash_key(key: str) -> str:
|
||||
# Keys are 256-bit random, so a fast unsalted hash is fine (no dictionary to attack).
|
||||
return hashlib.sha256(key.encode()).hexdigest()
|
||||
|
||||
|
||||
def current_user(request: Request) -> str | None:
|
||||
"""The signed-in user's sign-in name, lowercase; None without Microsoft sign-in (e.g. dev)."""
|
||||
user = request.headers.get("x-forwarded-email") or request.headers.get("x-forwarded-preferred-username")
|
||||
return user.strip().lower() if user and user.strip() else None
|
||||
|
||||
|
||||
def is_admin(request: Request) -> bool:
|
||||
user = current_user(request)
|
||||
return user is not None and user in request.app.state.settings.admins
|
||||
|
||||
|
||||
def require_admin(request: Request) -> str:
|
||||
if not is_admin(request):
|
||||
raise HTTPException(403, "Admins only. Admins are listed in BACKUP_DASH_ADMINS on the server.")
|
||||
return current_user(request)
|
||||
|
||||
|
||||
# Changes made from the admin page must carry this header. A page on another site can't add a
|
||||
# custom header to a request to the dashboard (the browser would first ask the API, which allows
|
||||
# no other origins), so a signed-in admin's browser can't be used to forge them.
|
||||
PAGE_HEADER = "X-Requested-By"
|
||||
PAGE_HEADER_VALUE = "backup-dashboard"
|
||||
|
||||
|
||||
def require_page_request(x_requested_by: str | None = Header(None)) -> None:
|
||||
if x_requested_by != PAGE_HEADER_VALUE:
|
||||
raise HTTPException(403, f"Missing {PAGE_HEADER} header")
|
||||
|
||||
|
||||
def require_ingest_key(
|
||||
authorization: str | None = Header(None),
|
||||
conn: sqlite3.Connection = Depends(get_db),
|
||||
) -> str:
|
||||
scheme, _, token = (authorization or "").partition(" ")
|
||||
token = token.strip()
|
||||
if scheme.lower() != "bearer" or not token:
|
||||
raise HTTPException(401, "Missing bearer token", headers={"WWW-Authenticate": "Bearer"})
|
||||
row = conn.execute(
|
||||
"SELECT id, name FROM api_keys WHERE key_hash = ? AND revoked_at IS NULL", (hash_key(token),)
|
||||
).fetchone()
|
||||
if row is None:
|
||||
raise HTTPException(401, "Invalid or revoked API key", headers={"WWW-Authenticate": "Bearer"})
|
||||
conn.execute("UPDATE api_keys SET last_used_at = ? WHERE id = ?", (to_db(utcnow()), row["id"]))
|
||||
return row["name"]
|
||||
@@ -0,0 +1,114 @@
|
||||
import sqlite3
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import Request
|
||||
|
||||
SCHEMA_VERSION = 2
|
||||
|
||||
SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS tenants (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE COLLATE NOCASE
|
||||
);
|
||||
|
||||
-- A job is identified by (tenant, source system, the source's own job id), so two
|
||||
-- products can't collide and renaming a job in Veeam doesn't create a duplicate.
|
||||
CREATE TABLE IF NOT EXISTS jobs (
|
||||
id INTEGER PRIMARY KEY,
|
||||
tenant_id INTEGER NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
|
||||
source_system TEXT NOT NULL,
|
||||
external_id TEXT NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
next_run_at TEXT,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
UNIQUE (tenant_id, source_system, external_id)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS runs (
|
||||
id INTEGER PRIMARY KEY,
|
||||
job_id INTEGER NOT NULL REFERENCES jobs(id) ON DELETE CASCADE,
|
||||
external_id TEXT NOT NULL,
|
||||
status TEXT NOT NULL CHECK (status IN ('running', 'success', 'warning', 'failed')),
|
||||
started_at TEXT NOT NULL,
|
||||
finished_at TEXT,
|
||||
message TEXT NOT NULL DEFAULT '',
|
||||
size_bytes INTEGER,
|
||||
details TEXT NOT NULL DEFAULT '{}',
|
||||
received_at TEXT NOT NULL,
|
||||
CHECK ((status = 'running') = (finished_at IS NULL)),
|
||||
UNIQUE (job_id, external_id)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS runs_job_started ON runs (job_id, started_at);
|
||||
CREATE INDEX IF NOT EXISTS runs_job_finished ON runs (job_id, finished_at);
|
||||
|
||||
-- Keys used by collectors to push results. Only the SHA-256 is stored.
|
||||
CREATE TABLE IF NOT EXISTS api_keys (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
key_hash TEXT NOT NULL UNIQUE,
|
||||
created_at TEXT NOT NULL,
|
||||
last_used_at TEXT,
|
||||
revoked_at TEXT
|
||||
);
|
||||
"""
|
||||
|
||||
|
||||
def connect(path: Path | str) -> sqlite3.Connection:
|
||||
# Autocommit mode; writes that must be atomic go through transaction().
|
||||
# check_same_thread=False: FastAPI may run a dependency and its endpoint on different
|
||||
# threadpool threads, but a connection is only ever used by one request at a time.
|
||||
conn = sqlite3.connect(path, timeout=10, isolation_level=None, check_same_thread=False)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("PRAGMA foreign_keys = ON")
|
||||
conn.execute("PRAGMA busy_timeout = 10000")
|
||||
return conn
|
||||
|
||||
|
||||
# Changes since SCHEMA (which is v1), applied in order to bring any database up to SCHEMA_VERSION
|
||||
MIGRATIONS = {
|
||||
2: [
|
||||
# Who created/revoked a key from the admin page, and which company it last reported for
|
||||
"ALTER TABLE api_keys ADD COLUMN created_by TEXT",
|
||||
"ALTER TABLE api_keys ADD COLUMN revoked_by TEXT",
|
||||
"ALTER TABLE api_keys ADD COLUMN last_used_tenant TEXT",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def init_db(path: Path | str) -> None:
|
||||
conn = connect(path)
|
||||
try:
|
||||
conn.execute("PRAGMA journal_mode = WAL")
|
||||
version = conn.execute("PRAGMA user_version").fetchone()[0]
|
||||
if version > SCHEMA_VERSION:
|
||||
raise RuntimeError(f"Database schema v{version} is newer than this app (v{SCHEMA_VERSION})")
|
||||
conn.executescript(SCHEMA) # v1 tables, for a new database
|
||||
with transaction(conn):
|
||||
for target in range(max(version, 1) + 1, SCHEMA_VERSION + 1):
|
||||
for statement in MIGRATIONS[target]:
|
||||
conn.execute(statement)
|
||||
conn.execute(f"PRAGMA user_version = {SCHEMA_VERSION}")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def transaction(conn: sqlite3.Connection) -> Iterator[None]:
|
||||
conn.execute("BEGIN IMMEDIATE")
|
||||
try:
|
||||
yield
|
||||
except BaseException:
|
||||
conn.execute("ROLLBACK")
|
||||
raise
|
||||
conn.execute("COMMIT")
|
||||
|
||||
|
||||
def get_db(request: Request) -> Iterator[sqlite3.Connection]:
|
||||
conn = connect(request.app.state.settings.db_path)
|
||||
try:
|
||||
yield conn
|
||||
finally:
|
||||
conn.close()
|
||||
@@ -0,0 +1,55 @@
|
||||
"""Collector API keys: create, list, revoke. Used by the admin API and manage.py."""
|
||||
|
||||
import re
|
||||
import sqlite3
|
||||
from typing import Any
|
||||
|
||||
from .auth import generate_key, hash_key
|
||||
from .timeutil import to_db, utcnow
|
||||
|
||||
# Letters, digits, '.', '_', '-'; starts with a letter or digit. E.g. "veeam-acme", "mars.fs01"
|
||||
NAME_PATTERN = r"[A-Za-z0-9][A-Za-z0-9._-]{0,63}"
|
||||
|
||||
_FIELDS = "id, name, created_at, created_by, last_used_at, last_used_tenant, revoked_at, revoked_by"
|
||||
|
||||
|
||||
class KeyNameTaken(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def valid_name(name: str) -> bool:
|
||||
return re.fullmatch(NAME_PATTERN, name) is not None
|
||||
|
||||
|
||||
def list_keys(conn: sqlite3.Connection) -> list[dict[str, Any]]:
|
||||
# Active keys first, then by name
|
||||
rows = conn.execute(f"SELECT {_FIELDS} FROM api_keys ORDER BY revoked_at IS NOT NULL, name").fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
|
||||
def get_key(conn: sqlite3.Connection, key_id: int) -> dict[str, Any] | None:
|
||||
row = conn.execute(f"SELECT {_FIELDS} FROM api_keys WHERE id = ?", (key_id,)).fetchone()
|
||||
return dict(row) if row else None
|
||||
|
||||
|
||||
def create_key(conn: sqlite3.Connection, name: str, created_by: str) -> tuple[dict[str, Any], str]:
|
||||
"""Returns the stored key's record and the key itself, which isn't stored and can't be shown again."""
|
||||
secret = generate_key()
|
||||
try:
|
||||
key_id = conn.execute(
|
||||
"INSERT INTO api_keys (name, key_hash, created_at, created_by) VALUES (?, ?, ?, ?) RETURNING id",
|
||||
(name, hash_key(secret), to_db(utcnow()), created_by),
|
||||
).fetchone()[0]
|
||||
except sqlite3.IntegrityError:
|
||||
# Names stay taken after revocation, so the audit trail stays unambiguous
|
||||
raise KeyNameTaken(name) from None
|
||||
return get_key(conn, key_id), secret
|
||||
|
||||
|
||||
def revoke_key(conn: sqlite3.Connection, key_id: int, revoked_by: str) -> dict[str, Any] | None:
|
||||
"""Revokes the key (no-op if already revoked). Returns its record, or None if there's no such key."""
|
||||
conn.execute(
|
||||
"UPDATE api_keys SET revoked_at = ?, revoked_by = ? WHERE id = ? AND revoked_at IS NULL",
|
||||
(to_db(utcnow()), revoked_by, key_id),
|
||||
)
|
||||
return get_key(conn, key_id)
|
||||
@@ -0,0 +1,133 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import sqlite3
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, Depends, FastAPI, HTTPException, Query, Request
|
||||
|
||||
from . import admin, db, reports, repo
|
||||
from .auth import is_admin, require_ingest_key
|
||||
from .models import IngestIn
|
||||
from .settings import HISTORY_DAYS, Settings, load_settings
|
||||
from .timeutil import to_db, utcnow
|
||||
|
||||
log = logging.getLogger("backup_dashboard")
|
||||
PURGE_INTERVAL_SECONDS = 6 * 3600
|
||||
|
||||
router = APIRouter(prefix="/api/v1")
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
def health(conn: sqlite3.Connection = Depends(db.get_db)):
|
||||
conn.execute("SELECT 1")
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@router.get("/me")
|
||||
def me(request: Request):
|
||||
# Set by nginx from oauth2-proxy's sign-in check (nginx overwrites anything the browser sends).
|
||||
# Display only: access control happens in nginx. Empty in dev or without Microsoft sign-in.
|
||||
return {
|
||||
"email": request.headers.get("x-forwarded-email") or None,
|
||||
"name": request.headers.get("x-forwarded-preferred-username") or request.headers.get("x-forwarded-user") or None,
|
||||
"is_admin": is_admin(request),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/jobs")
|
||||
def list_jobs(conn: sqlite3.Connection = Depends(db.get_db)):
|
||||
return {"generated_at": to_db(utcnow()), "jobs": repo.list_jobs(conn)}
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}")
|
||||
def get_job(job_id: int, conn: sqlite3.Connection = Depends(db.get_db)):
|
||||
job = repo.get_job(conn, job_id)
|
||||
if job is None:
|
||||
raise HTTPException(404, "Job not found")
|
||||
return job
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}/runs")
|
||||
def list_runs(
|
||||
job_id: int,
|
||||
days: int = Query(HISTORY_DAYS, ge=1, le=HISTORY_DAYS),
|
||||
conn: sqlite3.Connection = Depends(db.get_db),
|
||||
):
|
||||
if conn.execute("SELECT 1 FROM jobs WHERE id = ?", (job_id,)).fetchone() is None:
|
||||
raise HTTPException(404, "Job not found")
|
||||
since = utcnow() - timedelta(days=days)
|
||||
return {"job_id": job_id, "since": to_db(since), "runs": repo.list_runs(conn, job_id, since)}
|
||||
|
||||
|
||||
@router.get("/runs/{run_id}")
|
||||
def get_run(run_id: int, conn: sqlite3.Connection = Depends(db.get_db)):
|
||||
run = repo.get_run(conn, run_id)
|
||||
if run is None:
|
||||
raise HTTPException(404, "Run not found")
|
||||
return run
|
||||
|
||||
|
||||
@router.post("/ingest")
|
||||
def ingest(
|
||||
item: IngestIn,
|
||||
key_name: str = Depends(require_ingest_key),
|
||||
conn: sqlite3.Connection = Depends(db.get_db),
|
||||
):
|
||||
job_id, run_id = repo.ingest(conn, item)
|
||||
# Shown on the admin page, so you can tell what each key is used for
|
||||
conn.execute("UPDATE api_keys SET last_used_tenant = ? WHERE name = ?", (item.tenant, key_name))
|
||||
log.info("ingest key=%s tenant=%r job=%r run=%s", key_name, item.tenant, item.job.name, run_id)
|
||||
return {"job_id": job_id, "run_id": run_id}
|
||||
|
||||
|
||||
async def _purge_loop(settings: Settings) -> None:
|
||||
def purge() -> int:
|
||||
conn = db.connect(settings.db_path)
|
||||
try:
|
||||
return repo.purge_old_runs(conn, settings.retention_days)
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
while True:
|
||||
try:
|
||||
if deleted := await asyncio.to_thread(purge):
|
||||
log.info("purged %d runs older than %d days", deleted, settings.retention_days)
|
||||
except Exception:
|
||||
log.exception("run purge failed")
|
||||
await asyncio.sleep(PURGE_INTERVAL_SECONDS)
|
||||
|
||||
|
||||
def create_app(settings: Settings | None = None) -> FastAPI:
|
||||
settings = settings or load_settings()
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
db.init_db(settings.db_path)
|
||||
task = asyncio.create_task(_purge_loop(settings))
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
task.cancel()
|
||||
|
||||
# Interactive docs are off: nginx's strict CSP would block their CDN assets anyway.
|
||||
app = FastAPI(
|
||||
title="Backup Dashboard API",
|
||||
lifespan=lifespan,
|
||||
docs_url=None,
|
||||
redoc_url=None,
|
||||
openapi_url="/api/v1/openapi.json",
|
||||
)
|
||||
app.state.settings = settings
|
||||
app.include_router(router)
|
||||
app.include_router(admin.router)
|
||||
app.include_router(reports.router)
|
||||
|
||||
if settings.serve_static:
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
|
||||
public = Path(__file__).resolve().parents[2] / "public"
|
||||
app.mount("/", StaticFiles(directory=public, html=True), name="static")
|
||||
|
||||
return app
|
||||
@@ -0,0 +1,63 @@
|
||||
import json
|
||||
from typing import Annotated, Any, Literal
|
||||
|
||||
from pydantic import AwareDatetime, BaseModel, ConfigDict, Field, StringConstraints, field_validator, model_validator
|
||||
|
||||
RunStatus = Literal["running", "success", "warning", "failed"]
|
||||
Name = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=200)]
|
||||
|
||||
MAX_MESSAGE_CHARS = 32_000
|
||||
MAX_DETAILS_BYTES = 16_000
|
||||
|
||||
|
||||
class Strict(BaseModel):
|
||||
# Reject unknown fields so a typo in a collector script fails loudly instead of being dropped.
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
|
||||
class JobIn(Strict):
|
||||
external_id: Name = Field(description="The source system's own stable ID for the job")
|
||||
name: Name
|
||||
# Omit to leave the stored value untouched; send null to clear it.
|
||||
next_run_at: AwareDatetime | None = None
|
||||
|
||||
|
||||
class RunIn(Strict):
|
||||
external_id: Name | None = Field(
|
||||
None, description="The source's ID for this run. Defaults to started_at, so a 'running' "
|
||||
"report and its later completion report merge into one run.")
|
||||
status: RunStatus
|
||||
started_at: AwareDatetime
|
||||
# Omit for a finished run to use the time the report was received.
|
||||
finished_at: AwareDatetime | None = None
|
||||
message: str = ""
|
||||
size_bytes: int | None = Field(None, ge=0)
|
||||
details: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
@field_validator("message")
|
||||
@classmethod
|
||||
def truncate_message(cls, v: str) -> str:
|
||||
# Backup logs can be huge; keep the head, which is where the summary usually is.
|
||||
return v if len(v) <= MAX_MESSAGE_CHARS else v[:MAX_MESSAGE_CHARS] + "\n… [truncated]"
|
||||
|
||||
@field_validator("details")
|
||||
@classmethod
|
||||
def limit_details(cls, v: dict[str, Any]) -> dict[str, Any]:
|
||||
if len(json.dumps(v)) > MAX_DETAILS_BYTES:
|
||||
raise ValueError(f"details must serialize to at most {MAX_DETAILS_BYTES} bytes")
|
||||
return v
|
||||
|
||||
@model_validator(mode="after")
|
||||
def check_times(self) -> "RunIn":
|
||||
if self.status == "running" and self.finished_at is not None:
|
||||
raise ValueError("a running run cannot have finished_at")
|
||||
if self.finished_at is not None and self.finished_at < self.started_at:
|
||||
raise ValueError("finished_at is before started_at")
|
||||
return self
|
||||
|
||||
|
||||
class IngestIn(Strict):
|
||||
tenant: Name = Field(description="Company name as shown on the dashboard")
|
||||
source_system: Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=100)]
|
||||
job: JobIn
|
||||
run: RunIn | None = Field(None, description="Omit to only register/update the job (e.g. next_run_at)")
|
||||
@@ -0,0 +1,169 @@
|
||||
import json
|
||||
import sqlite3
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from .db import transaction
|
||||
from .models import IngestIn
|
||||
from .timeutil import to_db, utcnow
|
||||
|
||||
# One row per job: `cur` is the most recent run (may be in progress), `fin` the most recent
|
||||
# finished one. "Last run" on the dashboard is fin.finished_at, so a job that's currently
|
||||
# running still shows when its previous run completed.
|
||||
_JOB_SQL = """
|
||||
SELECT j.id, t.name AS company_name, j.name AS job_name, j.source_system, j.next_run_at,
|
||||
cur.status,
|
||||
CASE WHEN cur.status = 'running' THEN cur.started_at END AS running_since,
|
||||
fin.status AS last_status,
|
||||
fin.started_at AS last_started_at,
|
||||
fin.finished_at AS last_run_at,
|
||||
unixepoch(fin.finished_at) - unixepoch(fin.started_at) AS duration_seconds,
|
||||
fin.size_bytes,
|
||||
fin.details,
|
||||
COALESCE(fin.message, cur.message, '') AS message
|
||||
FROM jobs j
|
||||
JOIN tenants t ON t.id = j.tenant_id
|
||||
LEFT JOIN runs cur ON cur.id = (
|
||||
SELECT r.id FROM runs r WHERE r.job_id = j.id
|
||||
ORDER BY r.started_at DESC, r.id DESC LIMIT 1)
|
||||
LEFT JOIN runs fin ON fin.id = (
|
||||
SELECT r.id FROM runs r WHERE r.job_id = j.id AND r.finished_at IS NOT NULL
|
||||
ORDER BY r.finished_at DESC, r.id DESC LIMIT 1)
|
||||
"""
|
||||
|
||||
_SUMMARY_FIELDS = (
|
||||
"id", "company_name", "job_name", "source_system", "status",
|
||||
"running_since", "last_run_at", "next_run_at", "message",
|
||||
)
|
||||
|
||||
|
||||
def list_jobs(conn: sqlite3.Connection) -> list[dict[str, Any]]:
|
||||
rows = conn.execute(_JOB_SQL + " ORDER BY t.name, j.name").fetchall()
|
||||
return [{k: row[k] for k in _SUMMARY_FIELDS} for row in rows]
|
||||
|
||||
|
||||
def list_tenant_jobs(conn: sqlite3.Connection, tenant_id: int) -> list[dict[str, Any]]:
|
||||
"""A company's jobs with their current status, for reports."""
|
||||
rows = conn.execute(_JOB_SQL + " WHERE j.tenant_id = ? ORDER BY j.name", (tenant_id,)).fetchall()
|
||||
fields = (*_SUMMARY_FIELDS, "last_status")
|
||||
return [{k: row[k] for k in fields} for row in rows]
|
||||
|
||||
|
||||
def get_job(conn: sqlite3.Connection, job_id: int) -> dict[str, Any] | None:
|
||||
row = conn.execute(_JOB_SQL + " WHERE j.id = ?", (job_id,)).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
job = dict(row)
|
||||
job["details"] = json.loads(job["details"]) if job["details"] else {}
|
||||
return job
|
||||
|
||||
|
||||
def list_runs(conn: sqlite3.Connection, job_id: int, since: datetime) -> list[dict[str, Any]]:
|
||||
# Messages/details are left out: an hourly job has ~720 runs a month and logs can be large.
|
||||
# The UI fetches a single run's full record on demand.
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT id, status, started_at, finished_at,
|
||||
unixepoch(finished_at) - unixepoch(started_at) AS duration_seconds, size_bytes
|
||||
FROM runs WHERE job_id = ? AND started_at >= ?
|
||||
ORDER BY started_at DESC, id DESC
|
||||
""",
|
||||
(job_id, to_db(since)),
|
||||
).fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
|
||||
def get_run(conn: sqlite3.Connection, run_id: int) -> dict[str, Any] | None:
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT id, job_id, status, started_at, finished_at,
|
||||
unixepoch(finished_at) - unixepoch(started_at) AS duration_seconds,
|
||||
size_bytes, message, details
|
||||
FROM runs WHERE id = ?
|
||||
""",
|
||||
(run_id,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
run = dict(row)
|
||||
run["details"] = json.loads(run["details"])
|
||||
return run
|
||||
|
||||
|
||||
def ingest(conn: sqlite3.Connection, item: IngestIn) -> tuple[int, int | None]:
|
||||
"""Upsert tenant → job → run. Safe to call repeatedly with the same payload."""
|
||||
now = utcnow()
|
||||
now_s = to_db(now)
|
||||
with transaction(conn):
|
||||
conn.execute("INSERT INTO tenants (name) VALUES (?) ON CONFLICT (name) DO NOTHING", (item.tenant,))
|
||||
tenant_id = conn.execute("SELECT id FROM tenants WHERE name = ?", (item.tenant,)).fetchone()[0]
|
||||
|
||||
job_id = conn.execute(
|
||||
"""
|
||||
INSERT INTO jobs (tenant_id, source_system, external_id, name, next_run_at, created_at, updated_at)
|
||||
VALUES (:tenant_id, :source, :ext, :name, :next_run, :now, :now)
|
||||
ON CONFLICT (tenant_id, source_system, external_id) DO UPDATE SET
|
||||
name = excluded.name,
|
||||
next_run_at = CASE WHEN :set_next_run THEN excluded.next_run_at ELSE jobs.next_run_at END,
|
||||
updated_at = excluded.updated_at
|
||||
RETURNING id
|
||||
""",
|
||||
{
|
||||
"tenant_id": tenant_id,
|
||||
"source": item.source_system,
|
||||
"ext": item.job.external_id,
|
||||
"name": item.job.name,
|
||||
"next_run": to_db(item.job.next_run_at) if item.job.next_run_at else None,
|
||||
"set_next_run": "next_run_at" in item.job.model_fields_set,
|
||||
"now": now_s,
|
||||
},
|
||||
).fetchone()[0]
|
||||
|
||||
if item.run is None:
|
||||
return job_id, None
|
||||
|
||||
run = item.run
|
||||
started = to_db(run.started_at)
|
||||
finished = None
|
||||
if run.status != "running":
|
||||
finished = to_db(run.finished_at or max(now, run.started_at))
|
||||
params = {
|
||||
"job_id": job_id,
|
||||
"ext": run.external_id or started,
|
||||
"status": run.status,
|
||||
"started": started,
|
||||
"finished": finished,
|
||||
"message": run.message,
|
||||
"size": run.size_bytes,
|
||||
"details": json.dumps(run.details),
|
||||
"now": now_s,
|
||||
}
|
||||
# The WHERE stops a late/retried "running" report from reopening a run that already finished.
|
||||
row = conn.execute(
|
||||
"""
|
||||
INSERT INTO runs (job_id, external_id, status, started_at, finished_at, message,
|
||||
size_bytes, details, received_at)
|
||||
VALUES (:job_id, :ext, :status, :started, :finished, :message, :size, :details, :now)
|
||||
ON CONFLICT (job_id, external_id) DO UPDATE SET
|
||||
status = excluded.status,
|
||||
started_at = excluded.started_at,
|
||||
finished_at = excluded.finished_at,
|
||||
message = excluded.message,
|
||||
size_bytes = excluded.size_bytes,
|
||||
details = excluded.details,
|
||||
received_at = excluded.received_at
|
||||
WHERE NOT (runs.status != 'running' AND excluded.status = 'running')
|
||||
RETURNING id
|
||||
""",
|
||||
params,
|
||||
).fetchone()
|
||||
if row is None:
|
||||
row = conn.execute(
|
||||
"SELECT id FROM runs WHERE job_id = :job_id AND external_id = :ext", params
|
||||
).fetchone()
|
||||
return job_id, row[0]
|
||||
|
||||
|
||||
def purge_old_runs(conn: sqlite3.Connection, retention_days: int) -> int:
|
||||
cutoff = to_db(utcnow() - timedelta(days=retention_days))
|
||||
return conn.execute("DELETE FROM runs WHERE started_at < ?", (cutoff,)).rowcount
|
||||
@@ -0,0 +1,143 @@
|
||||
"""Per-company reports: every run of one company's backup jobs over a period, as JSON (for the
|
||||
printable report page) or CSV. Read-only, like the dashboard, so any signed-in user may view them."""
|
||||
|
||||
import csv
|
||||
import io
|
||||
import re
|
||||
import sqlite3
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, Response
|
||||
|
||||
from . import db, repo
|
||||
from .timeutil import DB_FORMAT, to_db, utcnow
|
||||
|
||||
router = APIRouter(prefix="/api/v1")
|
||||
|
||||
CSV_COLUMNS = ("Company", "Job", "Source", "Started", "Finished", "Duration (seconds)", "Status",
|
||||
"Size (bytes)", "Message")
|
||||
|
||||
|
||||
@router.get("/tenants")
|
||||
def list_tenants(conn: sqlite3.Connection = Depends(db.get_db)):
|
||||
"""Companies that have jobs, for choosing a report."""
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT t.name, COUNT(j.id) AS jobs FROM tenants t JOIN jobs j ON j.tenant_id = t.id
|
||||
GROUP BY t.id ORDER BY t.name COLLATE NOCASE
|
||||
"""
|
||||
).fetchall()
|
||||
return {"tenants": [dict(r) for r in rows]}
|
||||
|
||||
|
||||
def _load_report(request: Request, conn: sqlite3.Connection, tenant: str, days: int,
|
||||
all_messages: bool = False) -> dict[str, Any]:
|
||||
retention = request.app.state.settings.retention_days
|
||||
if days > retention:
|
||||
raise HTTPException(422, f"Reports can cover at most {retention} days: older runs aren't kept")
|
||||
row = conn.execute("SELECT id, name FROM tenants WHERE name = ?", (tenant.strip(),)).fetchone() # NOCASE
|
||||
if row is None:
|
||||
raise HTTPException(404, f"No company named {tenant!r}")
|
||||
|
||||
now = utcnow()
|
||||
since = now - timedelta(days=days)
|
||||
jobs = repo.list_tenant_jobs(conn, row["id"])
|
||||
runs_by_job: dict[int, list[dict[str, Any]]] = {job["id"]: [] for job in jobs}
|
||||
# By default only warnings' and failures' messages: successes all say much the same, and an
|
||||
# hourly job has thousands of runs in 90 days
|
||||
for run in conn.execute(
|
||||
"""
|
||||
SELECT r.job_id, r.status, r.started_at, r.finished_at,
|
||||
unixepoch(r.finished_at) - unixepoch(r.started_at) AS duration_seconds, r.size_bytes,
|
||||
CASE WHEN ? OR r.status IN ('warning', 'failed') THEN r.message END AS message
|
||||
FROM runs r JOIN jobs j ON j.id = r.job_id
|
||||
WHERE j.tenant_id = ? AND r.started_at >= ?
|
||||
ORDER BY r.started_at, r.id
|
||||
""",
|
||||
(all_messages, row["id"], to_db(since)),
|
||||
):
|
||||
run = dict(run)
|
||||
runs_by_job[run.pop("job_id")].append(run)
|
||||
# When the dashboard first knew of each job: its earliest run (collectors backfill history), or
|
||||
# when it was first reported if earlier. Days before then are "not monitored", not "no backup".
|
||||
tracked = dict(conn.execute(
|
||||
"""
|
||||
SELECT j.id, min(j.created_at, coalesce((SELECT MIN(r.started_at) FROM runs r WHERE r.job_id = j.id), j.created_at))
|
||||
FROM jobs j WHERE j.tenant_id = ?
|
||||
""",
|
||||
(row["id"],),
|
||||
).fetchall())
|
||||
for job in jobs:
|
||||
job["runs"] = runs_by_job[job["id"]]
|
||||
job["tracked_since"] = tracked[job["id"]]
|
||||
return {
|
||||
"tenant": row["name"],
|
||||
"days": days,
|
||||
"since": to_db(since),
|
||||
"until": to_db(now),
|
||||
"generated_at": to_db(now),
|
||||
"jobs": jobs,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/reports/tenant")
|
||||
def tenant_report(
|
||||
request: Request,
|
||||
tenant: str = Query(..., min_length=1),
|
||||
days: int = Query(30, ge=1),
|
||||
conn: sqlite3.Connection = Depends(db.get_db),
|
||||
):
|
||||
return _load_report(request, conn, tenant, days)
|
||||
|
||||
|
||||
def _safe_cell(value: Any) -> Any:
|
||||
"""Defuse text that a spreadsheet would run as a formula. Job names and messages come from the
|
||||
backup servers, so a value like '=HYPERLINK(...)' must stay text."""
|
||||
if isinstance(value, str) and value[:1] in ("=", "+", "-", "@", "\t", "\r"):
|
||||
return "'" + value
|
||||
return value
|
||||
|
||||
|
||||
@router.get("/reports/tenant.csv")
|
||||
def tenant_report_csv(
|
||||
request: Request,
|
||||
tenant: str = Query(..., min_length=1),
|
||||
days: int = Query(30, ge=1),
|
||||
tz: str = Query("UTC", description="IANA time zone for the times, e.g. America/Edmonton"),
|
||||
conn: sqlite3.Connection = Depends(db.get_db),
|
||||
):
|
||||
try:
|
||||
zone = ZoneInfo(tz)
|
||||
except (ZoneInfoNotFoundError, ValueError):
|
||||
raise HTTPException(422, f"Unknown time zone {tz!r}") from None
|
||||
report = _load_report(request, conn, tenant, days, all_messages=True)
|
||||
|
||||
def local(value: str | None) -> str:
|
||||
if not value:
|
||||
return ""
|
||||
return datetime.strptime(value, DB_FORMAT).replace(tzinfo=UTC).astimezone(zone).strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
out = io.StringIO()
|
||||
writer = csv.writer(out)
|
||||
writer.writerow([c.replace("Started", f"Started ({tz})").replace("Finished", f"Finished ({tz})") for c in CSV_COLUMNS])
|
||||
for job in report["jobs"]:
|
||||
for run in job["runs"]:
|
||||
writer.writerow([_safe_cell(v) for v in (
|
||||
report["tenant"], job["job_name"], job["source_system"],
|
||||
local(run["started_at"]), local(run["finished_at"]),
|
||||
run["duration_seconds"] if run["duration_seconds"] is not None else "",
|
||||
run["status"], run["size_bytes"] if run["size_bytes"] is not None else "",
|
||||
run["message"] or "",
|
||||
)])
|
||||
|
||||
slug = re.sub(r"[^A-Za-z0-9]+", "-", report["tenant"]).strip("-").lower() or "company"
|
||||
filename = f"backup-report-{slug}-{datetime.now(zone):%Y-%m-%d}.csv"
|
||||
return Response(
|
||||
# BOM so Excel reads it as UTF-8 (accents, en dashes in job names)
|
||||
content="" + out.getvalue(),
|
||||
media_type="text/csv; charset=utf-8",
|
||||
headers={"Content-Disposition": f"attachment; filename=\"{filename}\"; filename*=UTF-8''{quote(filename)}"},
|
||||
)
|
||||
@@ -0,0 +1,51 @@
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
# The modal shows this many days of run history; retention can never be shorter.
|
||||
HISTORY_DAYS = 30
|
||||
|
||||
# Shared by the systemd service (EnvironmentFile=) and manage.py, so both always use the same DB.
|
||||
DEFAULT_ENV_FILE = "/etc/backup-dashboard/backup-dashboard.env"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Settings:
|
||||
db_path: Path
|
||||
retention_days: int = 90
|
||||
# Dev only: serve ../public from the API so you can run without nginx.
|
||||
serve_static: bool = False
|
||||
# Microsoft 365 sign-in names allowed into the admin pages (lowercase)
|
||||
admins: frozenset[str] = frozenset()
|
||||
|
||||
|
||||
def read_env_file(path: Path) -> dict[str, str]:
|
||||
"""Parse simple KEY=VALUE lines (the subset systemd's EnvironmentFile accepts)."""
|
||||
values = {}
|
||||
try:
|
||||
lines = path.read_text().splitlines()
|
||||
except FileNotFoundError:
|
||||
return values
|
||||
for line in lines:
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, _, value = line.partition("=")
|
||||
values[key.strip()] = value.strip().strip("\"'")
|
||||
return values
|
||||
|
||||
|
||||
def load_settings() -> Settings:
|
||||
# Real environment variables win over the env file; the env file wins over dev defaults.
|
||||
env = read_env_file(Path(os.environ.get("BACKUP_DASH_ENV_FILE", DEFAULT_ENV_FILE))) | os.environ
|
||||
return Settings(
|
||||
db_path=Path(env.get("BACKUP_DASH_DB", "backup-dashboard.db")),
|
||||
retention_days=max(HISTORY_DAYS, int(env.get("BACKUP_DASH_RETENTION_DAYS", "90"))),
|
||||
serve_static=env.get("BACKUP_DASH_SERVE_STATIC") == "1",
|
||||
admins=parse_admins(env.get("BACKUP_DASH_ADMINS", "")),
|
||||
)
|
||||
|
||||
|
||||
def parse_admins(value: str) -> frozenset[str]:
|
||||
"""'[email protected], [email protected]' -> {'[email protected]', '[email protected]'}"""
|
||||
return frozenset(part.strip().lower() for part in value.replace(";", ",").split(",") if part.strip())
|
||||
@@ -0,0 +1,12 @@
|
||||
from datetime import UTC, datetime
|
||||
|
||||
# Every timestamp in the DB uses this exact format, so string comparison == chronological comparison.
|
||||
DB_FORMAT = "%Y-%m-%dT%H:%M:%SZ"
|
||||
|
||||
|
||||
def utcnow() -> datetime:
|
||||
return datetime.now(UTC).replace(microsecond=0)
|
||||
|
||||
|
||||
def to_db(dt: datetime) -> str:
|
||||
return dt.astimezone(UTC).strftime(DB_FORMAT)
|
||||
@@ -0,0 +1,140 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Admin CLI. Finds the database the same way the API does: BACKUP_DASH_DB from the environment,
|
||||
else from /etc/backup-dashboard/backup-dashboard.env, else ./backup-dashboard.db (dev).
|
||||
Run it as the service user so files it creates stay writable by the API:
|
||||
|
||||
sudo -u backupdash /opt/backup-dashboard/venv/bin/python manage.py create-key veeam-acme
|
||||
|
||||
python manage.py create-key veeam-acme # prints the key once
|
||||
python manage.py list-keys
|
||||
python manage.py revoke-key veeam-acme
|
||||
python manage.py list-jobs
|
||||
python manage.py delete-job 12 # e.g. a decommissioned job stuck as Overdue
|
||||
python manage.py seed-demo # 30 days of fake history for trying the UI
|
||||
python manage.py --db ./dev.db --create seed-demo
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import getpass
|
||||
import random
|
||||
import sys
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
|
||||
from app import db, keys, repo
|
||||
from app.models import IngestIn
|
||||
from app.settings import DEFAULT_ENV_FILE, load_settings
|
||||
from app.timeutil import utcnow
|
||||
|
||||
|
||||
def cli_user() -> str:
|
||||
# Recorded as who created/revoked a key; the admin page records the signed-in user instead
|
||||
try:
|
||||
return f"manage.py ({getpass.getuser()})"
|
||||
except Exception:
|
||||
return "manage.py"
|
||||
|
||||
|
||||
def create_key(conn, args):
|
||||
if not keys.valid_name(args.name):
|
||||
sys.exit("Key names use letters, digits, '.', '_' and '-', starting with a letter or digit")
|
||||
try:
|
||||
_, secret = keys.create_key(conn, args.name, created_by=cli_user())
|
||||
except keys.KeyNameTaken:
|
||||
sys.exit(f"A key named {args.name!r} already exists")
|
||||
print(f"API key for {args.name!r} (shown once, store it in the collector's config):\n{secret}")
|
||||
|
||||
|
||||
def list_keys(conn, args):
|
||||
for k in keys.list_keys(conn):
|
||||
state = f"revoked {k['revoked_at']}" if k["revoked_at"] else "active"
|
||||
tenant = f" ({k['last_used_tenant']})" if k["last_used_tenant"] else ""
|
||||
print(f"{k['name']:<30} {state:<30} created {k['created_at']} last used {k['last_used_at'] or 'never'}{tenant}")
|
||||
|
||||
|
||||
def revoke_key(conn, args):
|
||||
row = conn.execute("SELECT id FROM api_keys WHERE name = ? AND revoked_at IS NULL", (args.name,)).fetchone()
|
||||
if row is None:
|
||||
sys.exit(f"No active key named {args.name!r}")
|
||||
keys.revoke_key(conn, row["id"], revoked_by=cli_user())
|
||||
|
||||
|
||||
def list_jobs(conn, args):
|
||||
for j in repo.list_jobs(conn):
|
||||
print(f"{j['id']:>5} {j['company_name']:<28} {j['job_name']:<40} {j['status'] or '-':<8} last {j['last_run_at'] or 'never'}")
|
||||
|
||||
|
||||
def delete_job(conn, args):
|
||||
cur = conn.execute("DELETE FROM jobs WHERE id = ?", (args.job_id,))
|
||||
sys.exit(0 if cur.rowcount else f"No job with id {args.job_id}")
|
||||
|
||||
|
||||
DEMO = [
|
||||
("Acme Corp", "Veeam B&R", "File Server - Nightly", 24),
|
||||
("Acme Corp", "Veeam B&R", "SQL01 - Transaction Logs", 4),
|
||||
("Bluewater Dental", "Datto", "Workstations - Daily", 24),
|
||||
("Bluewater Dental", "Datto", "Imaging Server - Hourly", 1),
|
||||
("Northfield Logistics", "Acronis", "Exchange - Nightly", 24),
|
||||
]
|
||||
|
||||
|
||||
def seed_demo(conn, args):
|
||||
rng = random.Random(42)
|
||||
now = utcnow()
|
||||
for tenant, source, name, every_h in DEMO:
|
||||
ext = name.lower().replace(" ", "-")
|
||||
# Offset so runs don't all line up exactly on the current time
|
||||
start = now - timedelta(days=30) + timedelta(minutes=rng.randint(5, every_h * 60 - 5))
|
||||
while start < now:
|
||||
dur = timedelta(minutes=rng.randint(2, 50 if every_h > 1 else 15))
|
||||
end = start + dur
|
||||
status = "running" if end > now else rng.choices(["success", "warning", "failed"], [90, 7, 3])[0]
|
||||
run = {"status": status, "started_at": start.isoformat()}
|
||||
if status != "running":
|
||||
run |= {"finished_at": end.isoformat(), "size_bytes": rng.randint(10**8, 10**11),
|
||||
"message": {"success": "Job completed successfully.",
|
||||
"warning": "Completed with warnings: 1 file skipped (locked).",
|
||||
"failed": "Error: target repository unreachable."}[status]}
|
||||
repo.ingest(conn, IngestIn.model_validate({
|
||||
"tenant": tenant, "source_system": source,
|
||||
"job": {"external_id": ext, "name": name,
|
||||
"next_run_at": (start + timedelta(hours=every_h)).isoformat()},
|
||||
"run": run,
|
||||
}))
|
||||
start += timedelta(hours=every_h)
|
||||
print(f"Seeded {len(DEMO)} demo jobs")
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--db", type=Path, help="database file (overrides BACKUP_DASH_DB and the env file)")
|
||||
p.add_argument("--create", action="store_true", help="create the database if it doesn't exist")
|
||||
sub = p.add_subparsers(required=True)
|
||||
s = sub.add_parser("create-key"); s.add_argument("name"); s.set_defaults(fn=create_key)
|
||||
s = sub.add_parser("list-keys"); s.set_defaults(fn=list_keys)
|
||||
s = sub.add_parser("revoke-key"); s.add_argument("name"); s.set_defaults(fn=revoke_key)
|
||||
s = sub.add_parser("list-jobs"); s.set_defaults(fn=list_jobs)
|
||||
s = sub.add_parser("delete-job"); s.add_argument("job_id", type=int); s.set_defaults(fn=delete_job)
|
||||
s = sub.add_parser("seed-demo"); s.set_defaults(fn=seed_demo)
|
||||
args = p.parse_args()
|
||||
|
||||
db_path = args.db or load_settings().db_path
|
||||
# Refuse to silently create a fresh DB: that almost always means we're looking in the wrong
|
||||
# place, and anything written there (like a new API key) would be invisible to the service.
|
||||
if not db_path.exists() and not args.create:
|
||||
sys.exit(
|
||||
f"Database not found: {db_path.resolve()}\n"
|
||||
f"Set BACKUP_DASH_DB in {DEFAULT_ENV_FILE} (or pass --db), "
|
||||
"or add --create to start a new database there."
|
||||
)
|
||||
print(f"Using database {db_path.resolve()}", file=sys.stderr)
|
||||
db.init_db(db_path)
|
||||
conn = db.connect(db_path)
|
||||
try:
|
||||
args.fn(conn, args)
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,2 @@
|
||||
[pytest]
|
||||
pythonpath = .
|
||||
@@ -0,0 +1,2 @@
|
||||
httpx
|
||||
pytest
|
||||
@@ -0,0 +1,4 @@
|
||||
fastapi>=0.115
|
||||
uvicorn[standard]>=0.30
|
||||
# Time zone data for report times, where the OS has none
|
||||
tzdata
|
||||
@@ -0,0 +1,146 @@
|
||||
import sqlite3
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app import db
|
||||
from app.main import create_app
|
||||
from app.settings import Settings, parse_admins
|
||||
|
||||
ADMIN = {"X-Forwarded-Email": "[email protected]"}
|
||||
PAGE = {"X-Requested-By": "backup-dashboard"}
|
||||
ADMIN_PAGE = ADMIN | PAGE
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(tmp_path):
|
||||
settings = Settings(db_path=tmp_path / "test.db", admins=frozenset({"[email protected]"}))
|
||||
with TestClient(create_app(settings)) as c:
|
||||
yield c
|
||||
|
||||
|
||||
def create(client, name="veeam-acme", headers=ADMIN_PAGE):
|
||||
return client.post("/api/v1/admin/keys", json={"name": name}, headers=headers)
|
||||
|
||||
|
||||
def ingest(client, secret, tenant="Acme"):
|
||||
return client.post(
|
||||
"/api/v1/ingest",
|
||||
json={"tenant": tenant, "source_system": "Veeam B&R", "job": {"external_id": "j1", "name": "Job"}},
|
||||
headers={"Authorization": f"Bearer {secret}"},
|
||||
)
|
||||
|
||||
|
||||
def test_parse_admins():
|
||||
assert parse_admins(" [email protected], [email protected];[email protected] ,") == {
|
||||
"[email protected]", "[email protected]", "[email protected]"}
|
||||
assert parse_admins("") == frozenset()
|
||||
|
||||
|
||||
def test_me_says_whether_the_user_is_an_admin(client):
|
||||
assert client.get("/api/v1/me", headers=ADMIN).json()["is_admin"] is True
|
||||
assert client.get("/api/v1/me", headers={"X-Forwarded-Email": "[email protected]"}).json()["is_admin"] is False
|
||||
assert client.get("/api/v1/me").json()["is_admin"] is False
|
||||
|
||||
|
||||
@pytest.mark.parametrize("headers", [{}, {"X-Forwarded-Email": "[email protected]"}])
|
||||
def test_admin_pages_are_admins_only(client, headers):
|
||||
assert client.get("/api/v1/admin/keys", headers=headers).status_code == 403
|
||||
assert create(client, headers=headers | PAGE).status_code == 403
|
||||
assert client.post("/api/v1/admin/keys/1/revoke", headers=headers | PAGE).status_code == 403
|
||||
|
||||
|
||||
def test_admin_match_ignores_case_and_accepts_the_sign_in_name(client):
|
||||
assert client.get("/api/v1/admin/keys", headers={"X-Forwarded-Email": "[email protected]"}).status_code == 200
|
||||
assert client.get("/api/v1/admin/keys", headers={"X-Forwarded-Preferred-Username": "[email protected]"}).status_code == 200
|
||||
|
||||
|
||||
def test_changes_need_the_page_header(client):
|
||||
# Without it, a page on another site could make a signed-in admin's browser send the request
|
||||
assert create(client, headers=ADMIN).status_code == 403
|
||||
secret = create(client).json()["secret"]
|
||||
key_id = client.get("/api/v1/admin/keys", headers=ADMIN).json()["keys"][0]["id"]
|
||||
assert client.post(f"/api/v1/admin/keys/{key_id}/revoke", headers=ADMIN).status_code == 403
|
||||
assert ingest(client, secret).status_code == 200 # still active
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name", ["", " ", "-leading-dash", "has space", "a" * 65, "semi;colon"])
|
||||
def test_rejects_bad_key_names(client, name):
|
||||
assert create(client, name=name).status_code == 422
|
||||
|
||||
|
||||
def test_created_key_is_shown_once_and_works_for_ingest(client):
|
||||
r = create(client, name=" veeam-acme ")
|
||||
assert r.status_code == 201
|
||||
body = r.json()
|
||||
assert body["secret"].startswith("bdk_")
|
||||
assert body["key"]["name"] == "veeam-acme"
|
||||
assert body["key"]["created_by"] == "[email protected]"
|
||||
listed = client.get("/api/v1/admin/keys", headers=ADMIN).json()["keys"]
|
||||
assert "secret" not in listed[0] and "key_hash" not in listed[0]
|
||||
assert ingest(client, body["secret"]).status_code == 200
|
||||
|
||||
|
||||
def test_key_records_when_and_for_which_company_it_was_last_used(client):
|
||||
secret = create(client).json()["secret"]
|
||||
ingest(client, secret, tenant="Acme Corp")
|
||||
key = client.get("/api/v1/admin/keys", headers=ADMIN).json()["keys"][0]
|
||||
assert key["last_used_at"] is not None
|
||||
assert key["last_used_tenant"] == "Acme Corp"
|
||||
|
||||
|
||||
def test_duplicate_names_are_refused_even_after_revoking(client):
|
||||
key_id = create(client).json()["key"]["id"]
|
||||
assert create(client).status_code == 409
|
||||
client.post(f"/api/v1/admin/keys/{key_id}/revoke", headers=ADMIN_PAGE)
|
||||
assert create(client).status_code == 409
|
||||
|
||||
|
||||
def test_revoking_stops_the_key_and_records_who(client):
|
||||
created = create(client).json()
|
||||
r = client.post(f"/api/v1/admin/keys/{created['key']['id']}/revoke", headers=ADMIN_PAGE)
|
||||
assert r.status_code == 200
|
||||
assert r.json()["key"]["revoked_by"] == "[email protected]"
|
||||
assert r.json()["key"]["revoked_at"] is not None
|
||||
assert ingest(client, created["secret"]).status_code == 401
|
||||
|
||||
|
||||
def test_revoking_twice_keeps_the_first_revocation(client):
|
||||
key_id = create(client).json()["key"]["id"]
|
||||
first = client.post(f"/api/v1/admin/keys/{key_id}/revoke", headers=ADMIN_PAGE).json()["key"]
|
||||
again = client.post(f"/api/v1/admin/keys/{key_id}/revoke",
|
||||
headers={"X-Forwarded-Email": "[email protected]"} | PAGE).json()["key"]
|
||||
assert again["revoked_at"] == first["revoked_at"]
|
||||
|
||||
|
||||
def test_revoking_an_unknown_key_is_404(client):
|
||||
assert client.post("/api/v1/admin/keys/999/revoke", headers=ADMIN_PAGE).status_code == 404
|
||||
|
||||
|
||||
def test_lists_active_keys_first(client):
|
||||
for name in ("b-key", "a-key", "c-key"):
|
||||
create(client, name=name)
|
||||
a_id = next(k["id"] for k in client.get("/api/v1/admin/keys", headers=ADMIN).json()["keys"] if k["name"] == "a-key")
|
||||
client.post(f"/api/v1/admin/keys/{a_id}/revoke", headers=ADMIN_PAGE)
|
||||
names = [k["name"] for k in client.get("/api/v1/admin/keys", headers=ADMIN).json()["keys"]]
|
||||
assert names == ["b-key", "c-key", "a-key"]
|
||||
|
||||
|
||||
def test_upgrades_a_version_1_database_keeping_its_keys(tmp_path):
|
||||
path = tmp_path / "old.db"
|
||||
conn = sqlite3.connect(path)
|
||||
conn.executescript(db.SCHEMA) # the v1 schema, as deployed before admin pages
|
||||
conn.execute("INSERT INTO api_keys (name, key_hash, created_at) VALUES ('old-key', 'abc', '2026-10-01T00:00:00Z')")
|
||||
conn.execute("PRAGMA user_version = 1")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
db.init_db(path)
|
||||
db.init_db(path) # and running it again is harmless
|
||||
|
||||
conn = db.connect(path)
|
||||
assert conn.execute("PRAGMA user_version").fetchone()[0] == db.SCHEMA_VERSION
|
||||
row = dict(conn.execute("SELECT * FROM api_keys").fetchone())
|
||||
assert row["name"] == "old-key"
|
||||
assert {"created_by", "revoked_by", "last_used_tenant"} <= row.keys()
|
||||
conn.close()
|
||||
@@ -0,0 +1,130 @@
|
||||
from datetime import timedelta
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app import db
|
||||
from app.auth import generate_key, hash_key
|
||||
from app.main import create_app
|
||||
from app.settings import Settings
|
||||
from app.timeutil import to_db, utcnow
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(tmp_path):
|
||||
settings = Settings(db_path=tmp_path / "test.db")
|
||||
with TestClient(create_app(settings)) as c:
|
||||
key = generate_key()
|
||||
conn = db.connect(settings.db_path)
|
||||
conn.execute("INSERT INTO api_keys (name, key_hash, created_at) VALUES ('test', ?, ?)",
|
||||
(hash_key(key), to_db(utcnow())))
|
||||
conn.close()
|
||||
c.headers["Authorization"] = f"Bearer {key}"
|
||||
yield c
|
||||
|
||||
|
||||
def iso(delta: timedelta) -> str:
|
||||
return (utcnow() + delta).isoformat()
|
||||
|
||||
|
||||
def payload(run=None, **job):
|
||||
body = {"tenant": "Acme Corp", "source_system": "Veeam B&R",
|
||||
"job": {"external_id": "job-1", "name": "File Server", **job}}
|
||||
if run is not None:
|
||||
body["run"] = run
|
||||
return body
|
||||
|
||||
|
||||
def test_ingest_requires_valid_key(client):
|
||||
assert client.post("/api/v1/ingest", json=payload(), headers={"Authorization": ""}).status_code == 401
|
||||
assert client.post("/api/v1/ingest", json=payload(), headers={"Authorization": "Bearer nope"}).status_code == 401
|
||||
|
||||
|
||||
def test_running_then_finished_merges_into_one_run(client):
|
||||
started = iso(-timedelta(hours=1))
|
||||
r = client.post("/api/v1/ingest", json=payload({"status": "running", "started_at": started}))
|
||||
assert r.status_code == 200, r.text
|
||||
job_id = r.json()["job_id"]
|
||||
|
||||
job = client.get("/api/v1/jobs").json()["jobs"][0]
|
||||
assert job["status"] == "running"
|
||||
assert job["running_since"] is not None
|
||||
assert job["last_run_at"] is None # nothing has finished yet
|
||||
|
||||
finished = iso(-timedelta(minutes=10))
|
||||
client.post("/api/v1/ingest", json=payload(
|
||||
{"status": "warning", "started_at": started, "finished_at": finished, "message": "1 file skipped"}))
|
||||
|
||||
detail = client.get(f"/api/v1/jobs/{job_id}").json()
|
||||
assert detail["status"] == "warning"
|
||||
assert detail["running_since"] is None
|
||||
assert detail["last_run_at"] == to_db(utcnow().fromisoformat(finished))
|
||||
assert detail["duration_seconds"] == 50 * 60
|
||||
assert detail["message"] == "1 file skipped"
|
||||
assert len(client.get(f"/api/v1/jobs/{job_id}/runs").json()["runs"]) == 1
|
||||
|
||||
|
||||
def test_late_running_report_does_not_reopen_finished_run(client):
|
||||
started = iso(-timedelta(hours=1))
|
||||
client.post("/api/v1/ingest", json=payload({"status": "success", "started_at": started, "finished_at": iso(-timedelta(minutes=5))}))
|
||||
client.post("/api/v1/ingest", json=payload({"status": "running", "started_at": started}))
|
||||
assert client.get("/api/v1/jobs").json()["jobs"][0]["status"] == "success"
|
||||
|
||||
|
||||
def test_last_run_is_previous_finish_while_new_run_in_progress(client):
|
||||
prev_end = iso(-timedelta(hours=23))
|
||||
client.post("/api/v1/ingest", json=payload({"status": "failed", "started_at": iso(-timedelta(hours=24)), "finished_at": prev_end, "message": "boom"}))
|
||||
client.post("/api/v1/ingest", json=payload({"status": "running", "started_at": iso(-timedelta(minutes=5))}))
|
||||
job = client.get("/api/v1/jobs").json()["jobs"][0]
|
||||
assert job["status"] == "running"
|
||||
assert job["last_run_at"] == to_db(utcnow().fromisoformat(prev_end))
|
||||
detail = client.get(f"/api/v1/jobs/{job['id']}").json()
|
||||
assert detail["last_status"] == "failed"
|
||||
assert detail["message"] == "boom"
|
||||
|
||||
|
||||
def test_next_run_at_only_changes_when_sent(client):
|
||||
nxt = iso(timedelta(hours=5))
|
||||
client.post("/api/v1/ingest", json=payload(next_run_at=nxt))
|
||||
client.post("/api/v1/ingest", json=payload()) # omitted → keep
|
||||
assert client.get("/api/v1/jobs").json()["jobs"][0]["next_run_at"] == to_db(utcnow().fromisoformat(nxt))
|
||||
client.post("/api/v1/ingest", json=payload(next_run_at=None)) # explicit null → clear
|
||||
assert client.get("/api/v1/jobs").json()["jobs"][0]["next_run_at"] is None
|
||||
|
||||
|
||||
def test_history_window_and_run_detail(client):
|
||||
for days_ago in (40, 20, 1):
|
||||
start = iso(-timedelta(days=days_ago))
|
||||
client.post("/api/v1/ingest", json=payload({"status": "success", "started_at": start,
|
||||
"finished_at": start, "message": f"{days_ago}d"}))
|
||||
job_id = client.get("/api/v1/jobs").json()["jobs"][0]["id"]
|
||||
runs = client.get(f"/api/v1/jobs/{job_id}/runs").json()["runs"]
|
||||
assert len(runs) == 2 # the 40-day-old run is outside the 30-day window
|
||||
assert "message" not in runs[0]
|
||||
run = client.get(f"/api/v1/runs/{runs[0]['id']}").json()
|
||||
assert run["message"] == "1d"
|
||||
assert client.get(f"/api/v1/jobs/{job_id}/runs?days=31").status_code == 422
|
||||
|
||||
|
||||
@pytest.mark.parametrize("run", [
|
||||
{"status": "running", "started_at": iso(-timedelta(hours=1)), "finished_at": iso(timedelta(0))},
|
||||
{"status": "success", "started_at": iso(timedelta(0)), "finished_at": iso(-timedelta(hours=1))},
|
||||
{"status": "success", "started_at": "2026-01-01T00:00:00"}, # no timezone
|
||||
{"status": "done", "started_at": iso(timedelta(0))},
|
||||
])
|
||||
def test_rejects_bad_runs(client, run):
|
||||
assert client.post("/api/v1/ingest", json=payload(run)).status_code == 422
|
||||
|
||||
|
||||
def test_tenant_name_is_case_insensitive(client):
|
||||
client.post("/api/v1/ingest", json=payload())
|
||||
body = payload()
|
||||
body["tenant"] = "ACME CORP"
|
||||
client.post("/api/v1/ingest", json=body)
|
||||
assert len(client.get("/api/v1/jobs").json()["jobs"]) == 1
|
||||
|
||||
|
||||
def test_me_reports_signed_in_user_from_proxy_headers(client):
|
||||
assert client.get("/api/v1/me").json() == {"email": None, "name": None, "is_admin": False}
|
||||
r = client.get("/api/v1/me", headers={"X-Forwarded-Email": "[email protected]", "X-Forwarded-User": "abc-123"})
|
||||
assert r.json() == {"email": "[email protected]", "name": "abc-123", "is_admin": False}
|
||||
@@ -0,0 +1,52 @@
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from app.settings import load_settings
|
||||
|
||||
BACKEND = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def run_manage(*args, env, cwd):
|
||||
return subprocess.run([sys.executable, str(BACKEND / "manage.py"), *args],
|
||||
capture_output=True, text=True, env=env, cwd=cwd)
|
||||
|
||||
|
||||
def base_env(tmp_path, monkeypatch):
|
||||
env_file = tmp_path / "backup-dashboard.env"
|
||||
monkeypatch.setenv("BACKUP_DASH_ENV_FILE", str(env_file))
|
||||
monkeypatch.delenv("BACKUP_DASH_DB", raising=False)
|
||||
return env_file, {"BACKUP_DASH_ENV_FILE": str(env_file), "PYTHONPATH": str(BACKEND), "PATH": "/usr/bin:/bin"}
|
||||
|
||||
|
||||
def test_env_file_supplies_db_path_and_real_env_wins(tmp_path, monkeypatch):
|
||||
env_file, _ = base_env(tmp_path, monkeypatch)
|
||||
env_file.write_text('# comment\nBACKUP_DASH_DB="/srv/from-file.db"\nBACKUP_DASH_RETENTION_DAYS=120\n')
|
||||
assert load_settings().db_path == Path("/srv/from-file.db")
|
||||
assert load_settings().retention_days == 120
|
||||
monkeypatch.setenv("BACKUP_DASH_DB", "/srv/from-env.db")
|
||||
assert load_settings().db_path == Path("/srv/from-env.db")
|
||||
|
||||
|
||||
def test_manage_refuses_to_create_db_implicitly(tmp_path, monkeypatch):
|
||||
_, env = base_env(tmp_path, monkeypatch)
|
||||
workdir = tmp_path / "somewhere-else"
|
||||
workdir.mkdir()
|
||||
r = run_manage("create-key", "k", env=env, cwd=workdir)
|
||||
assert r.returncode != 0
|
||||
assert "Database not found" in r.stderr
|
||||
assert not (workdir / "backup-dashboard.db").exists()
|
||||
|
||||
|
||||
def test_manage_uses_db_from_env_file(tmp_path, monkeypatch):
|
||||
env_file, env = base_env(tmp_path, monkeypatch)
|
||||
service_db = tmp_path / "service.db"
|
||||
env_file.write_text(f"BACKUP_DASH_DB={service_db}\n")
|
||||
assert run_manage("--create", "list-keys", env=env, cwd=tmp_path).returncode == 0 # service's first start
|
||||
workdir = tmp_path / "somewhere-else"
|
||||
workdir.mkdir()
|
||||
r = run_manage("create-key", "veeam-acme", env=env, cwd=workdir)
|
||||
assert r.returncode == 0, r.stderr
|
||||
assert str(service_db) in r.stderr
|
||||
assert "veeam-acme" in run_manage("list-keys", env=env, cwd=tmp_path).stdout
|
||||
assert not (workdir / "backup-dashboard.db").exists()
|
||||
@@ -0,0 +1,133 @@
|
||||
import csv
|
||||
import io
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app import db
|
||||
from app.auth import generate_key, hash_key
|
||||
from app.main import create_app
|
||||
from app.settings import Settings
|
||||
from app.timeutil import to_db, utcnow
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(tmp_path):
|
||||
settings = Settings(db_path=tmp_path / "test.db", retention_days=90)
|
||||
with TestClient(create_app(settings)) as c:
|
||||
key = generate_key()
|
||||
conn = db.connect(settings.db_path)
|
||||
conn.execute("INSERT INTO api_keys (name, key_hash, created_at) VALUES ('t', ?, ?)", (hash_key(key), to_db(utcnow())))
|
||||
conn.close()
|
||||
c.headers["Authorization"] = f"Bearer {key}"
|
||||
yield c
|
||||
|
||||
|
||||
def report_run(client, tenant, job, status, started, minutes=10, message="", size=None):
|
||||
run = {"status": status, "started_at": started.isoformat(), "message": message}
|
||||
if status != "running":
|
||||
run["finished_at"] = (started + timedelta(minutes=minutes)).isoformat()
|
||||
if size is not None:
|
||||
run["size_bytes"] = size
|
||||
r = client.post("/api/v1/ingest", json={
|
||||
"tenant": tenant, "source_system": "Veeam B&R", "job": {"external_id": job, "name": job}, "run": run})
|
||||
assert r.status_code == 200, r.text
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def alpha(client):
|
||||
now = utcnow()
|
||||
report_run(client, "Alpha Construction", "File Server", "success", now - timedelta(days=1), size=1000,
|
||||
message="Job completed successfully.")
|
||||
report_run(client, "Alpha Construction", "File Server", "failed", now - timedelta(days=2),
|
||||
message="=HYPERLINK(\"http://evil\")\nRepository unreachable")
|
||||
report_run(client, "Alpha Construction", "File Server", "success", now - timedelta(days=45)) # outside 30 days
|
||||
report_run(client, "Alpha Construction", "SQL01", "warning", now - timedelta(hours=5), message="Log truncation skipped")
|
||||
report_run(client, "Alpha Construction", "SQL01", "running", now - timedelta(minutes=5))
|
||||
report_run(client, "Bluewater Dental", "Imaging", "success", now - timedelta(days=1))
|
||||
return now
|
||||
|
||||
|
||||
def test_lists_companies_with_jobs(client, alpha):
|
||||
assert client.get("/api/v1/tenants").json()["tenants"] == [
|
||||
{"name": "Alpha Construction", "jobs": 2}, {"name": "Bluewater Dental", "jobs": 1}]
|
||||
|
||||
|
||||
def test_report_covers_one_company_for_the_period(client, alpha):
|
||||
r = client.get("/api/v1/reports/tenant", params={"tenant": "alpha construction"}) # any case
|
||||
assert r.status_code == 200
|
||||
report = r.json()
|
||||
assert report["tenant"] == "Alpha Construction"
|
||||
assert report["days"] == 30
|
||||
jobs = {j["job_name"]: j for j in report["jobs"]}
|
||||
assert set(jobs) == {"File Server", "SQL01"} # not Bluewater's
|
||||
assert [run["status"] for run in jobs["File Server"]["runs"]] == ["failed", "success"] # oldest first; 45-day-old run left out
|
||||
assert [run["status"] for run in jobs["SQL01"]["runs"]] == ["warning", "running"]
|
||||
assert jobs["SQL01"]["status"] == "running"
|
||||
|
||||
|
||||
def test_report_says_when_each_job_was_first_tracked(client, alpha):
|
||||
jobs = {j["job_name"]: j for j in client.get("/api/v1/reports/tenant", params={"tenant": "Alpha Construction"}).json()["jobs"]}
|
||||
# File Server's earliest run (45 days ago, outside the report) predates its first report to the API
|
||||
assert jobs["File Server"]["tracked_since"] == to_db(alpha - timedelta(days=45))
|
||||
# SQL01's runs are all recent; it was first reported (created) when its first run arrived
|
||||
assert jobs["SQL01"]["tracked_since"] <= to_db(alpha - timedelta(hours=5))
|
||||
|
||||
|
||||
def test_report_includes_messages_only_for_warnings_and_failures(client, alpha):
|
||||
jobs = {j["job_name"]: j for j in client.get("/api/v1/reports/tenant", params={"tenant": "Alpha Construction"}).json()["jobs"]}
|
||||
failed, success = jobs["File Server"]["runs"]
|
||||
assert "Repository unreachable" in failed["message"]
|
||||
assert success["message"] is None
|
||||
assert success["size_bytes"] == 1000
|
||||
assert success["duration_seconds"] == 600
|
||||
|
||||
|
||||
def test_report_period_is_limited_to_retention(client, alpha):
|
||||
assert client.get("/api/v1/reports/tenant", params={"tenant": "Alpha Construction", "days": 90}).status_code == 200
|
||||
r = client.get("/api/v1/reports/tenant", params={"tenant": "Alpha Construction", "days": 91})
|
||||
assert r.status_code == 422
|
||||
assert "90 days" in r.json()["detail"]
|
||||
assert client.get("/api/v1/reports/tenant", params={"tenant": "Alpha Construction", "days": 0}).status_code == 422
|
||||
|
||||
|
||||
def test_unknown_company_is_404(client, alpha):
|
||||
assert client.get("/api/v1/reports/tenant", params={"tenant": "Nobody Inc"}).status_code == 404
|
||||
|
||||
|
||||
def read_csv(response):
|
||||
text = response.content.decode("utf-8")
|
||||
assert text.startswith("") # so Excel reads it as UTF-8
|
||||
return list(csv.reader(io.StringIO(text[1:])))
|
||||
|
||||
|
||||
def test_csv_has_every_run_with_times_in_the_given_zone(client):
|
||||
started = datetime(2026, 1, 15, 12, 0, tzinfo=UTC) # 05:00 in Edmonton (MST, -7)
|
||||
# The fixture's timestamps are relative to now; this run needs a fixed winter date in range
|
||||
client.app.state.settings = Settings(db_path=client.app.state.settings.db_path, retention_days=10_000)
|
||||
report_run(client, "Alpha Construction", "File Server", "success", started, minutes=20, size=5,
|
||||
message="Job completed successfully.")
|
||||
r = client.get("/api/v1/reports/tenant.csv", params={"tenant": "Alpha Construction", "days": 5000, "tz": "America/Edmonton"})
|
||||
assert r.status_code == 200
|
||||
assert r.headers["content-type"].startswith("text/csv")
|
||||
rows = read_csv(r)
|
||||
assert rows[0] == ["Company", "Job", "Source", "Started (America/Edmonton)", "Finished (America/Edmonton)",
|
||||
"Duration (seconds)", "Status", "Size (bytes)", "Message"]
|
||||
assert rows[1] == ["Alpha Construction", "File Server", "Veeam B&R", "2026-01-15 05:00:00", "2026-01-15 05:20:00",
|
||||
"1200", "success", "5", "Job completed successfully."]
|
||||
|
||||
|
||||
def test_csv_defuses_spreadsheet_formulas_and_keeps_full_messages(client, alpha):
|
||||
rows = read_csv(client.get("/api/v1/reports/tenant.csv", params={"tenant": "Alpha Construction"}))
|
||||
by_status = {row[6]: row for row in rows[1:]}
|
||||
assert by_status["failed"][8].startswith("'=HYPERLINK")
|
||||
assert by_status["success"][8] == "Job completed successfully." # successes' messages too, unlike the JSON
|
||||
assert by_status["running"][4] == "" # not finished
|
||||
assert len(rows) == 1 + 4 # header + the 4 runs in the last 30 days
|
||||
|
||||
|
||||
def test_csv_filename_and_bad_time_zone(client, alpha):
|
||||
r = client.get("/api/v1/reports/tenant.csv", params={"tenant": "Alpha Construction"})
|
||||
assert 'filename="backup-report-alpha-construction-' in r.headers["content-disposition"]
|
||||
assert client.get("/api/v1/reports/tenant.csv", params={"tenant": "Alpha Construction", "tz": "Mars/Olympus"}).status_code == 422
|
||||
@@ -0,0 +1,270 @@
|
||||
<#
|
||||
Shared code for the Backup Dashboard collectors: config and state files, logging, posting to
|
||||
the ingest API, and installing a collector as a scheduled task.
|
||||
|
||||
Each collector imports this from next to itself (installed: C:\Program Files\BackupDashboard)
|
||||
or from ..\common (in the repo). Collectors on the same server share one installed copy, so
|
||||
keep these functions backward compatible.
|
||||
|
||||
Works on Windows PowerShell 5.1 and PowerShell 7.
|
||||
#>
|
||||
|
||||
$script:Config = $null
|
||||
$script:DryRun = $false
|
||||
$script:LogPath = $null
|
||||
$script:StatePath = $null
|
||||
$script:LookbackHours = 48
|
||||
$script:Sent = @{}
|
||||
$script:RunsSent = 0
|
||||
$script:MaxLogBytes = 5MB
|
||||
$script:Invariant = [Globalization.CultureInfo]::InvariantCulture
|
||||
|
||||
# ---------- Logging and values ----------
|
||||
|
||||
function Write-Log {
|
||||
param([string]$Message, [string]$Level = 'INFO')
|
||||
$line = '{0} [{1}] {2}' -f (Get-Date).ToString('yyyy-MM-dd HH:mm:ss'), $Level, $Message
|
||||
Write-Host $line
|
||||
if ($script:DryRun -or -not $script:LogPath) { return }
|
||||
if ((Test-Path $script:LogPath) -and (Get-Item $script:LogPath).Length -gt $script:MaxLogBytes) {
|
||||
Move-Item $script:LogPath "$($script:LogPath).1" -Force
|
||||
}
|
||||
Add-Content -Path $script:LogPath -Value $line -Encoding UTF8
|
||||
}
|
||||
|
||||
function Get-ErrorText($ErrorRecord) {
|
||||
# Windows PowerShell puts the HTTP response body (e.g. the API's 422 explanation) here
|
||||
if ($ErrorRecord.ErrorDetails -and $ErrorRecord.ErrorDetails.Message) {
|
||||
return "$($ErrorRecord.Exception.Message) $($ErrorRecord.ErrorDetails.Message)"
|
||||
}
|
||||
return $ErrorRecord.Exception.Message
|
||||
}
|
||||
|
||||
# The API needs an explicit UTC offset. DateTimeOffset keeps a Local/Unspecified time as local and a
|
||||
# Utc time as +00:00, so convert UTC-valued times to Kind Utc before calling this.
|
||||
function ConvertTo-IsoTime($Value) {
|
||||
if ($null -eq $Value) { return $null }
|
||||
$dt = [datetime]$Value
|
||||
if ($dt.Year -le 1900) { return $null } # "not set" (DateTime.MinValue, MARS's 1601-01-01, ...)
|
||||
return ([DateTimeOffset]$dt).ToString("yyyy-MM-dd'T'HH:mm:sszzz", $script:Invariant)
|
||||
}
|
||||
|
||||
# First property of $Object, out of $Names, that exists and isn't empty. Backup products' object
|
||||
# models aren't fully documented, so reads go through this rather than assuming one spelling.
|
||||
function Get-FirstValue($Object, [string[]]$Names) {
|
||||
if ($null -eq $Object) { return $null }
|
||||
foreach ($name in $Names) {
|
||||
try {
|
||||
$prop = $Object.PSObject.Properties[$name]
|
||||
if (-not $prop) { continue }
|
||||
$value = $prop.Value
|
||||
# Check emptiness by type: casting to [string] would make a one-item list of objects ''
|
||||
if ($null -eq $value) { continue }
|
||||
if ($value -is [string] -and $value -eq '') { continue }
|
||||
if ($value -is [Collections.ICollection] -and $value.Count -eq 0) { continue }
|
||||
return $value
|
||||
} catch { }
|
||||
}
|
||||
return $null
|
||||
}
|
||||
|
||||
# ---------- Collector run ----------
|
||||
|
||||
<#
|
||||
Loads the config ($ConfigPath, JSON) and the record of runs already sent, and sets up logging
|
||||
and TLS. Log and state files sit next to the config: <Name>-collector.log / .state.json.
|
||||
Returns the config object; its LookbackHours is resolved (the parameter, else the config, else
|
||||
48). Throws if the config can't be read or lacks one of $RequiredKeys.
|
||||
#>
|
||||
function Initialize-Collector {
|
||||
param(
|
||||
[Parameter(Mandatory)][string]$Name,
|
||||
[Parameter(Mandatory)][string]$ConfigPath,
|
||||
[int]$LookbackHours = 0,
|
||||
[switch]$DryRun,
|
||||
[string[]]$RequiredKeys = @('ApiUrl', 'ApiKey', 'Tenant')
|
||||
)
|
||||
# Logging first, so a bad config can still be logged
|
||||
$dataDir = Split-Path -Parent $ConfigPath
|
||||
$script:LogPath = Join-Path $dataDir "$Name-collector.log"
|
||||
$script:StatePath = Join-Path $dataDir "$Name-collector.state.json"
|
||||
$script:DryRun = [bool]$DryRun
|
||||
$script:RunsSent = 0
|
||||
$script:Sent = @{}
|
||||
|
||||
$config = Get-Content -Path $ConfigPath -Raw | ConvertFrom-Json
|
||||
foreach ($key in $RequiredKeys) {
|
||||
if (-not $config.$key) { throw "Config $ConfigPath is missing '$key'" }
|
||||
}
|
||||
if ($LookbackHours -le 0) {
|
||||
$LookbackHours = if ($config.LookbackHours) { [int]$config.LookbackHours } else { 48 }
|
||||
}
|
||||
$config | Add-Member -NotePropertyName LookbackHours -NotePropertyValue $LookbackHours -Force
|
||||
$script:LookbackHours = $LookbackHours
|
||||
$script:Config = $config
|
||||
|
||||
# Windows PowerShell 5.1 may default to TLS 1.0
|
||||
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
|
||||
|
||||
# Run key -> start time (Unix seconds), for runs already delivered in their final state.
|
||||
# Numbers rather than ISO strings: PowerShell 7's ConvertFrom-Json turns date strings into
|
||||
# DateTimes and loses the offset.
|
||||
if (Test-Path $script:StatePath) {
|
||||
$saved = Get-Content -Path $script:StatePath -Raw | ConvertFrom-Json
|
||||
if ($saved) { foreach ($p in $saved.PSObject.Properties) { $script:Sent[$p.Name] = [long]$p.Value } }
|
||||
}
|
||||
return $config
|
||||
}
|
||||
|
||||
function Test-RunSent([string]$Key) {
|
||||
return $script:Sent.ContainsKey($Key)
|
||||
}
|
||||
|
||||
<#
|
||||
Posts one report to the ingest API: the job alone (to refresh next_run_at) or the job with a
|
||||
run. A run that has finished is then recorded as sent under $SentKey, so later collector runs
|
||||
skip it; a running one isn't, so it's re-sent until it finishes. Throws on an HTTP error.
|
||||
#>
|
||||
function Send-JobReport {
|
||||
param(
|
||||
[Parameter(Mandatory)][string]$Tenant,
|
||||
[Parameter(Mandatory)][string]$SourceSystem,
|
||||
[Parameter(Mandatory)]$Job,
|
||||
$Run,
|
||||
[string]$SentKey,
|
||||
$StartedAt
|
||||
)
|
||||
$body = [ordered]@{ tenant = $Tenant; source_system = $SourceSystem; job = $Job }
|
||||
if ($Run) { $body.run = $Run }
|
||||
$json = $body | ConvertTo-Json -Depth 6 -Compress
|
||||
if ($script:DryRun) {
|
||||
Write-Host $json
|
||||
} else {
|
||||
# Send bytes: Windows PowerShell would otherwise encode a string body as Latin-1,
|
||||
# mangling non-ASCII job names (e.g. en dashes)
|
||||
$params = @{
|
||||
Uri = "$($script:Config.ApiUrl.TrimEnd('/'))/api/v1/ingest"
|
||||
Method = 'Post'
|
||||
Headers = @{ Authorization = "Bearer $($script:Config.ApiKey)" }
|
||||
ContentType = 'application/json; charset=utf-8'
|
||||
Body = [Text.Encoding]::UTF8.GetBytes($json)
|
||||
TimeoutSec = 30
|
||||
}
|
||||
Invoke-RestMethod @params | Out-Null
|
||||
}
|
||||
if ($Run) {
|
||||
$script:RunsSent++
|
||||
if ($Run.status -ne 'running' -and $SentKey) {
|
||||
$script:Sent[$SentKey] = ([DateTimeOffset][datetime]$StartedAt).ToUnixTimeSeconds()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# True when a failed Send-JobReport was the API rejecting the key: no point trying more jobs
|
||||
function Test-AuthRejected($ErrorRecord) {
|
||||
$status = 0
|
||||
try { $status = [int]$ErrorRecord.Exception.Response.StatusCode } catch { }
|
||||
return $status -eq 401
|
||||
}
|
||||
|
||||
<#
|
||||
Saves the sent-runs record (dropping runs older than the lookback window plus a day) and logs
|
||||
the summary line. Doesn't exit: the collector does that with the returned exit code.
|
||||
#>
|
||||
function Complete-Collector {
|
||||
param([int]$JobCount = -1, [int]$Failures = 0)
|
||||
if (-not $script:DryRun) {
|
||||
$cutoff = [DateTimeOffset]::Now.AddHours(-$script:LookbackHours - 24).ToUnixTimeSeconds()
|
||||
$keep = [ordered]@{}
|
||||
foreach ($key in $script:Sent.Keys) {
|
||||
if ($script:Sent[$key] -ge $cutoff) { $keep[$key] = $script:Sent[$key] }
|
||||
}
|
||||
try {
|
||||
$keep | ConvertTo-Json -Compress | Set-Content -Path $script:StatePath -Encoding UTF8
|
||||
} catch {
|
||||
Write-Log "Couldn't save state file: $(Get-ErrorText $_)" 'WARN'
|
||||
}
|
||||
}
|
||||
if ($JobCount -ge 0) {
|
||||
Write-Log ("Reported {0} jobs, {1} runs, {2} failures" -f $JobCount, $script:RunsSent, $Failures)
|
||||
} else {
|
||||
Write-Log ("Reported {0} runs, {1} failures" -f $script:RunsSent, $Failures)
|
||||
}
|
||||
if ($Failures) { return 1 }
|
||||
return 0
|
||||
}
|
||||
|
||||
# ---------- Installing ----------
|
||||
|
||||
<#
|
||||
The part of every collector installer after its own checks: prompt for the API key, write the
|
||||
config to ProgramData\BackupDashboard (locked to SYSTEM and Administrators, since it holds the
|
||||
key), copy the collector script and this module to $InstallDir (Program Files: only admins can
|
||||
modify what runs as SYSTEM), and register a scheduled task running it as SYSTEM.
|
||||
#>
|
||||
function Install-CollectorTask {
|
||||
param(
|
||||
[Parameter(Mandatory)][string]$Name, # 'veeam' -> veeam-collector.json / .log
|
||||
[Parameter(Mandatory)][string]$ScriptPath, # the collector script to install
|
||||
[Parameter(Mandatory)][string]$TaskName,
|
||||
[Parameter(Mandatory)][string]$TaskDescription,
|
||||
[Parameter(Mandatory)][string]$Shell, # powershell.exe or pwsh.exe, full path
|
||||
[Parameter(Mandatory)][string]$ApiUrl,
|
||||
[Parameter(Mandatory)][int]$IntervalMinutes,
|
||||
[Parameter(Mandatory)][string]$InstallDir,
|
||||
[System.Collections.IDictionary]$Settings = @{} # extra config: Tenant, LookbackHours, ...
|
||||
)
|
||||
$dataDir = Join-Path $env:ProgramData 'BackupDashboard'
|
||||
$configPath = Join-Path $dataDir "$Name-collector.json"
|
||||
|
||||
if ($ApiUrl -notmatch '^https://') {
|
||||
Write-Warning 'ApiUrl is not HTTPS: the API key would be sent in clear text.'
|
||||
}
|
||||
|
||||
# Read the key as a SecureString so it isn't echoed to the console or kept in shell history
|
||||
$secure = Read-Host -AsSecureString 'API key (from: python manage.py create-key <name>)'
|
||||
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
|
||||
try { $apiKey = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) }
|
||||
finally { [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr) }
|
||||
if (-not $apiKey.StartsWith('bdk_')) { throw "That doesn't look like a dashboard API key (expected 'bdk_...')." }
|
||||
|
||||
# Config folder: SYSTEM + Administrators only (SIDs so this works on non-English Windows)
|
||||
New-Item -ItemType Directory -Path $dataDir -Force | Out-Null
|
||||
& icacls.exe $dataDir /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' | Out-Null
|
||||
if ($LASTEXITCODE) { throw "icacls failed on $dataDir" }
|
||||
|
||||
$config = [ordered]@{ ApiUrl = $ApiUrl.TrimEnd('/'); ApiKey = $apiKey }
|
||||
foreach ($key in $Settings.Keys) { $config[$key] = $Settings[$key] }
|
||||
$config | ConvertTo-Json -Depth 4 | Set-Content -Path $configPath -Encoding UTF8
|
||||
|
||||
# The collector and this module, side by side (the collector looks for the module next to itself)
|
||||
New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null
|
||||
$installedScript = Join-Path $InstallDir (Split-Path -Leaf $ScriptPath)
|
||||
foreach ($file in $ScriptPath, $PSCommandPath) {
|
||||
$target = Join-Path $InstallDir (Split-Path -Leaf $file)
|
||||
Copy-Item -Path $file -Destination $target -Force
|
||||
Unblock-File -Path $target
|
||||
}
|
||||
|
||||
$action = New-ScheduledTaskAction -Execute $Shell `
|
||||
-Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$installedScript`""
|
||||
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date).AddMinutes(1) `
|
||||
-RepetitionInterval (New-TimeSpan -Minutes $IntervalMinutes)
|
||||
$principal = New-ScheduledTaskPrincipal -UserId 'S-1-5-18' -LogonType ServiceAccount -RunLevel Highest
|
||||
$taskSettings = New-ScheduledTaskSettingsSet -ExecutionTimeLimit (New-TimeSpan -Minutes 30) `
|
||||
-MultipleInstances IgnoreNew -StartWhenAvailable
|
||||
Register-ScheduledTask -TaskName $TaskName -Action $action -Trigger $trigger -Principal $principal `
|
||||
-Settings $taskSettings -Description $TaskDescription -Force | Out-Null
|
||||
|
||||
Write-Host "Installed. Config: $configPath"
|
||||
Write-Host "Task '$TaskName' runs every $IntervalMinutes minutes as SYSTEM, using $Shell."
|
||||
Write-Host ''
|
||||
Write-Host 'Next steps:'
|
||||
Write-Host " 1. Preview what will be sent: & '$Shell' -File '$installedScript' -DryRun"
|
||||
Write-Host " 2. Backfill 30 days of history: & '$Shell' -File '$installedScript' -LookbackHours 720"
|
||||
Write-Host " 3. Check the log: Get-Content '$dataDir\$Name-collector.log' -Tail 20"
|
||||
}
|
||||
|
||||
Export-ModuleMember -Function Write-Log, Get-ErrorText, ConvertTo-IsoTime, Get-FirstValue,
|
||||
Initialize-Collector, Test-RunSent, Send-JobReport, Test-AuthRejected, Complete-Collector,
|
||||
Install-CollectorTask
|
||||
@@ -0,0 +1,102 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Read-only dump of what the MARS agent's cmdlets return, for troubleshooting the collector.
|
||||
|
||||
.DESCRIPTION
|
||||
The collector reads properties of Get-OBJob / Get-OBPolicy / Get-OBSchedule results whose names
|
||||
aren't documented. This prints those objects' real structure (property names, types and values,
|
||||
a few levels deep) so mismatches can be fixed. Nothing is changed. Output is also saved to a
|
||||
text file.
|
||||
|
||||
.EXAMPLE
|
||||
.\Get-MarsCollectorDiagnostics.ps1
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[int]$Jobs = 3,
|
||||
[string]$OutFile = (Join-Path $env:TEMP 'mars-collector-diagnostics.txt')
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$MaxDepth = 6
|
||||
|
||||
if (-not (Get-Command Get-OBJob -ErrorAction SilentlyContinue)) {
|
||||
if (Get-Module -ListAvailable -Name MSOnlineBackup) {
|
||||
Import-Module MSOnlineBackup
|
||||
} else {
|
||||
Import-Module (Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\Modules\MSOnlineBackup')
|
||||
}
|
||||
}
|
||||
|
||||
function Test-Scalar($Value) {
|
||||
return $null -eq $Value -or $Value -is [string] -or $Value -is [ValueType] -or $Value -is [guid]
|
||||
}
|
||||
|
||||
# Recursive "name (type) = value" listing. Collections show their first 3 items.
|
||||
function Show-Object($Value, [string]$Indent = ' ', [int]$Depth = 0) {
|
||||
if (Test-Scalar $Value) { return }
|
||||
if ($Depth -ge $MaxDepth) { "$Indent..."; return }
|
||||
if ($Value -is [Collections.IEnumerable]) {
|
||||
$items = @($Value)
|
||||
"$Indent[$($items.Count) item(s)]"
|
||||
$i = 0
|
||||
foreach ($item in $items | Select-Object -First 3) {
|
||||
"$Indent[$i] ($($item.GetType().FullName))$(if (Test-Scalar $item) { " = $item" })"
|
||||
Show-Object $item "$Indent " ($Depth + 1)
|
||||
$i++
|
||||
}
|
||||
return
|
||||
}
|
||||
foreach ($p in $Value.PSObject.Properties) {
|
||||
try { $v = $p.Value } catch { "$Indent$($p.Name) = <error: $($_.Exception.Message)>"; continue }
|
||||
$type = if ($null -eq $v) { 'null' } else { $v.GetType().FullName }
|
||||
if (Test-Scalar $v) {
|
||||
"$Indent$($p.Name) ($type) = $v"
|
||||
} else {
|
||||
$text = [string]$v
|
||||
if ($text.Length -gt 100) { $text = $text.Substring(0, 97) + '...' }
|
||||
"$Indent$($p.Name) ($type) = $text"
|
||||
Show-Object $v "$Indent " ($Depth + 1)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function Show-Section([string]$Title, [scriptblock]$Get) {
|
||||
"== $Title"
|
||||
try {
|
||||
$result = @(& $Get)
|
||||
if ($result.Count -eq 0 -or ($result.Count -eq 1 -and $null -eq $result[0])) { ' (nothing returned)' }
|
||||
$n = 0
|
||||
foreach ($item in $result) {
|
||||
if ($null -eq $item) { continue }
|
||||
" [$n] ($($item.GetType().FullName))"
|
||||
Show-Object $item ' '
|
||||
$n++
|
||||
}
|
||||
} catch {
|
||||
" error: $($_.Exception.Message)"
|
||||
}
|
||||
''
|
||||
}
|
||||
|
||||
$report = & {
|
||||
$agent = try {
|
||||
(Get-Item (Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\cbengine.exe')).VersionInfo.ProductVersion
|
||||
} catch { 'unknown' }
|
||||
"Generated $(Get-Date -Format o) on $env:COMPUTERNAME, MARS agent $agent, PowerShell $($PSVersionTable.PSVersion)"
|
||||
''
|
||||
Show-Section 'Get-OBJob (in progress)' { Get-OBJob }
|
||||
Show-Section "Get-OBJob -Previous $Jobs" { Get-OBJob -Previous $Jobs }
|
||||
Show-Section 'Get-OBPolicy' { Get-OBPolicy }
|
||||
Show-Section 'Get-OBPolicy | Get-OBSchedule' { Get-OBSchedule -Policy (Get-OBPolicy) }
|
||||
if (Get-Command Get-OBSystemStatePolicy -ErrorAction SilentlyContinue) {
|
||||
Show-Section 'Get-OBSystemStatePolicy | Get-OBSchedule' {
|
||||
$ssPolicy = Get-OBSystemStatePolicy
|
||||
if ($ssPolicy) { Get-OBSchedule -Policy $ssPolicy } else { '(no System State policy configured)' }
|
||||
}
|
||||
}
|
||||
} | Out-String -Width 300
|
||||
|
||||
$report
|
||||
$report | Set-Content -Path $OutFile -Encoding UTF8
|
||||
Write-Host "Saved to $OutFile"
|
||||
@@ -0,0 +1,55 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Installs the MARS (Azure Backup agent) collector as a SYSTEM scheduled task.
|
||||
|
||||
.DESCRIPTION
|
||||
- Copies Send-MarsBackupStatus.ps1 to Program Files (only admins can modify it, which
|
||||
matters because it runs as SYSTEM).
|
||||
- Writes the config to ProgramData\BackupDashboard and restricts that folder to SYSTEM and
|
||||
Administrators, since it contains the API key.
|
||||
- Registers a scheduled task that runs every IntervalMinutes under Windows PowerShell 5.1
|
||||
(powershell.exe), which the agent's MSOnlineBackup module targets.
|
||||
Safe to re-run to change settings or update the script. Can sit alongside the Veeam collector.
|
||||
|
||||
.EXAMPLE
|
||||
.\Install-MarsCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
(prompts for the API key)
|
||||
#>
|
||||
#Requires -RunAsAdministrator
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory)][string]$ApiUrl,
|
||||
[Parameter(Mandatory)][string]$Tenant,
|
||||
# Name shown on the dashboard; defaults to "<server> - Azure Backup"
|
||||
[string]$JobName,
|
||||
[string]$SourceSystem = 'Azure Backup (MARS)',
|
||||
[ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
|
||||
[ValidateRange(1, 720)][int]$LookbackHours = 48,
|
||||
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard')
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$WindowsPowerShell = Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\powershell.exe'
|
||||
|
||||
# Check the agent is here before changing anything
|
||||
$agentModule = Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\Modules\MSOnlineBackup'
|
||||
if (-not (Get-Module -ListAvailable -Name MSOnlineBackup) -and -not (Test-Path $agentModule)) {
|
||||
throw 'The MARS agent (MSOnlineBackup PowerShell module) is not installed on this server.'
|
||||
}
|
||||
|
||||
# Shared installer code: in the repo it's in ..\common, so copy the whole collectors folder
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
throw "BackupDashboard.Collector.psm1 not found. Copy the whole collectors folder, including common."
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
$settings = [ordered]@{ Tenant = $Tenant; SourceSystem = $SourceSystem; LookbackHours = $LookbackHours }
|
||||
if ($JobName) { $settings.JobName = $JobName }
|
||||
Install-CollectorTask -Name 'mars' -ScriptPath (Join-Path $PSScriptRoot 'Send-MarsBackupStatus.ps1') `
|
||||
-TaskName 'Backup Dashboard - MARS collector' `
|
||||
-TaskDescription 'Sends Azure Backup (MARS) results to the Backup Dashboard' `
|
||||
-Shell $WindowsPowerShell -ApiUrl $ApiUrl -IntervalMinutes $IntervalMinutes -InstallDir $InstallDir -Settings $settings
|
||||
@@ -0,0 +1,68 @@
|
||||
# Azure Backup (MARS agent) collector
|
||||
|
||||
A PowerShell script that runs on each server with the Microsoft Azure Recovery Services (MARS)
|
||||
agent and reports its backups to the dashboard. Each server appears as one job,
|
||||
named `<SERVER> - Azure Backup` by default.
|
||||
|
||||
> **Checked against:** MARS agent 2.0.9274, a Files & Folders policy with completed backups only.
|
||||
> Not yet seen on a real agent: a failed backup, a backup in progress, or a System State policy.
|
||||
> Those paths are written to the same object model, but if one looks wrong, run
|
||||
> `Get-MarsCollectorDiagnostics.ps1` while it's in that state and send the output.
|
||||
|
||||
## Install (on the server, as an administrator)
|
||||
|
||||
Runs under Windows PowerShell 5.1 (`powershell.exe`), which the agent's `MSOnlineBackup` module
|
||||
targets and which every supported Windows Server has.
|
||||
|
||||
1. Create a key for this server on the dashboard's **Admin** page, or on the dashboard host:
|
||||
```bash
|
||||
python manage.py create-key mars-acme-fs01
|
||||
```
|
||||
2. Copy the whole `collectors` folder to the server (the installer needs `common`
|
||||
as well as this folder), then from `collectors\mars` run:
|
||||
```powershell
|
||||
.\Install-MarsCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
```
|
||||
Add `-JobName "File Server"` to choose the name shown on the dashboard. It prompts for the
|
||||
key, then:
|
||||
- writes `C:\ProgramData\BackupDashboard\mars-collector.json` (readable only by SYSTEM and Administrators)
|
||||
- copies the script to `C:\Program Files\BackupDashboard\`
|
||||
- registers the scheduled task **Backup Dashboard - MARS collector** (SYSTEM, every 60 minutes)
|
||||
3. Preview the payloads, then backfill 30 days so the history view is populated straight away:
|
||||
```powershell
|
||||
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-MarsBackupStatus.ps1' -DryRun
|
||||
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-MarsBackupStatus.ps1' -LookbackHours 720
|
||||
```
|
||||
|
||||
It can be installed alongside the other collectors: it uses its own config, state, log and task.
|
||||
Re-run the installer to change settings or update the script. It also installs
|
||||
`BackupDashboard.Collector.psm1`, the code shared by all collectors, next to the script.
|
||||
|
||||
## What it sends
|
||||
|
||||
| Dashboard | MARS |
|
||||
|---|---|
|
||||
| Company | `Tenant` from the config |
|
||||
| Job | One per server, ID `mars:<computer name>`, so changing `JobName` renames rather than duplicates it |
|
||||
| Status | Backup job state: Failed → **Failed**. Completed with failed bytes or an error → **Warning**. Aborted/cancelled or unrecognised → **Warning**. In progress → **Running**. Restore jobs are ignored |
|
||||
| Started / finished | Job start / end time. MARS reports these in UTC without marking them as UTC; the collector handles that |
|
||||
| Next scheduled | Next slot from the backup policy schedule, or the System State policy's if that's sooner. Empty (so never Overdue) for schedules that run every 2+ weeks, since which week is next can't be told |
|
||||
| Message | Job and per-datasource errors (non-zero error codes), failed file counts, and the failed-file log path |
|
||||
| Size | Bytes transferred, summed across datasources |
|
||||
|
||||
Each run covers the last 48 hours (`LookbackHours`). Jobs already delivered are recorded in
|
||||
`mars-collector.state.json`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Log: `C:\ProgramData\BackupDashboard\mars-collector.log`. Exit codes: `0` OK, `1` some runs
|
||||
failed to send (retried next run), `2` setup failed (config or MARS module).
|
||||
- Wrong status, sizes or messages: run `.\Get-MarsCollectorDiagnostics.ps1` (read-only). It prints
|
||||
the real structure of the objects the agent's cmdlets return.
|
||||
|
||||
## Testing without MARS
|
||||
|
||||
```powershell
|
||||
. ./tests/MockMars.ps1
|
||||
./Send-MarsBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||
```
|
||||
@@ -0,0 +1,320 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Reports Microsoft Azure Recovery Services (MARS) agent backups to the Backup Dashboard.
|
||||
|
||||
.DESCRIPTION
|
||||
Runs on a server with the MARS agent, normally as a SYSTEM scheduled task every 60 minutes
|
||||
(see Install-MarsCollector.ps1). The server appears on the dashboard as one job. Each run sends:
|
||||
- the next scheduled backup, worked out from the backup (and System State) policy schedules,
|
||||
which drives the dashboard's Overdue flag, and
|
||||
- every backup job from the last LookbackHours that the dashboard hasn't already received in
|
||||
its final state. A backup in progress is sent as Running until it finishes.
|
||||
|
||||
Already-reported jobs are tracked in a state file, and the ingest API is idempotent, so a
|
||||
failed or repeated run is harmless: anything not confirmed is retried next time.
|
||||
|
||||
Written for Windows PowerShell 5.1, which the MARS agent's MSOnlineBackup module targets.
|
||||
|
||||
.PARAMETER ConfigPath
|
||||
JSON config with ApiUrl, ApiKey and Tenant (optional: JobName, SourceSystem, LookbackHours).
|
||||
|
||||
.PARAMETER LookbackHours
|
||||
Overrides the config value. Use 720 once to backfill 30 days of history.
|
||||
|
||||
.PARAMETER DryRun
|
||||
Print the payloads instead of sending them. Doesn't touch the state or log files.
|
||||
|
||||
.EXAMPLE
|
||||
.\Send-MarsBackupStatus.ps1 -DryRun
|
||||
.EXAMPLE
|
||||
.\Send-MarsBackupStatus.ps1 -LookbackHours 720
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$ConfigPath = (Join-Path $env:ProgramData 'BackupDashboard\mars-collector.json'),
|
||||
[int]$LookbackHours = 0,
|
||||
[switch]$DryRun
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# Shared collector code: installed next to this script; in the repo, in ..\common
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
Write-Host "BackupDashboard.Collector.psm1 not found next to $PSCommandPath; re-run the installer."
|
||||
exit 2
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
# MARS keeps a limited job history; this comfortably covers 30 days at 3 backups a day
|
||||
$MaxPreviousJobs = 200
|
||||
|
||||
# ---------- Helpers ----------
|
||||
|
||||
function Import-Mars {
|
||||
if (Get-Command Get-OBJob -ErrorAction SilentlyContinue) { return } # already loaded
|
||||
if (Get-Module -ListAvailable -Name MSOnlineBackup) {
|
||||
Import-Module MSOnlineBackup
|
||||
return
|
||||
}
|
||||
# The agent normally adds its module folder to PSModulePath; fall back to the default install path
|
||||
$agentModule = Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\Modules\MSOnlineBackup'
|
||||
Import-Module $agentModule
|
||||
}
|
||||
|
||||
# ---------- Schedule ----------
|
||||
|
||||
# Next slot strictly after $After for a MARS schedule (days of week + times of day).
|
||||
# Returns $null when there's no schedule, or when it runs every N>1 weeks: we can't tell which
|
||||
# week is next, and a wrong guess would raise a false Overdue alarm.
|
||||
function Get-NextSlot($Schedule, [datetime]$After) {
|
||||
if ($null -eq $Schedule) { return $null }
|
||||
$weeks = Get-FirstValue $Schedule @('ScheduleWeeklyFrequency', 'WeeksInterval')
|
||||
if ($weeks -and [int]$weeks -gt 1) { return $null }
|
||||
|
||||
$days = @(Get-FirstValue $Schedule @('ScheduleRunDays', 'DaysOfWeek') | ForEach-Object { [string]$_ })
|
||||
# TimeSpans (time of day, local) on real agents; accept DateTimes too
|
||||
$times = @(Get-FirstValue $Schedule @('ScheduleRunTimes', 'TimesOfDay') | ForEach-Object {
|
||||
if ($_ -is [timespan]) { $_ } else { ([datetime]$_).TimeOfDay }
|
||||
} | Sort-Object)
|
||||
if ($days.Count -eq 0 -or $times.Count -eq 0) { return $null }
|
||||
|
||||
for ($offset = 0; $offset -le 7; $offset++) {
|
||||
$date = $After.Date.AddDays($offset)
|
||||
if ($days -notcontains [string]$date.DayOfWeek) { continue }
|
||||
foreach ($time in $times) {
|
||||
$candidate = $date + $time
|
||||
if ($candidate -gt $After) { return $candidate }
|
||||
}
|
||||
}
|
||||
return $null
|
||||
}
|
||||
|
||||
function Get-NextRun {
|
||||
$next = $null
|
||||
$policies = @()
|
||||
try { $policies += @(Get-OBPolicy) } catch { Write-Log "Couldn't read the backup policy: $(Get-ErrorText $_)" 'WARN' }
|
||||
# Windows Server can have a separate System State policy with its own schedule
|
||||
if (Get-Command Get-OBSystemStatePolicy -ErrorAction SilentlyContinue) {
|
||||
try { $policies += @(Get-OBSystemStatePolicy) } catch { }
|
||||
}
|
||||
foreach ($policy in $policies) {
|
||||
if ($null -eq $policy) { continue }
|
||||
try {
|
||||
$slot = Get-NextSlot (Get-OBSchedule -Policy $policy) (Get-Date)
|
||||
if ($slot -and (-not $next -or $slot -lt $next)) { $next = $slot }
|
||||
} catch {
|
||||
Write-Log "Couldn't read a policy schedule: $(Get-ErrorText $_)" 'WARN'
|
||||
}
|
||||
}
|
||||
return ConvertTo-IsoTime $next
|
||||
}
|
||||
|
||||
# ---------- Jobs ----------
|
||||
|
||||
# A job's StartTime/EndTime are UTC values, though not necessarily marked as UTC: on a real agent
|
||||
# StartTime matched the "StartTimeUTC" in the status text, while DisplayTime held local time.
|
||||
# Returns $null for "not set" (1601-01-01), e.g. EndTime while a backup is running.
|
||||
function Get-JobTimeUtc($Job, [string]$Field) {
|
||||
$value = Get-FirstValue $Job.JobStatus @($Field)
|
||||
if ($null -eq $value) { return $null }
|
||||
$dt = [datetime]$value
|
||||
if ($dt.Year -le 1900) { return $null }
|
||||
return [datetime]::SpecifyKind($dt, [DateTimeKind]::Utc)
|
||||
}
|
||||
|
||||
# MARS always includes an ErrorInfo; ErrorCode 0 means no error
|
||||
function Test-HasError($ErrorInfo) {
|
||||
if (-not $ErrorInfo) { return $false }
|
||||
$code = Get-FirstValue $ErrorInfo @('ErrorCode')
|
||||
$detailed = Get-FirstValue $ErrorInfo @('DetailedErrorCode')
|
||||
return ($code -and [long]$code -ne 0) -or ($detailed -and [long]$detailed -ne 0)
|
||||
}
|
||||
|
||||
# DatasourceStatus.Datasource is an object whose DataSourceName is e.g. "E:\" or "System State"
|
||||
function Get-DatasourceName($DatasourceStatus) {
|
||||
$ds = Get-FirstValue $DatasourceStatus @('Datasource', 'DataSource')
|
||||
$name = Get-FirstValue $ds @('DataSourceName', 'FriendlyNameDSPS', 'Name')
|
||||
if ($name) { return [string]$name }
|
||||
if ($ds -is [string]) { return $ds }
|
||||
return 'Datasource'
|
||||
}
|
||||
|
||||
function Get-FailedFiles($Job) {
|
||||
$sum = [long]0
|
||||
foreach ($ds in Get-DatasourceStatuses $Job) {
|
||||
$value = Get-FirstValue (Get-FirstValue $ds @('FileProgress')) @('Failed')
|
||||
if ($value) { $sum += [long]$value }
|
||||
}
|
||||
return $sum
|
||||
}
|
||||
|
||||
function Get-JobStatus($Job) {
|
||||
$status = $Job.JobStatus
|
||||
$state = [string](Get-FirstValue $status @('JobState', 'State'))
|
||||
if (-not (Get-JobTimeUtc $Job 'EndTime') -and $state -notmatch 'Fail|Abort|Cancel|Complete|Success|Succeed|Warn') {
|
||||
return 'running'
|
||||
}
|
||||
if ($state -match 'Fail') { return 'failed' }
|
||||
if ($state -match 'Warn|Abort|Cancel|Stop') { return 'warning' }
|
||||
if ($state -match 'Complete|Success|Succeed') {
|
||||
# MARS can report Completed while individual files failed or an error was recorded
|
||||
if ((Get-FailedBytes $Job) -gt 0 -or (Get-FailedFiles $Job) -gt 0 -or
|
||||
(Test-HasError (Get-FirstValue $status @('ErrorInfo')))) { return 'warning' }
|
||||
return 'success'
|
||||
}
|
||||
return 'warning' # finished in a state we don't recognise: surface it rather than hide it
|
||||
}
|
||||
|
||||
function Get-DatasourceStatuses($Job) {
|
||||
return @(Get-FirstValue $Job.JobStatus @('DatasourceStatus', 'DataSourceStatus') | Where-Object { $_ })
|
||||
}
|
||||
|
||||
function Get-ByteTotal($Job, [string]$Field) {
|
||||
$sum = [long]0
|
||||
foreach ($ds in Get-DatasourceStatuses $Job) {
|
||||
$progress = Get-FirstValue $ds @('ByteProgress')
|
||||
$value = Get-FirstValue $progress @($Field)
|
||||
if ($value) { $sum += [long]$value }
|
||||
}
|
||||
return $sum
|
||||
}
|
||||
|
||||
function Get-FailedBytes($Job) { return Get-ByteTotal $Job 'Failed' }
|
||||
|
||||
function Format-ErrorInfo($ErrorInfo) {
|
||||
if (-not (Test-HasError $ErrorInfo)) { return $null }
|
||||
$text = Get-FirstValue $ErrorInfo @('ErrorMessage', 'Message', 'Description')
|
||||
if ($text) { return ([string]$text).Trim() }
|
||||
$code = Get-FirstValue $ErrorInfo @('ErrorCode', 'DetailedErrorCode')
|
||||
$params = @(Get-FirstValue $ErrorInfo @('ErrorParamList') | Where-Object { $_ } | ForEach-Object {
|
||||
$name = Get-FirstValue $_ @('Name')
|
||||
$value = Get-FirstValue $_ @('Value')
|
||||
if ($name) { "$name=$value" } else { [string]$_ }
|
||||
} | Where-Object { $_ })
|
||||
$line = if ($code) { "Error code $code" } else { [string]$ErrorInfo }
|
||||
if ($params.Count) { $line += " ($($params -join ', '))" }
|
||||
return $line
|
||||
}
|
||||
|
||||
function Get-RunMessage($Job, [string]$Status) {
|
||||
$lines = New-Object 'System.Collections.Generic.List[string]'
|
||||
$state = [string](Get-FirstValue $Job.JobStatus @('JobState', 'State'))
|
||||
$jobError = Format-ErrorInfo (Get-FirstValue $Job.JobStatus @('ErrorInfo'))
|
||||
if ($jobError) { $lines.Add($jobError) }
|
||||
|
||||
foreach ($ds in Get-DatasourceStatuses $Job) {
|
||||
$name = Get-DatasourceName $ds
|
||||
$dsError = Format-ErrorInfo (Get-FirstValue $ds @('ErrorInfo'))
|
||||
$failedFiles = Get-FirstValue (Get-FirstValue $ds @('FileProgress')) @('Failed')
|
||||
$failedBytes = Get-FirstValue (Get-FirstValue $ds @('ByteProgress')) @('Failed')
|
||||
if ($dsError) { $lines.Add("${name}: $dsError") }
|
||||
if ($failedFiles -and [long]$failedFiles -gt 0) { $lines.Add("${name}: $failedFiles file(s) failed to back up") }
|
||||
elseif ($failedBytes -and [long]$failedBytes -gt 0) { $lines.Add("${name}: $failedBytes bytes failed to back up") }
|
||||
}
|
||||
$failedLog = Get-FirstValue $Job.JobStatus @('FailedFileLog')
|
||||
if ($failedLog -and $lines.Count) { $lines.Add("Failed file log: $failedLog") }
|
||||
|
||||
if ($lines.Count -eq 0) {
|
||||
switch ($Status) {
|
||||
'success' { return 'Backup completed successfully.' }
|
||||
'running' { return '' }
|
||||
default { return "Backup finished with state '$state'." }
|
||||
}
|
||||
}
|
||||
if ($state -and $Status -ne 'success') { $lines.Insert(0, "Job state: $state") }
|
||||
return ($lines -join "`n")
|
||||
}
|
||||
|
||||
function ConvertTo-Run($Job) {
|
||||
$status = Get-JobStatus $Job
|
||||
$datasources = @(Get-DatasourceStatuses $Job | ForEach-Object { Get-DatasourceName $_ })
|
||||
|
||||
$details = [ordered]@{ server = $env:COMPUTERNAME }
|
||||
if ($datasources.Count) { $details.datasources = $datasources }
|
||||
$agentState = Get-FirstValue $Job.JobStatus @('JobState', 'State')
|
||||
if ($agentState) { $details.mars_state = [string]$agentState }
|
||||
|
||||
$run = [ordered]@{
|
||||
external_id = [string]$Job.JobId
|
||||
status = $status
|
||||
started_at = ConvertTo-IsoTime (Get-JobTimeUtc $Job 'StartTime')
|
||||
message = Get-RunMessage $Job $status
|
||||
details = $details
|
||||
}
|
||||
if ($status -ne 'running') {
|
||||
$finished = ConvertTo-IsoTime (Get-JobTimeUtc $Job 'EndTime')
|
||||
if ($finished) { $run.finished_at = $finished } # otherwise the API uses receipt time
|
||||
$transferred = Get-ByteTotal $Job 'Progress'
|
||||
if ($transferred -gt 0) { $run.size_bytes = $transferred }
|
||||
$failedBytes = Get-FailedBytes $Job
|
||||
if ($failedBytes -gt 0) { $details.failed_bytes = $failedBytes }
|
||||
$failedFiles = Get-FailedFiles $Job
|
||||
if ($failedFiles -gt 0) { $details.failed_files = $failedFiles }
|
||||
}
|
||||
return $run
|
||||
}
|
||||
|
||||
function Test-IsBackupJob($Job) {
|
||||
# Get-OBJob also lists restores (JobType Recovery); only backups belong on the dashboard
|
||||
$type = [string](Get-FirstValue $Job @('JobType', 'Type'))
|
||||
return (-not $type) -or $type -match 'Backup'
|
||||
}
|
||||
|
||||
# ---------- Main ----------
|
||||
|
||||
try {
|
||||
$Config = Initialize-Collector -Name 'mars' -ConfigPath $ConfigPath -LookbackHours $LookbackHours -DryRun:$DryRun
|
||||
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Azure Backup (MARS)' }
|
||||
$JobName = if ($Config.JobName) { [string]$Config.JobName } else { "$env:COMPUTERNAME - Azure Backup" }
|
||||
$LookbackHours = $Config.LookbackHours
|
||||
|
||||
Import-Mars
|
||||
|
||||
$sinceUtc = (Get-Date).ToUniversalTime().AddHours(-$LookbackHours)
|
||||
# Get-OBJob with no arguments lists jobs in progress; -Previous lists finished ones
|
||||
$jobs = @(@(Get-OBJob) + @(Get-OBJob -Previous $MaxPreviousJobs) | Where-Object {
|
||||
$_ -and (Test-IsBackupJob $_)
|
||||
})
|
||||
$byId = [ordered]@{}
|
||||
foreach ($job in $jobs) {
|
||||
$start = Get-JobTimeUtc $job 'StartTime'
|
||||
if (-not $start -or $start -lt $sinceUtc) { continue }
|
||||
$byId[[string]$job.JobId] = $job # a job in both lists: the finished entry (later) wins
|
||||
}
|
||||
$pending = @($byId.Values |
|
||||
Where-Object { -not (Test-RunSent ([string]$_.JobId)) } |
|
||||
Sort-Object { Get-JobTimeUtc $_ 'StartTime' })
|
||||
|
||||
$jobInfo = [ordered]@{
|
||||
# Stable per server, so renaming the dashboard job via JobName doesn't create a new one
|
||||
external_id = "mars:$($env:COMPUTERNAME.ToLowerInvariant())"
|
||||
name = $JobName
|
||||
# Always sent (null when unscheduled) so the dashboard's value tracks the agent's
|
||||
next_run_at = Get-NextRun
|
||||
}
|
||||
} catch {
|
||||
Write-Log "Collector setup failed: $(Get-ErrorText $_)" 'ERROR'
|
||||
exit 2
|
||||
}
|
||||
|
||||
$failures = 0
|
||||
try {
|
||||
if ($pending.Count -eq 0) {
|
||||
# Keep next_run_at fresh even when nothing new ran
|
||||
Send-JobReport -Tenant $Config.Tenant -SourceSystem $SourceSystem -Job $jobInfo
|
||||
}
|
||||
foreach ($job in $pending) {
|
||||
$run = ConvertTo-Run $job
|
||||
Send-JobReport -Tenant $Config.Tenant -SourceSystem $SourceSystem -Job $jobInfo -Run $run -SentKey $run.external_id -StartedAt ((Get-JobTimeUtc $job 'StartTime'))
|
||||
}
|
||||
} catch {
|
||||
$failures++
|
||||
Write-Log "Failed to report: $(Get-ErrorText $_)" 'ERROR'
|
||||
}
|
||||
|
||||
$exitCode = Complete-Collector -Failures $failures
|
||||
if ($exitCode) { exit $exitCode }
|
||||
@@ -0,0 +1,79 @@
|
||||
# Stand-ins for the MARS agent cmdlets. Object shapes follow Get-MarsCollectorDiagnostics.ps1
|
||||
# output from a real agent (2.0.9274): job times are UTC values with no UTC marking, "not set" is
|
||||
# 1601-01-01, ErrorInfo is always present (ErrorCode 0 = no error), Datasource is an object, and
|
||||
# schedule times are TimeSpans.
|
||||
#
|
||||
# . ./tests/MockMars.ps1
|
||||
# ./Send-MarsBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||
|
||||
# Fixed to the top of the hour, and IDs derived from times, so repeated runs see the same jobs
|
||||
# MOCK_NOW (ISO 8601, set by the test suite) pins the time everything is relative to
|
||||
$now = if ($env:MOCK_NOW) { [datetime]$env:MOCK_NOW } else { Get-Date -Minute 0 -Second 0 -Millisecond 0 }
|
||||
$notSet = [datetime]::new(1601, 1, 1)
|
||||
function New-StableGuid([string]$Seed) {
|
||||
[guid]::new([Security.Cryptography.MD5]::Create().ComputeHash([Text.Encoding]::UTF8.GetBytes($Seed)))
|
||||
}
|
||||
|
||||
# Local time -> the UTC value MARS reports, with Kind Unspecified like the real objects
|
||||
function ConvertTo-MarsTime([datetime]$Local) {
|
||||
[datetime]::SpecifyKind($Local.ToUniversalTime(), [DateTimeKind]::Unspecified)
|
||||
}
|
||||
|
||||
function New-MockErrorInfo([int]$Code = 0, $Params = @()) {
|
||||
[pscustomobject]@{ DetailedErrorCode = 0; DetailedErrorSource = $null; ErrorCode = $Code; ErrorParamList = $Params }
|
||||
}
|
||||
|
||||
function New-MockDatasource($Name, [uint64]$Done, [uint64]$FailedBytes = 0, [uint64]$FailedFiles = 0, $ErrorInfo = (New-MockErrorInfo)) {
|
||||
[pscustomobject]@{
|
||||
ByteProgress = [pscustomobject]@{ Changed = $Done; Failed = $FailedBytes; Progress = $Done; Total = 2TB }
|
||||
Datasource = [pscustomobject]@{ DataSourceName = $Name; DataSourceId = 6310158806455554328 }
|
||||
ErrorInfo = $ErrorInfo
|
||||
FileProgress = [pscustomobject]@{ Changed = 100; CurrentFile = ''; Failed = $FailedFiles; Progress = 100; Total = 100 }
|
||||
JobState = 'Completed'
|
||||
}
|
||||
}
|
||||
|
||||
function New-MockJob($Type, [datetime]$Start, $End, $State, $Datasources = @(), $ErrorInfo = (New-MockErrorInfo), $FailedFileLog = '') {
|
||||
$endValue = if ($End) { ConvertTo-MarsTime $End } else { $notSet }
|
||||
[pscustomobject]@{
|
||||
IsRetried = $false
|
||||
JobId = New-StableGuid "$Type|$($Start.Ticks)"
|
||||
JobType = $Type
|
||||
DisplayTime = $Start # the only local time on the object
|
||||
JobStatus = [pscustomobject]@{
|
||||
DatasourceStatus = $Datasources
|
||||
EndTime = $endValue
|
||||
ErrorInfo = $ErrorInfo
|
||||
FailedFileLog = $FailedFileLog
|
||||
JobState = $State
|
||||
StartTime = ConvertTo-MarsTime $Start
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$global:MockMarsRunning = @(
|
||||
(New-MockJob 'Backup' $now.AddMinutes(-10) $null 'InProgress' @((New-MockDatasource 'E:\' 2GB)))
|
||||
)
|
||||
$global:MockMarsPrevious = @(
|
||||
(New-MockJob 'Backup' $now.AddHours(-12) $now.AddHours(-11.5) 'Completed' @((New-MockDatasource 'E:\' 523407872))),
|
||||
# Completed, but some files failed
|
||||
(New-MockJob 'Backup' $now.AddHours(-36) $now.AddHours(-35) 'Completed' @(
|
||||
(New-MockDatasource 'E:\' 3GB 40MB 12)) -FailedFileLog 'C:\Program Files\Microsoft Azure Recovery Services Agent\Temp\Errors.log'),
|
||||
(New-MockJob 'Backup' $now.AddHours(-30) $now.AddHours(-29.5) 'Failed' @() `
|
||||
(New-MockErrorInfo 100050 @([pscustomobject]@{ Name = 'Volume'; Value = 'E:' }))),
|
||||
(New-MockJob 'Recovery' $now.AddHours(-5) $now.AddHours(-4) 'Completed'), # a restore: must be ignored
|
||||
(New-MockJob 'Backup' $now.AddDays(-5) $now.AddDays(-5).AddHours(1) 'Completed') # outside 48h lookback
|
||||
)
|
||||
|
||||
# Every day at 21:00 local, as on the real server
|
||||
$global:MockMarsSchedule = [pscustomobject]@{
|
||||
ScheduleRunDays = [DayOfWeek[]](0..6)
|
||||
ScheduleRunTimes = [timespan[]]@([timespan]::FromHours(21))
|
||||
ScheduleWeeklyFrequency = 1
|
||||
TypeOfBackupSchedule = 'Daily'
|
||||
}
|
||||
|
||||
function Get-OBJob { param([int]$Previous) if ($PSBoundParameters.ContainsKey('Previous')) { $global:MockMarsPrevious } else { $global:MockMarsRunning } }
|
||||
function Get-OBPolicy { [pscustomobject]@{ PolicyName = 'policy'; BackupSchedule = $global:MockMarsSchedule } }
|
||||
function Get-OBSchedule { param($Policy) $global:MockMarsSchedule }
|
||||
function Get-OBSystemStatePolicy { $null } # none configured, as on the real server
|
||||
@@ -0,0 +1,139 @@
|
||||
# The collector installers, with stand-ins for the Windows-only commands they call (icacls, Task
|
||||
# Scheduler, Unblock-File, the API key prompt). Checks what each would write and register.
|
||||
|
||||
BeforeAll {
|
||||
Import-Module (Join-Path $PSScriptRoot 'TestHelpers.psm1') -Force
|
||||
$collectorsDir = Split-Path -Parent $PSScriptRoot
|
||||
$shell = if ($env:COLLECTOR_TEST_SHELL) { $env:COLLECTOR_TEST_SHELL } else { (Get-Process -Id $PID).Path }
|
||||
$pwshPath = (Get-Process -Id $PID).Path # stands in for an installed PowerShell 7
|
||||
|
||||
# Runs one installer in a fresh process and sandbox; returns what it wrote and registered
|
||||
function Invoke-Installer {
|
||||
param([string]$Collector, [string]$Arguments, [switch]$NoProductModule)
|
||||
$sandbox = New-TestDir 'installer'
|
||||
$src = Join-Path $sandbox 'collectors'
|
||||
Copy-Item -Path $collectorsDir -Destination $src -Recurse
|
||||
$installer = (Get-ChildItem (Join-Path $src $Collector) -Filter 'Install-*.ps1').FullName
|
||||
# The test runner isn't elevated
|
||||
(Get-Content $installer) -notmatch '^#Requires -RunAsAdministrator' | Set-Content $installer
|
||||
foreach ($d in 'ProgramData', 'ProgramFiles', 'SystemRoot') { New-Item -ItemType Directory -Path (Join-Path $sandbox $d) | Out-Null }
|
||||
$record = Join-Path $sandbox 'record.txt'
|
||||
|
||||
$stubs = @'
|
||||
function global:Record($what) { Add-Content $env:REC_FILE $what }
|
||||
function global:Read-Host { param([switch]$AsSecureString, $Prompt) ConvertTo-SecureString 'bdk_testkey' -AsPlainText -Force }
|
||||
function global:icacls.exe { Record "icacls $($args -join ' ')"; $global:LASTEXITCODE = 0 }
|
||||
function global:Unblock-File { param($Path) Record "unblock $(Split-Path -Leaf $Path)" }
|
||||
function global:Get-Module { param([switch]$ListAvailable, $Name) if (-not $env:STUB_NO_MODULE) { [pscustomobject]@{ Name = $Name } } }
|
||||
function global:New-ScheduledTaskAction { param($Execute, $Argument) Record "exe $Execute"; Record "args $Argument" }
|
||||
function global:New-ScheduledTaskTrigger { param([switch]$Once, $At, $RepetitionInterval) Record "every $([int]$RepetitionInterval.TotalMinutes)" }
|
||||
function global:New-ScheduledTaskPrincipal { param($UserId, $LogonType, $RunLevel) Record "principal $UserId $LogonType $RunLevel" }
|
||||
function global:New-ScheduledTaskSettingsSet { param($ExecutionTimeLimit, $MultipleInstances, [switch]$StartWhenAvailable) Record "settings $MultipleInstances" }
|
||||
function global:Register-ScheduledTask { param($TaskName, $Action, $Trigger, $Principal, $Settings, $Description, [switch]$Force) Record "task $TaskName" }
|
||||
'@
|
||||
$env:REC_FILE = $record
|
||||
$env:STUB_NO_MODULE = if ($NoProductModule) { '1' } else { '' }
|
||||
$saved = @{ ProgramData = $env:ProgramData; ProgramFiles = $env:ProgramFiles; SystemRoot = $env:SystemRoot }
|
||||
try {
|
||||
foreach ($d in $saved.Keys) { Set-Item "env:$d" (Join-Path $sandbox $d) }
|
||||
Push-Location (Split-Path $installer)
|
||||
$output = @(& $shell -NoProfile -NonInteractive -Command "$stubs; & '$installer' $Arguments" 2>&1 | ForEach-Object { "$_" })
|
||||
} finally {
|
||||
Pop-Location
|
||||
foreach ($d in $saved.Keys) { Set-Item "env:$d" $saved[$d] }
|
||||
}
|
||||
|
||||
$configFile = Get-ChildItem (Join-Path (Join-Path $sandbox 'ProgramData') 'BackupDashboard') -Filter '*-collector.json' -ErrorAction SilentlyContinue
|
||||
$installDir = Join-Path (Join-Path $sandbox 'ProgramFiles') 'BackupDashboard'
|
||||
return [pscustomobject]@{
|
||||
Output = $output -join "`n"
|
||||
Config = if ($configFile) { Get-Content $configFile.FullName -Raw | ConvertFrom-Json } else { $null }
|
||||
Files = @(Get-ChildItem $installDir -ErrorAction SilentlyContinue | ForEach-Object Name | Sort-Object)
|
||||
Record = @(if (Test-Path $record) { Get-Content $record })
|
||||
Sandbox = $sandbox
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Collector installers' {
|
||||
Context 'Veeam' {
|
||||
BeforeAll { $i = Invoke-Installer veeam "-ApiUrl https://dash.example.com/ -Tenant 'Acme Corp' -PwshPath '$pwshPath'" }
|
||||
|
||||
It 'writes the config, including the API key from the prompt' {
|
||||
$i.Config.ApiUrl | Should -Be 'https://dash.example.com'
|
||||
$i.Config.ApiKey | Should -Be 'bdk_testkey'
|
||||
$i.Config.Tenant | Should -Be 'Acme Corp'
|
||||
$i.Config.SourceSystem | Should -Be 'Veeam B&R'
|
||||
$i.Config.LookbackHours | Should -Be 48
|
||||
}
|
||||
It 'locks the config folder to SYSTEM and Administrators' {
|
||||
$i.Record | Where-Object { $_ -like 'icacls*' } | Should -BeLike '*/inheritance:r /grant:r `*S-1-5-18:(OI)(CI)F `*S-1-5-32-544:(OI)(CI)F'
|
||||
}
|
||||
It 'installs the collector and the shared module side by side, unblocked' {
|
||||
$i.Files | Should -Be @('BackupDashboard.Collector.psm1', 'Send-VeeamBackupStatus.ps1')
|
||||
$i.Record | Should -Contain 'unblock BackupDashboard.Collector.psm1'
|
||||
$i.Record | Should -Contain 'unblock Send-VeeamBackupStatus.ps1'
|
||||
}
|
||||
It 'registers an hourly SYSTEM task running the collector under PowerShell 7' {
|
||||
$i.Record | Should -Contain "exe $pwshPath"
|
||||
$i.Record | Where-Object { $_ -like 'args*' } | Should -BeLike '*-File "*Send-VeeamBackupStatus.ps1"'
|
||||
$i.Record | Should -Contain 'every 60'
|
||||
$i.Record | Should -Contain 'principal S-1-5-18 ServiceAccount Highest'
|
||||
$i.Record | Should -Contain 'settings IgnoreNew'
|
||||
$i.Record | Should -Contain 'task Backup Dashboard - Veeam collector'
|
||||
}
|
||||
It 'refuses a PowerShell 7 path that does not exist, before changing anything' {
|
||||
$bad = Invoke-Installer veeam "-ApiUrl https://dash.example.com -Tenant Acme -PwshPath '/no/such/pwsh.exe'"
|
||||
$bad.Output | Should -BeLike '*PowerShell 7 not found*'
|
||||
$bad.Config | Should -BeNullOrEmpty
|
||||
$bad.Files | Should -BeNullOrEmpty
|
||||
}
|
||||
}
|
||||
|
||||
Context 'MARS' {
|
||||
It 'writes JobName, runs hourly under Windows PowerShell, and honours -IntervalMinutes' {
|
||||
$i = Invoke-Installer mars "-ApiUrl https://dash.example.com -Tenant 'Acme Corp' -JobName 'File Server' -IntervalMinutes 30"
|
||||
$i.Config.JobName | Should -Be 'File Server'
|
||||
$i.Config.SourceSystem | Should -Be 'Azure Backup (MARS)'
|
||||
$i.Files | Should -Be @('BackupDashboard.Collector.psm1', 'Send-MarsBackupStatus.ps1')
|
||||
$i.Record | Where-Object { $_ -like 'exe*' } | Should -BeLike '*WindowsPowerShell*powershell.exe'
|
||||
$i.Record | Should -Contain 'every 30'
|
||||
$i.Record | Should -Contain 'task Backup Dashboard - MARS collector'
|
||||
}
|
||||
It 'refuses to install where the MARS agent is missing, before changing anything' {
|
||||
$i = Invoke-Installer mars "-ApiUrl https://dash.example.com -Tenant Acme" -NoProductModule
|
||||
$i.Output | Should -BeLike '*MARS agent*not installed*'
|
||||
$i.Config | Should -BeNullOrEmpty
|
||||
}
|
||||
}
|
||||
|
||||
Context 'Windows Server Backup' {
|
||||
It 'warns when the dashboard URL is not HTTPS' {
|
||||
$i = Invoke-Installer wsb "-ApiUrl http://dash.example.com -Tenant 'Acme Corp'"
|
||||
$i.Output | Should -BeLike '*not HTTPS*'
|
||||
$i.Files | Should -Be @('BackupDashboard.Collector.psm1', 'Send-WsbBackupStatus.ps1')
|
||||
$i.Record | Should -Contain 'task Backup Dashboard - WSB collector'
|
||||
}
|
||||
It 'refuses to install without the Windows Server Backup feature' {
|
||||
$i = Invoke-Installer wsb "-ApiUrl https://dash.example.com -Tenant Acme" -NoProductModule
|
||||
# (PowerShell wraps long error messages, so match a short part)
|
||||
$i.Output | Should -BeLike '*WindowsServerBackup PowerShell module is not available*'
|
||||
$i.Config | Should -BeNullOrEmpty
|
||||
}
|
||||
}
|
||||
|
||||
Context 'VB365' {
|
||||
It 'writes the organization-to-company map, with no Tenant unless given' {
|
||||
$i = Invoke-Installer vb365 "-ApiUrl https://dash.example.com -OrganizationTenants @{ 'acme.onmicrosoft.com' = 'Acme Corp'; 'blue.onmicrosoft.com' = 'Bluewater' }"
|
||||
$i.Config.OrganizationTenants.'acme.onmicrosoft.com' | Should -Be 'Acme Corp'
|
||||
$i.Config.OrganizationTenants.'blue.onmicrosoft.com' | Should -Be 'Bluewater'
|
||||
$i.Config.PSObject.Properties.Name | Should -Not -Contain 'Tenant'
|
||||
$i.Record | Where-Object { $_ -like 'exe*' } | Should -BeLike '*powershell.exe'
|
||||
}
|
||||
It 'uses PowerShell 7 when given -PwshPath' {
|
||||
$i = Invoke-Installer vb365 "-ApiUrl https://dash.example.com -Tenant 'Solo Co' -PwshPath '$pwshPath'"
|
||||
$i.Config.Tenant | Should -Be 'Solo Co'
|
||||
$i.Record | Should -Contain "exe $pwshPath"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Runs the collector test suite (Pester): the shared module, each collector against its mock,
|
||||
and the installers.
|
||||
|
||||
.DESCRIPTION
|
||||
Needs PowerShell 7.2+ and Pester 5.5+:
|
||||
Install-Module Pester -MinimumVersion 5.5 -Scope CurrentUser
|
||||
|
||||
Each collector runs in its own process, by default under the same PowerShell running the
|
||||
tests. On Windows, use -CollectorShell to run them under Windows PowerShell 5.1 instead, as
|
||||
the MARS, WSB and VB365 servers do.
|
||||
|
||||
Exits non-zero if any test fails, so it can gate a commit or CI job.
|
||||
|
||||
.EXAMPLE
|
||||
./collectors/tests/Invoke-CollectorTests.ps1
|
||||
.EXAMPLE
|
||||
./collectors/tests/Invoke-CollectorTests.ps1 -CollectorShell powershell.exe
|
||||
.EXAMPLE
|
||||
./collectors/tests/Invoke-CollectorTests.ps1 -Path ./collectors/tests/Wsb.Tests.ps1
|
||||
#>
|
||||
#Requires -Version 7.2
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
# powershell.exe / pwsh / a full path: the shell the collectors and installers run under
|
||||
[string]$CollectorShell,
|
||||
# Test files or folders to run; defaults to every *.Tests.ps1 here
|
||||
[string[]]$Path = @($PSScriptRoot),
|
||||
[ValidateSet('None', 'Normal', 'Detailed', 'Diagnostic')][string]$Output = 'Detailed'
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
$pester = Get-Module -ListAvailable -Name Pester | Where-Object Version -GE '5.5' |
|
||||
Sort-Object Version -Descending | Select-Object -First 1
|
||||
if (-not $pester) {
|
||||
throw 'Pester 5.5 or newer is needed: Install-Module Pester -MinimumVersion 5.5 -Scope CurrentUser'
|
||||
}
|
||||
Import-Module $pester
|
||||
|
||||
if ($CollectorShell) {
|
||||
$resolved = Get-Command $CollectorShell -ErrorAction SilentlyContinue
|
||||
if (-not $resolved) { throw "Shell not found: $CollectorShell" }
|
||||
$env:COLLECTOR_TEST_SHELL = $resolved.Source
|
||||
}
|
||||
$shellName = if ($env:COLLECTOR_TEST_SHELL) { $env:COLLECTOR_TEST_SHELL } else { "this PowerShell ($($PSVersionTable.PSVersion))" }
|
||||
Write-Host "Collectors run under: $shellName"
|
||||
|
||||
# All temp folders for this run, removed at the end
|
||||
$env:COLLECTOR_TEST_TEMP = Join-Path ([IO.Path]::GetTempPath()) ("backup-dashboard-tests-" + [guid]::NewGuid().ToString('N').Substring(0, 8))
|
||||
try {
|
||||
$config = New-PesterConfiguration
|
||||
$config.Run.Path = $Path
|
||||
$config.Run.PassThru = $true
|
||||
$config.Output.Verbosity = $Output
|
||||
$result = Invoke-Pester -Configuration $config
|
||||
} finally {
|
||||
Remove-Item -Recurse -Force $env:COLLECTOR_TEST_TEMP -ErrorAction SilentlyContinue
|
||||
Remove-Item env:COLLECTOR_TEST_TEMP
|
||||
}
|
||||
exit $(if ($result.Result -eq 'Passed') { 0 } else { 1 })
|
||||
@@ -0,0 +1,95 @@
|
||||
# Azure Backup (MARS agent) collector, against collectors/mars/tests/MockMars.ps1 (shapes from a
|
||||
# real agent, 2.0.9274: job times are UTC values with no UTC marking, ErrorInfo is always present).
|
||||
|
||||
BeforeAll {
|
||||
Import-Module (Join-Path $PSScriptRoot 'TestHelpers.psm1') -Force
|
||||
$now = Set-MockNow
|
||||
$config = @{ ApiUrl = 'http://localhost:1'; ApiKey = 'bdk_x'; Tenant = 'Acme' }
|
||||
# MARS reports job times in UTC; the API should get them with +00:00
|
||||
function Utc([datetime]$Local) { ConvertTo-ApiTime $Local.ToUniversalTime() }
|
||||
}
|
||||
|
||||
Describe 'MARS collector' {
|
||||
Context 'what it reports (dry run)' {
|
||||
BeforeAll {
|
||||
$r = Invoke-Collector mars -Config $config -DryRun
|
||||
$runs = @($r.Reports | Where-Object run | ForEach-Object run)
|
||||
$byStart = @{}
|
||||
foreach ($run in $runs) { $byStart[$run.started_at] = $run }
|
||||
}
|
||||
|
||||
It 'runs cleanly' {
|
||||
$r.ExitCode | Should -Be 0
|
||||
$r.Log[-1] | Should -Be '[INFO] Reported 4 runs, 0 failures'
|
||||
}
|
||||
It 'reports the server as one job, named after it' {
|
||||
$r.Reports.job.external_id | Select-Object -Unique | Should -Be 'mars:srv01'
|
||||
$r.Reports.job.name | Select-Object -Unique | Should -Be 'SRV01 - Azure Backup'
|
||||
$r.Reports.source_system | Select-Object -Unique | Should -Be 'Azure Backup (MARS)'
|
||||
}
|
||||
It 'computes the next run from the schedule (daily at 21:00)' {
|
||||
$next = (Get-Date).Date.AddHours(21)
|
||||
if ((Get-Date) -ge $next) { $next = $next.AddDays(1) }
|
||||
$r.Reports[0].job.next_run_at | Should -Be (ConvertTo-ApiTime $next)
|
||||
}
|
||||
It 'treats job times as UTC (MARS gives UTC values without marking them)' {
|
||||
$byStart.Keys | Should -Contain (Utc $now.AddHours(-12))
|
||||
$byStart[(Utc $now.AddHours(-12))].finished_at | Should -Be (Utc $now.AddHours(-11.5))
|
||||
}
|
||||
It 'reports a completed backup as success, despite its (always present) ErrorInfo with code 0' {
|
||||
$run = $byStart[(Utc $now.AddHours(-12))]
|
||||
$run.status | Should -Be 'success'
|
||||
$run.message | Should -Be 'Backup completed successfully.'
|
||||
$run.size_bytes | Should -Be 523407872
|
||||
$run.details.datasources | Should -Be @('E:\')
|
||||
}
|
||||
It 'reports a completed backup with failed files as a warning, with the count and log' {
|
||||
$run = $byStart[(Utc $now.AddHours(-36))]
|
||||
$run.status | Should -Be 'warning'
|
||||
$run.message | Should -BeLike '*E:\: 12 file(s) failed to back up*'
|
||||
$run.message | Should -BeLike '*Failed file log: C:\Program Files\Microsoft Azure Recovery Services Agent\Temp\Errors.log*'
|
||||
$run.details.failed_files | Should -Be 12
|
||||
$run.details.failed_bytes | Should -Be 40MB
|
||||
}
|
||||
It 'reports a failed backup with its error code and parameters' {
|
||||
$run = $byStart[(Utc $now.AddHours(-30))]
|
||||
$run.status | Should -Be 'failed'
|
||||
$run.message | Should -BeLike '*Error code 100050 (Volume=E:)*'
|
||||
}
|
||||
It 'reports a backup in progress as running, without a finish time' {
|
||||
$run = $byStart[(Utc $now.AddMinutes(-10))]
|
||||
$run.status | Should -Be 'running'
|
||||
$run.PSObject.Properties.Name | Should -Not -Contain 'finished_at'
|
||||
}
|
||||
It 'ignores restores and backups outside the lookback window' {
|
||||
$runs.Count | Should -Be 4
|
||||
}
|
||||
It 'uses JobName from the config when set' {
|
||||
$named = Invoke-Collector mars -Config ($config + @{ JobName = 'File Server' }) -DryRun
|
||||
$named.Reports.job.name | Select-Object -Unique | Should -Be 'File Server'
|
||||
$named.Reports.job.external_id | Select-Object -Unique | Should -Be 'mars:srv01'
|
||||
}
|
||||
}
|
||||
|
||||
Context 'sending to the API' {
|
||||
BeforeAll {
|
||||
$stub = Start-ApiStub
|
||||
$live = $config.Clone(); $live.ApiUrl = $stub.Url
|
||||
$first = Invoke-Collector mars -Config $live
|
||||
$firstRequests = Get-StubRequests $stub
|
||||
$second = Invoke-Collector mars -Config $live -WorkDir $first.WorkDir
|
||||
$secondRequests = Get-StubRequests $stub
|
||||
}
|
||||
AfterAll { Stop-ApiStub $stub }
|
||||
|
||||
It 'sends every run on the first run' {
|
||||
$first.ExitCode | Should -Be 0
|
||||
$firstRequests.Count | Should -Be 4
|
||||
}
|
||||
It 'on the next run, resends only the backup in progress' {
|
||||
$first.State.Count | Should -Be 3
|
||||
$secondRequests.Count | Should -Be 1
|
||||
$secondRequests[0].Payload.run.status | Should -Be 'running'
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
# Tests for collectors/common/BackupDashboard.Collector.psm1, the code all collectors share.
|
||||
|
||||
BeforeAll {
|
||||
Import-Module (Join-Path (Split-Path -Parent $PSScriptRoot) 'common/BackupDashboard.Collector.psm1') -Force
|
||||
Import-Module (Join-Path $PSScriptRoot 'TestHelpers.psm1') -Force
|
||||
|
||||
function New-TempConfig([hashtable]$Values, [hashtable]$State) {
|
||||
$dir = New-TestDir 'module'
|
||||
$path = Join-Path $dir 'test-collector.json'
|
||||
$Values | ConvertTo-Json | Set-Content -Path $path
|
||||
if ($State) { $State | ConvertTo-Json | Set-Content -Path (Join-Path $dir 'test-collector.state.json') }
|
||||
return $path
|
||||
}
|
||||
$config = @{ ApiUrl = 'http://localhost:1'; ApiKey = 'bdk_x'; Tenant = 'Acme' }
|
||||
}
|
||||
|
||||
Describe 'Get-FirstValue' {
|
||||
It 'returns the first property that has a value' {
|
||||
Get-FirstValue ([pscustomobject]@{ A = $null; B = ''; C = 'yes' }) @('A', 'B', 'C') | Should -Be 'yes'
|
||||
}
|
||||
It 'treats an empty collection as no value' {
|
||||
Get-FirstValue ([pscustomobject]@{ A = @(); B = 'yes' }) @('A', 'B') | Should -Be 'yes'
|
||||
}
|
||||
It 'returns a one-item list of objects (which [string] would turn into an empty string)' {
|
||||
$item = [pscustomobject]@{ Name = 'E:\' }
|
||||
$result = @(Get-FirstValue ([pscustomobject]@{ List = @($item) }) @('List'))
|
||||
$result.Count | Should -Be 1
|
||||
$result[0].Name | Should -Be 'E:\'
|
||||
}
|
||||
It 'returns $null for a missing property or object' {
|
||||
Get-FirstValue ([pscustomobject]@{ A = 1 }) @('Nope') | Should -BeNullOrEmpty
|
||||
Get-FirstValue $null @('A') | Should -BeNullOrEmpty
|
||||
}
|
||||
It 'keeps $false (a real value, not "missing")' {
|
||||
Get-FirstValue ([pscustomobject]@{ Enabled = $false }) @('Enabled') | Should -BeFalse
|
||||
$null -eq (Get-FirstValue ([pscustomobject]@{ Enabled = $false }) @('Enabled')) | Should -BeFalse
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'ConvertTo-IsoTime' {
|
||||
It 'gives local times their offset' {
|
||||
$local = [datetime]::new(2026, 10, 8, 21, 0, 0, [DateTimeKind]::Local)
|
||||
ConvertTo-IsoTime $local | Should -Be ([DateTimeOffset]$local).ToString("yyyy-MM-dd'T'HH:mm:sszzz")
|
||||
}
|
||||
It 'gives UTC times +00:00' {
|
||||
ConvertTo-IsoTime ([datetime]::new(2026, 10, 8, 3, 0, 5, [DateTimeKind]::Utc)) | Should -Be '2026-10-08T03:00:05+00:00'
|
||||
}
|
||||
It 'treats "not set" values as no time' {
|
||||
ConvertTo-IsoTime ([datetime]::MinValue) | Should -BeNullOrEmpty
|
||||
ConvertTo-IsoTime ([datetime]::new(1601, 1, 1)) | Should -BeNullOrEmpty # MARS
|
||||
ConvertTo-IsoTime ([datetime]::new(1900, 1, 1)) | Should -BeNullOrEmpty # Veeam
|
||||
ConvertTo-IsoTime $null | Should -BeNullOrEmpty
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Initialize-Collector' {
|
||||
It 'requires ApiUrl, ApiKey and Tenant by default' {
|
||||
$path = New-TempConfig @{ ApiUrl = 'http://x'; ApiKey = 'bdk_x' }
|
||||
{ Initialize-Collector -Name 'test' -ConfigPath $path } | Should -Throw "*missing 'Tenant'*"
|
||||
}
|
||||
It 'accepts a custom set of required keys' {
|
||||
$path = New-TempConfig @{ ApiUrl = 'http://x'; ApiKey = 'bdk_x' }
|
||||
{ Initialize-Collector -Name 'test' -ConfigPath $path -RequiredKeys 'ApiUrl', 'ApiKey' } | Should -Not -Throw
|
||||
}
|
||||
It 'resolves LookbackHours: parameter, then config, then 48' {
|
||||
(Initialize-Collector -Name 'test' -ConfigPath (New-TempConfig $config)).LookbackHours | Should -Be 48
|
||||
(Initialize-Collector -Name 'test' -ConfigPath (New-TempConfig ($config + @{ LookbackHours = 720 }))).LookbackHours | Should -Be 720
|
||||
(Initialize-Collector -Name 'test' -ConfigPath (New-TempConfig ($config + @{ LookbackHours = 720 })) -LookbackHours 12).LookbackHours | Should -Be 12
|
||||
}
|
||||
It 'loads runs already sent from the state file' {
|
||||
$path = New-TempConfig $config @{ 'run-1' = [DateTimeOffset]::Now.ToUnixTimeSeconds() }
|
||||
$null = Initialize-Collector -Name 'test' -ConfigPath $path
|
||||
Test-RunSent 'run-1' | Should -BeTrue
|
||||
Test-RunSent 'run-2' | Should -BeFalse
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Send-JobReport and Complete-Collector' {
|
||||
BeforeEach {
|
||||
$path = New-TempConfig $config @{
|
||||
'recent' = [DateTimeOffset]::Now.AddHours(-10).ToUnixTimeSeconds()
|
||||
'stale' = [DateTimeOffset]::Now.AddHours(-100).ToUnixTimeSeconds() # > 48h + 1 day ago
|
||||
}
|
||||
$statePath = Join-Path (Split-Path $path) 'test-collector.state.json'
|
||||
$job = [ordered]@{ external_id = 'j1'; name = 'Job'; next_run_at = $null }
|
||||
}
|
||||
It 'records a finished run as sent, but not a running one' {
|
||||
$null = Initialize-Collector -Name 'test' -ConfigPath $path -DryRun
|
||||
$null = Send-JobReport -Tenant 'Acme' -SourceSystem 'Test' -Job $job -Run ([ordered]@{ status = 'success' }) -SentKey 'done' -StartedAt (Get-Date) 6>&1
|
||||
$null = Send-JobReport -Tenant 'Acme' -SourceSystem 'Test' -Job $job -Run ([ordered]@{ status = 'running' }) -SentKey 'busy' -StartedAt (Get-Date) 6>&1
|
||||
Test-RunSent 'done' | Should -BeTrue
|
||||
Test-RunSent 'busy' | Should -BeFalse
|
||||
}
|
||||
It 'prints the payload in a dry run, with the job and run' {
|
||||
$null = Initialize-Collector -Name 'test' -ConfigPath $path -DryRun
|
||||
$printed = Send-JobReport -Tenant 'Acme' -SourceSystem 'Test' -Job $job -Run ([ordered]@{ status = 'success' }) -SentKey 'x' -StartedAt (Get-Date) 6>&1
|
||||
$payload = "$printed" | ConvertFrom-Json
|
||||
$payload.tenant | Should -Be 'Acme'
|
||||
$payload.source_system | Should -Be 'Test'
|
||||
$payload.job.external_id | Should -Be 'j1'
|
||||
$payload.run.status | Should -Be 'success'
|
||||
}
|
||||
It 'saves the state, dropping runs older than the lookback window plus a day' {
|
||||
$null = Initialize-Collector -Name 'test' -ConfigPath $path
|
||||
$null = Complete-Collector -Failures 0 6>&1
|
||||
$saved = Get-Content $statePath -Raw | ConvertFrom-Json
|
||||
$saved.PSObject.Properties.Name | Should -Be @('recent')
|
||||
}
|
||||
It "doesn't touch the state in a dry run" {
|
||||
$before = Get-Content $statePath -Raw
|
||||
$null = Initialize-Collector -Name 'test' -ConfigPath $path -DryRun
|
||||
$null = Complete-Collector -Failures 0 6>&1
|
||||
Get-Content $statePath -Raw | Should -Be $before
|
||||
}
|
||||
It 'returns exit code 1 when anything failed, 0 otherwise' {
|
||||
$null = Initialize-Collector -Name 'test' -ConfigPath $path -DryRun
|
||||
Complete-Collector -Failures 0 6>$null | Should -Be 0
|
||||
Complete-Collector -Failures 2 6>$null | Should -Be 1
|
||||
}
|
||||
It 'logs a summary with or without a job count' {
|
||||
$null = Initialize-Collector -Name 'test' -ConfigPath $path -DryRun
|
||||
"$(Complete-Collector -JobCount 3 -Failures 0 6>&1 | Select-Object -First 1)" | Should -Match 'Reported 3 jobs, 0 runs, 0 failures$'
|
||||
"$(Complete-Collector -Failures 1 6>&1 | Select-Object -First 1)" | Should -Match 'Reported 0 runs, 1 failures$'
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Test-AuthRejected' {
|
||||
It 'is true only for HTTP 401' {
|
||||
$error401 = [pscustomobject]@{ Exception = [pscustomobject]@{ Response = [pscustomobject]@{ StatusCode = 401 } } }
|
||||
$error500 = [pscustomobject]@{ Exception = [pscustomobject]@{ Response = [pscustomobject]@{ StatusCode = 500 } } }
|
||||
$noResponse = [pscustomobject]@{ Exception = [pscustomobject]@{ Message = 'timeout' } }
|
||||
Test-AuthRejected $error401 | Should -BeTrue
|
||||
Test-AuthRejected $error500 | Should -BeFalse
|
||||
Test-AuthRejected $noResponse | Should -BeFalse
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
# Collector tests
|
||||
|
||||
Pester tests for the collectors, the shared module and the installers. Each collector runs against
|
||||
its mock (`collectors/<name>/tests/Mock*.ps1`, built from real servers' diagnostic output) and a
|
||||
stand-in for the ingest API, so no backup product or dashboard is needed.
|
||||
|
||||
```powershell
|
||||
Install-Module Pester -MinimumVersion 5.5 -Scope CurrentUser # once
|
||||
./collectors/tests/Invoke-CollectorTests.ps1 # everything, ~15 s
|
||||
./collectors/tests/Invoke-CollectorTests.ps1 -Path ./collectors/tests/Wsb.Tests.ps1
|
||||
```
|
||||
|
||||
Needs PowerShell 7.2+ to run the tests. Collectors run in their own process, under the same
|
||||
PowerShell by default. On Windows, run them under Windows PowerShell 5.1, as the MARS, WSB and
|
||||
VB365 servers do:
|
||||
|
||||
```powershell
|
||||
./collectors/tests/Invoke-CollectorTests.ps1 -CollectorShell powershell.exe
|
||||
```
|
||||
|
||||
The runner exits non-zero if anything fails.
|
||||
|
||||
## What's covered
|
||||
|
||||
| File | Checks |
|
||||
|---|---|
|
||||
| `Module.Tests.ps1` | `BackupDashboard.Collector.psm1`: property lookup, time format, config and lookback, state (recording sent runs, pruning), dry runs, exit codes, 401 detection |
|
||||
| `Veeam.Tests.ps1` | Next run from the live schedule (VBR 13), none for copy/disabled jobs; copy-job worker sessions; statuses and messages; state and resending; rejected key; missing config |
|
||||
| `Mars.Tests.ps1` | UTC job times; ErrorCode 0 isn't an error; failed files → warning; next run from the schedule; JobName; state |
|
||||
| `Wsb.Tests.ps1` | Culture-formatted times (en-US and en-GB); VMs nested in VirtualMachineList; HRESULTs; failure log only when relevant; running → finished merging on start time |
|
||||
| `Vb365.Tests.ps1` | Organization → company mapping; `[Warning]`/`[Failed]` log prefixes; retries; "187.2 MB" sizes; state |
|
||||
| `Installers.Tests.ps1` | Config written (incl. the VB365 map), files copied (collector + module), task registered (shell, interval, SYSTEM), refusals before changing anything |
|
||||
|
||||
Each of these was a real bug or a real-server finding. To check the suite still catches them,
|
||||
put one back (e.g. read `$Job.ScheduleOptions.NextRun` instead of `GetScheduleOptions()` in the
|
||||
Veeam collector) and run the tests: they should fail.
|
||||
|
||||
## Adding a collector
|
||||
|
||||
1. Write `collectors/<name>/tests/Mock<Name>.ps1` from the product's diagnostic output, taking
|
||||
its time from `$env:MOCK_NOW` like the others.
|
||||
2. Add `<name>` to `Invoke-Collector`'s `ValidateSet` in `TestHelpers.psm1`.
|
||||
3. Add `<Name>.Tests.ps1`, one `It` per behaviour confirmed on a real server.
|
||||
@@ -0,0 +1,166 @@
|
||||
<#
|
||||
Helpers for the collector tests: run a collector against its mock in a separate PowerShell
|
||||
process, and stand in for the dashboard's ingest API.
|
||||
|
||||
Each collector run gets its own process so the mocks' global functions and the collector's
|
||||
`exit` don't leak between tests. The shell is the one running the tests, or
|
||||
$env:COLLECTOR_TEST_SHELL (e.g. powershell.exe, to run the collectors under Windows
|
||||
PowerShell 5.1 like most servers do).
|
||||
#>
|
||||
|
||||
$script:CollectorsDir = Split-Path -Parent $PSScriptRoot
|
||||
$script:Invariant = [Globalization.CultureInfo]::InvariantCulture
|
||||
|
||||
# Temp folders for this test run live under one root, which Invoke-CollectorTests.ps1 removes
|
||||
function New-TestDir([string]$Prefix = 'test') {
|
||||
$root = if ($env:COLLECTOR_TEST_TEMP) { $env:COLLECTOR_TEST_TEMP } else { Join-Path ([IO.Path]::GetTempPath()) 'backup-dashboard-tests' }
|
||||
$dir = Join-Path $root ("$Prefix-" + [guid]::NewGuid().ToString('N').Substring(0, 12))
|
||||
New-Item -ItemType Directory -Path $dir -Force | Out-Null
|
||||
return $dir
|
||||
}
|
||||
|
||||
function Get-TestShell {
|
||||
if ($env:COLLECTOR_TEST_SHELL) { return $env:COLLECTOR_TEST_SHELL }
|
||||
return (Get-Process -Id $PID).Path
|
||||
}
|
||||
|
||||
<#
|
||||
Pins the time the mocks are relative to (the top of the current hour) for this test session,
|
||||
so a test can't fail because the hour turned over between the run and the assertions.
|
||||
Returns it as a local DateTime.
|
||||
#>
|
||||
function Set-MockNow {
|
||||
$now = Get-Date -Minute 0 -Second 0 -Millisecond 0
|
||||
$env:MOCK_NOW = $now.ToString('o', $script:Invariant)
|
||||
return [datetime]$env:MOCK_NOW
|
||||
}
|
||||
|
||||
# The API's time format: "2026-10-08T21:00:00-06:00"
|
||||
function ConvertTo-ApiTime([datetime]$Value) {
|
||||
return ([DateTimeOffset]$Value).ToString("yyyy-MM-dd'T'HH:mm:sszzz", $script:Invariant)
|
||||
}
|
||||
|
||||
# JSON with date-looking strings left as strings (PowerShell 7.5+ would otherwise make them DateTimes)
|
||||
function ConvertFrom-JsonText([string]$Json) {
|
||||
if ((Get-Command ConvertFrom-Json).Parameters.ContainsKey('DateKind')) {
|
||||
return $Json | ConvertFrom-Json -DateKind String
|
||||
}
|
||||
return $Json | ConvertFrom-Json
|
||||
}
|
||||
|
||||
function ConvertTo-PSLiteral([string]$Value) { return "'" + $Value.Replace("'", "''") + "'" }
|
||||
|
||||
<#
|
||||
Runs one collector against its mock. Returns:
|
||||
ExitCode the collector's exit code (0 when it ends normally)
|
||||
Reports in -DryRun, the payloads it would have sent
|
||||
Log its log lines (timestamp stripped), from the console
|
||||
State the sent-runs state file, as a hashtable (empty if none)
|
||||
WorkDir pass back in to run again with the same config and state
|
||||
#>
|
||||
function Invoke-Collector {
|
||||
param(
|
||||
[Parameter(Mandatory, Position = 0)][ValidateSet('veeam', 'mars', 'wsb', 'vb365')][string]$Name,
|
||||
[Parameter(Mandatory)][hashtable]$Config,
|
||||
[string]$WorkDir,
|
||||
[switch]$DryRun,
|
||||
[string]$MockArgs = '',
|
||||
[string]$Culture = 'en-US',
|
||||
[string]$ExtraArgs = '',
|
||||
[string]$ComputerName = 'SRV01'
|
||||
)
|
||||
$dir = Join-Path $script:CollectorsDir $Name
|
||||
$script = (Get-ChildItem (Join-Path $dir 'Send-*.ps1')).FullName
|
||||
$mock = (Get-ChildItem (Join-Path (Join-Path $dir 'tests') 'Mock*.ps1')).FullName
|
||||
if (-not $WorkDir) {
|
||||
$WorkDir = New-TestDir $Name
|
||||
}
|
||||
$configPath = Join-Path $WorkDir "$Name-collector.json"
|
||||
$Config | ConvertTo-Json -Depth 5 | Set-Content -Path $configPath -Encoding UTF8
|
||||
|
||||
$command = @(
|
||||
"[Globalization.CultureInfo]::CurrentCulture = '$Culture'"
|
||||
". $(ConvertTo-PSLiteral $mock) $MockArgs"
|
||||
"& $(ConvertTo-PSLiteral $script) -ConfigPath $(ConvertTo-PSLiteral $configPath) $(if ($DryRun) { '-DryRun' }) $ExtraArgs"
|
||||
'exit $LASTEXITCODE'
|
||||
) -join '; '
|
||||
|
||||
$savedComputer = $env:COMPUTERNAME
|
||||
$env:COMPUTERNAME = $ComputerName
|
||||
try {
|
||||
$output = @(& (Get-TestShell) -NoProfile -NonInteractive -Command $command 2>&1 | ForEach-Object { "$_" })
|
||||
$exitCode = $LASTEXITCODE
|
||||
} finally {
|
||||
$env:COMPUTERNAME = $savedComputer
|
||||
}
|
||||
|
||||
$state = @{}
|
||||
$statePath = Join-Path $WorkDir "$Name-collector.state.json"
|
||||
if (Test-Path $statePath) {
|
||||
$saved = ConvertFrom-JsonText (Get-Content $statePath -Raw)
|
||||
if ($saved) { foreach ($p in $saved.PSObject.Properties) { $state[$p.Name] = $p.Value } }
|
||||
}
|
||||
return [pscustomobject]@{
|
||||
ExitCode = $exitCode
|
||||
# Payloads are single-line JSON; a logged API error body spans several lines starting with '{'
|
||||
Reports = @($output | Where-Object { $_ -match '^\{".*\}$' } | ForEach-Object { ConvertFrom-JsonText $_ })
|
||||
Log = @($output | Where-Object { $_ -match '^\d{4}-\d\d-\d\d \d\d:\d\d:\d\d \[' } | ForEach-Object { $_.Substring(20) })
|
||||
Output = $output
|
||||
State = $state
|
||||
WorkDir = $WorkDir
|
||||
}
|
||||
}
|
||||
|
||||
<#
|
||||
A stand-in for the ingest API on a free localhost port. Records every request; answers 401
|
||||
for API keys containing "bad", 200 otherwise. Stop it with Stop-ApiStub.
|
||||
#>
|
||||
function Start-ApiStub {
|
||||
$probe = [Net.Sockets.TcpListener]::new([Net.IPAddress]::Loopback, 0)
|
||||
$probe.Start(); $port = $probe.LocalEndpoint.Port; $probe.Stop()
|
||||
|
||||
$listener = [Net.HttpListener]::new()
|
||||
$listener.Prefixes.Add("http://localhost:$port/")
|
||||
$listener.Start()
|
||||
$requests = [Collections.Concurrent.ConcurrentQueue[object]]::new()
|
||||
$job = Start-ThreadJob -ArgumentList $listener, $requests -ScriptBlock {
|
||||
param($listener, $requests)
|
||||
while ($listener.IsListening) {
|
||||
try { $context = $listener.GetContext() } catch { break }
|
||||
$body = [IO.StreamReader]::new($context.Request.InputStream, [Text.Encoding]::UTF8).ReadToEnd()
|
||||
$auth = $context.Request.Headers['Authorization']
|
||||
$requests.Enqueue([pscustomobject]@{
|
||||
Path = $context.Request.Url.AbsolutePath; Auth = $auth
|
||||
ContentType = $context.Request.ContentType; Body = $body
|
||||
})
|
||||
$rejected = $auth -match 'bad'
|
||||
$reply = if ($rejected) { '{"detail":"Invalid or revoked API key"}' } else { '{"job_id":1,"run_id":1}' }
|
||||
$bytes = [Text.Encoding]::UTF8.GetBytes($reply)
|
||||
$context.Response.StatusCode = $(if ($rejected) { 401 } else { 200 })
|
||||
$context.Response.ContentType = 'application/json'
|
||||
$context.Response.OutputStream.Write($bytes, 0, $bytes.Length)
|
||||
$context.Response.Close()
|
||||
}
|
||||
}
|
||||
return [pscustomobject]@{ Url = "http://localhost:$port"; Listener = $listener; Requests = $requests; Job = $job }
|
||||
}
|
||||
|
||||
# Takes the requests recorded since the last call; Body is the parsed JSON payload
|
||||
function Get-StubRequests($Stub) {
|
||||
$taken = @()
|
||||
$item = $null
|
||||
while ($Stub.Requests.TryDequeue([ref]$item)) {
|
||||
$item | Add-Member -NotePropertyName Payload -NotePropertyValue (ConvertFrom-JsonText $item.Body) -Force
|
||||
$taken += $item
|
||||
}
|
||||
return , $taken
|
||||
}
|
||||
|
||||
function Stop-ApiStub($Stub) {
|
||||
$Stub.Listener.Stop()
|
||||
$Stub.Listener.Close()
|
||||
$Stub.Job | Stop-Job -PassThru | Remove-Job -Force
|
||||
}
|
||||
|
||||
Export-ModuleMember -Function New-TestDir, Set-MockNow, ConvertTo-ApiTime, ConvertFrom-JsonText, Invoke-Collector,
|
||||
Start-ApiStub, Get-StubRequests, Stop-ApiStub
|
||||
@@ -0,0 +1,116 @@
|
||||
# Veeam Backup for Microsoft 365 collector, against collectors/vb365/tests/MockVb365.ps1 (shapes from
|
||||
# a VB365 8.4 server: log items carry their status as a Title prefix, retries are separate sessions).
|
||||
|
||||
BeforeAll {
|
||||
Import-Module (Join-Path $PSScriptRoot 'TestHelpers.psm1') -Force
|
||||
$now = Set-MockNow
|
||||
$base = @{ ApiUrl = 'http://localhost:1'; ApiKey = 'bdk_x' }
|
||||
|
||||
function Get-JobRuns($Reports, [string]$JobName) {
|
||||
@($Reports | Where-Object { $_.job.name -eq $JobName -and $_.run } | ForEach-Object run)
|
||||
}
|
||||
function Get-Tenants($Reports) {
|
||||
$map = @{}
|
||||
foreach ($report in $Reports) { $map[$report.job.name] = $report.tenant }
|
||||
return $map
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'VB365 collector' {
|
||||
Context 'what it reports (dry run)' {
|
||||
BeforeAll {
|
||||
$r = Invoke-Collector vb365 -Config ($base + @{ Tenant = 'Default Co' }) -DryRun
|
||||
$acme = Get-JobRuns $r.Reports 'Acme 365 Backup'
|
||||
$blue = Get-JobRuns $r.Reports 'Bluewater 365 Backup'
|
||||
}
|
||||
|
||||
It 'runs cleanly, reporting every job' {
|
||||
$r.ExitCode | Should -Be 0
|
||||
$r.Log[-1] | Should -Be '[INFO] Reported 3 jobs, 5 runs, 0 failures'
|
||||
$r.Reports.source_system | Select-Object -Unique | Should -Be 'Veeam Backup for M365'
|
||||
}
|
||||
It "uses each job's NextRun, and none for a disabled job" {
|
||||
$next = $now.Date.AddHours(17).AddDays($(if ($now.Hour -ge 17) { 1 } else { 0 }))
|
||||
($r.Reports | Where-Object { $_.job.name -eq 'Acme 365 Backup' })[0].job.next_run_at | Should -Be (ConvertTo-ApiTime $next)
|
||||
($r.Reports | Where-Object { $_.job.name -eq 'Bluewater Old Mailboxes' })[0].job.next_run_at | Should -BeNullOrEmpty
|
||||
}
|
||||
It 'reports a successful session with its size ("187.2 MB" in bytes) and objects' {
|
||||
$run = $acme | Where-Object status -EQ 'success'
|
||||
$run.started_at | Should -Be (ConvertTo-ApiTime $now.Date.AddDays(-1).AddHours(17))
|
||||
$run.size_bytes | Should -Be ([long](187.2 * 1MB))
|
||||
$run.details.processed_objects | Should -Be 568
|
||||
$run.details.organization | Should -Be 'acme.onmicrosoft.com'
|
||||
$run.message | Should -Be 'Job completed successfully.'
|
||||
}
|
||||
It 'reports a session in progress as running, without a finish time' {
|
||||
$run = $acme | Where-Object status -EQ 'running'
|
||||
$run.PSObject.Properties.Name | Should -Not -Contain 'finished_at'
|
||||
$run.message | Should -Be ''
|
||||
}
|
||||
It 'builds the message from [Warning]/[Failed] log items, without the prefix' {
|
||||
$run = $blue | Where-Object status -EQ 'warning'
|
||||
$run.message | Should -BeLike 'Processing mailbox [email protected] finished with warning: Item "Q3.pst" exceeds the maximum size*'
|
||||
# Literal check: in -like patterns, [Warning] is a character set
|
||||
$run.message.Contains('[Warning]') | Should -BeFalse
|
||||
}
|
||||
It 'ignores [Success] log items, even ones mentioning "error"' {
|
||||
($blue | Where-Object status -EQ 'warning').message | Should -Not -BeLike '*error-reports*'
|
||||
}
|
||||
It 'flags a session VB365 will retry, and links the retry back to it' {
|
||||
$warning = $blue | Where-Object status -EQ 'warning'
|
||||
$retry = $blue | Where-Object { $_.details.retry_of }
|
||||
$warning.message | Should -BeLike '*Veeam will retry this job.'
|
||||
$warning.details.will_retry | Should -BeTrue
|
||||
$retry.status | Should -Be 'success'
|
||||
$retry.details.retry_of | Should -Be $warning.external_id
|
||||
}
|
||||
It 'reports a failed session with its error' {
|
||||
$run = $blue | Where-Object status -EQ 'failed'
|
||||
$run.message | Should -BeLike '*(401) Unauthorized*'
|
||||
}
|
||||
It 'skips sessions outside the lookback window' {
|
||||
Get-JobRuns $r.Reports 'Bluewater Old Mailboxes' | Should -BeNullOrEmpty
|
||||
}
|
||||
}
|
||||
|
||||
Context 'which company a job appears under' {
|
||||
It 'uses OrganizationTenants (not case-sensitive), then Tenant for unmapped organizations' {
|
||||
$r = Invoke-Collector vb365 -DryRun -Config ($base + @{
|
||||
Tenant = 'Default Co'; OrganizationTenants = @{ 'ACME.onmicrosoft.com' = 'Acme Corp' } })
|
||||
$tenants = Get-Tenants $r.Reports
|
||||
$tenants['Acme 365 Backup'] | Should -Be 'Acme Corp'
|
||||
$tenants['Bluewater 365 Backup'] | Should -Be 'Default Co'
|
||||
}
|
||||
It "falls back to the organization's name with no mapping and no Tenant" {
|
||||
$r = Invoke-Collector vb365 -Config $base -DryRun
|
||||
$r.ExitCode | Should -Be 0
|
||||
$tenants = Get-Tenants $r.Reports
|
||||
$tenants['Acme 365 Backup'] | Should -Be 'acme.onmicrosoft.com'
|
||||
$tenants['Bluewater 365 Backup'] | Should -Be 'bluewater.onmicrosoft.com'
|
||||
}
|
||||
}
|
||||
|
||||
Context 'sending to the API' {
|
||||
BeforeAll {
|
||||
$stub = Start-ApiStub
|
||||
$live = $base + @{ Tenant = 'Default Co' }; $live.ApiUrl = $stub.Url
|
||||
$first = Invoke-Collector vb365 -Config $live
|
||||
$firstRequests = Get-StubRequests $stub
|
||||
$second = Invoke-Collector vb365 -Config $live -WorkDir $first.WorkDir
|
||||
$secondRequests = Get-StubRequests $stub
|
||||
}
|
||||
AfterAll { Stop-ApiStub $stub }
|
||||
|
||||
It 'sends every session, remembering the finished ones' {
|
||||
$first.ExitCode | Should -Be 0
|
||||
@($firstRequests | Where-Object { $_.Payload.run }).Count | Should -Be 5
|
||||
$first.State.Count | Should -Be 4
|
||||
}
|
||||
It 'on the next run, resends only the running session (and refreshes every job)' {
|
||||
$resent = @($secondRequests | Where-Object { $_.Payload.run })
|
||||
$resent.Count | Should -Be 1
|
||||
$resent[0].Payload.run.status | Should -Be 'running'
|
||||
$secondRequests.Count | Should -Be 3
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
# Veeam B&R collector, against collectors/veeam/tests/MockVeeam.ps1 (shapes from a VBR 13 server).
|
||||
|
||||
BeforeAll {
|
||||
Import-Module (Join-Path $PSScriptRoot 'TestHelpers.psm1') -Force
|
||||
$now = Set-MockNow
|
||||
$dash = [char]0x2013 # job names use an en dash, as real ones often do
|
||||
$fileServer = "File Server $dash Nightly"
|
||||
$config = @{ ApiUrl = 'http://localhost:1'; ApiKey = 'bdk_x'; Tenant = 'Acme' }
|
||||
|
||||
function Get-Runs($Reports, [string]$JobName) {
|
||||
@($Reports | Where-Object { $_.job.name -eq $JobName -and $_.run } | ForEach-Object run)
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Veeam collector' {
|
||||
Context 'what it reports (dry run)' {
|
||||
BeforeAll {
|
||||
$r = Invoke-Collector veeam -Config $config -DryRun
|
||||
$jobs = @{}
|
||||
foreach ($report in $r.Reports) { $jobs[$report.job.name] = $report.job }
|
||||
}
|
||||
|
||||
It 'runs cleanly' {
|
||||
$r.ExitCode | Should -Be 0
|
||||
$r.Log[-1] | Should -Be '[INFO] Reported 5 jobs, 6 runs, 0 failures'
|
||||
}
|
||||
It 'reports every job, under the configured tenant' {
|
||||
$jobs.Keys | Sort-Object | Should -Be @('Cloud Copy', $fileServer, 'Offsite Copy', 'Old Exchange', "SQL01 $dash Logs")
|
||||
$r.Reports.tenant | Select-Object -Unique | Should -Be 'Acme'
|
||||
$r.Reports.source_system | Select-Object -Unique | Should -Be 'Veeam B&R'
|
||||
}
|
||||
It "reads the next run from the live schedule (VBR 13's ScheduleOptions property is empty)" {
|
||||
$jobs[$fileServer].next_run_at | Should -Be (ConvertTo-ApiTime $now.Date.AddDays(1).AddMinutes(15))
|
||||
$jobs["SQL01 $dash Logs"].next_run_at | Should -Be (ConvertTo-ApiTime $now.AddMinutes(40))
|
||||
}
|
||||
It 'gives no next run to immediate-mode copy jobs (their NextRun is stale), disabled or unscheduled jobs' {
|
||||
$jobs['Cloud Copy'].next_run_at | Should -BeNullOrEmpty
|
||||
$jobs['Old Exchange'].next_run_at | Should -BeNullOrEmpty
|
||||
$jobs['Offsite Copy'].next_run_at | Should -BeNullOrEmpty
|
||||
}
|
||||
It 'reports a successful session with its times and size' {
|
||||
$run = Get-Runs $r.Reports $fileServer | Where-Object status -EQ 'success'
|
||||
$run.started_at | Should -Be (ConvertTo-ApiTime $now.AddDays(-1).AddHours(-2))
|
||||
$run.finished_at | Should -Be (ConvertTo-ApiTime $now.AddDays(-1).AddHours(-1))
|
||||
$run.size_bytes | Should -Be 128GB
|
||||
$run.message | Should -Be 'Job completed successfully.'
|
||||
$run.details.objects | Should -Be 2
|
||||
}
|
||||
It 'reports a session in progress as running, without a finish time' {
|
||||
$run = Get-Runs $r.Reports $fileServer | Where-Object status -EQ 'running'
|
||||
$run.started_at | Should -Be (ConvertTo-ApiTime $now.AddMinutes(-20))
|
||||
$run.PSObject.Properties.Name | Should -Not -Contain 'finished_at'
|
||||
$run.message | Should -Be ''
|
||||
}
|
||||
It 'names the object behind a warning' {
|
||||
$run = Get-Runs $r.Reports "SQL01 $dash Logs"
|
||||
$run.status | Should -Be 'warning'
|
||||
$run.message | Should -BeLike 'SQL01: Warning - Failed to truncate transaction logs*'
|
||||
$run.details.objects_warning | Should -Be 1
|
||||
}
|
||||
It "includes a failure's reason and failed log lines, but not successful ones" {
|
||||
$run = Get-Runs $r.Reports 'Offsite Copy'
|
||||
$run.status | Should -Be 'failed'
|
||||
$run.message | Should -BeLike '*Unable to connect to repository NAS02*'
|
||||
$run.message | Should -BeLike '*Network path not found (0x80070035)*'
|
||||
$run.message | Should -Not -BeLike '*Preparing backup chain*'
|
||||
}
|
||||
It "files copy jobs' worker sessions under the copy job, naming the source" {
|
||||
$runs = Get-Runs $r.Reports 'Cloud Copy'
|
||||
$runs.status | Sort-Object | Should -Be @('success', 'warning')
|
||||
$runs.details.copied_from | Select-Object -Unique | Should -Be $fileServer
|
||||
}
|
||||
It "skips sessions outside the lookback window and from jobs it doesn't know" {
|
||||
@($r.Reports | Where-Object run).Count | Should -Be 6
|
||||
Get-Runs $r.Reports 'Old Exchange' | Should -BeNullOrEmpty
|
||||
}
|
||||
}
|
||||
|
||||
Context 'sending to the API' {
|
||||
BeforeAll {
|
||||
$stub = Start-ApiStub
|
||||
$live = $config.Clone(); $live.ApiUrl = $stub.Url; $live.ApiKey = 'bdk_good'
|
||||
$first = Invoke-Collector veeam -Config $live
|
||||
$firstRequests = Get-StubRequests $stub
|
||||
$second = Invoke-Collector veeam -Config $live -WorkDir $first.WorkDir
|
||||
$secondRequests = Get-StubRequests $stub
|
||||
}
|
||||
AfterAll { Stop-ApiStub $stub }
|
||||
|
||||
It 'posts each report to /api/v1/ingest with the API key, as UTF-8 JSON' {
|
||||
$first.ExitCode | Should -Be 0
|
||||
$firstRequests.Count | Should -Be 7
|
||||
$firstRequests.Path | Select-Object -Unique | Should -Be '/api/v1/ingest'
|
||||
$firstRequests.Auth | Select-Object -Unique | Should -Be 'Bearer bdk_good'
|
||||
$firstRequests.ContentType | Select-Object -Unique | Should -Be 'application/json; charset=utf-8'
|
||||
}
|
||||
It 'keeps non-ASCII job names intact' {
|
||||
$firstRequests.Payload.job.name | Should -Contain $fileServer
|
||||
}
|
||||
It 'remembers finished sessions, but not running ones' {
|
||||
$first.State.Count | Should -Be 5
|
||||
}
|
||||
It 'on the next run, resends only the running session (and refreshes every job)' {
|
||||
$resent = @($secondRequests | Where-Object { $_.Payload.run })
|
||||
$resent.Count | Should -Be 1
|
||||
$resent[0].Payload.run.status | Should -Be 'running'
|
||||
$secondRequests.Count | Should -Be 5
|
||||
}
|
||||
}
|
||||
|
||||
Context 'problems' {
|
||||
It 'stops after the first job when the API key is rejected, exiting 1' {
|
||||
$stub = Start-ApiStub
|
||||
try {
|
||||
$r = Invoke-Collector veeam -Config @{ ApiUrl = $stub.Url; ApiKey = 'bdk_bad'; Tenant = 'Acme' }
|
||||
(Get-StubRequests $stub).Count | Should -Be 1
|
||||
$r.ExitCode | Should -Be 1
|
||||
$r.Log | Should -Contain '[ERROR] API key rejected; check ApiKey in the config. Stopping.'
|
||||
} finally { Stop-ApiStub $stub }
|
||||
}
|
||||
It 'exits 2 when the config lacks a tenant' {
|
||||
$r = Invoke-Collector veeam -Config @{ ApiUrl = 'http://localhost:1'; ApiKey = 'bdk_x' } -DryRun
|
||||
$r.ExitCode | Should -Be 2
|
||||
$r.Log[0] | Should -BeLike "*is missing 'Tenant'*"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
# Windows Server Backup collector, against collectors/wsb/tests/MockWsb.ps1 (shapes from a Server
|
||||
# 2019 Hyper-V host: job times are culture-formatted local-time strings, HRESULTs are strings,
|
||||
# VMs are nested under a VirtualMachineList item).
|
||||
|
||||
BeforeAll {
|
||||
Import-Module (Join-Path $PSScriptRoot 'TestHelpers.psm1') -Force
|
||||
$now = Set-MockNow
|
||||
$config = @{ ApiUrl = 'http://localhost:1'; ApiKey = 'bdk_x'; Tenant = 'Acme' }
|
||||
$vms = @('AAOG-EDM-AD1', 'Host Component')
|
||||
$hostComponentBytes = 447729
|
||||
}
|
||||
|
||||
Describe 'Windows Server Backup collector' {
|
||||
Context 'what it reports (dry run)' {
|
||||
BeforeAll {
|
||||
$r = Invoke-Collector wsb -Config $config -DryRun
|
||||
$runs = @($r.Reports | Where-Object run | ForEach-Object run)
|
||||
$byStart = @{}
|
||||
foreach ($run in $runs) { $byStart[$run.started_at] = $run }
|
||||
function At([double]$HoursAgo) { $byStart[(ConvertTo-ApiTime $now.AddHours(-$HoursAgo))] }
|
||||
}
|
||||
|
||||
It 'runs cleanly' {
|
||||
$r.ExitCode | Should -Be 0
|
||||
$r.Log[-1] | Should -Be '[INFO] Reported 4 runs, 0 failures'
|
||||
}
|
||||
It 'reports the server as one job, with the next backup from Get-WBSummary' {
|
||||
$r.Reports.job.external_id | Select-Object -Unique | Should -Be 'wsb:srv01'
|
||||
$r.Reports.job.name | Select-Object -Unique | Should -Be 'SRV01 - Windows Server Backup'
|
||||
$r.Reports.source_system | Select-Object -Unique | Should -Be 'Windows Server Backup'
|
||||
$next = $now.Date.AddHours(21).AddDays($(if ($now.Hour -ge 21) { 1 } else { 0 }))
|
||||
$r.Reports[0].job.next_run_at | Should -Be (ConvertTo-ApiTime $next)
|
||||
}
|
||||
It 'reads job times as local time, and sends no run ID (WSB jobs have none; the API keys on start time)' {
|
||||
(At 12).finished_at | Should -Be (ConvertTo-ApiTime $now.AddHours(-11.6))
|
||||
$runs | ForEach-Object { $_.PSObject.Properties.Name | Should -Not -Contain 'external_id' }
|
||||
}
|
||||
It 'reports a Hyper-V backup with its VMs (nested in VirtualMachineList) and size' {
|
||||
$run = At 12
|
||||
$run.status | Should -Be 'success'
|
||||
$run.message | Should -Be 'Backup completed successfully.'
|
||||
$run.details.items | Should -Be $vms
|
||||
$run.size_bytes | Should -Be (124756706036 + $hostComponentBytes)
|
||||
}
|
||||
It "doesn't cite the failure log for a success (WSB fills it in regardless)" {
|
||||
(At 12).message | Should -Not -BeLike '*Failure log*'
|
||||
}
|
||||
It 'reports a backup where one VM failed as a warning, naming the VM' {
|
||||
$run = At 36
|
||||
$run.status | Should -Be 'warning'
|
||||
$run.message | Should -BeLike '*AAOG-EDM-AD1 - Access is denied.*'
|
||||
$run.message | Should -BeLike '*Failure log: C:\Windows\Logs\WindowsServerBackup\Backup_Error.log*'
|
||||
}
|
||||
It 'reports a backup with an error HRESULT as failed, with the code in 0x8 form' {
|
||||
$run = At 30
|
||||
$run.status | Should -Be 'failed'
|
||||
$run.message | Should -BeLike '*The backup storage location could not be found.*'
|
||||
$run.message | Should -BeLike '*error 0x80042302*'
|
||||
}
|
||||
It 'reports a volume backup with its volumes and size' {
|
||||
$run = At 20
|
||||
$run.status | Should -Be 'success'
|
||||
$run.details.items | Should -Be @('C:', 'D:')
|
||||
$run.size_bytes | Should -Be (240GB)
|
||||
}
|
||||
It "ignores restores, old backups, and Get-WBJob's empty job when nothing is running" {
|
||||
$runs.Count | Should -Be 4
|
||||
}
|
||||
It 'reads dd/MM dates correctly on servers with other regional settings' {
|
||||
$gb = Invoke-Collector wsb -Config $config -DryRun -Culture 'en-GB'
|
||||
($gb.Reports | ConvertTo-Json -Depth 6) | Should -Be ($r.Reports | ConvertTo-Json -Depth 6)
|
||||
}
|
||||
It 'reports a backup in progress as running, with no message' {
|
||||
$running = Invoke-Collector wsb -Config $config -DryRun -MockArgs '-WithRunningJob'
|
||||
$run = $running.Reports | Where-Object { $_.run.status -eq 'running' } | ForEach-Object run
|
||||
$run.started_at | Should -Be (ConvertTo-ApiTime $now.AddMinutes(-10))
|
||||
$run.message | Should -Be ''
|
||||
$run.PSObject.Properties.Name | Should -Not -Contain 'finished_at'
|
||||
$run.details.items | Should -Be $vms
|
||||
}
|
||||
}
|
||||
|
||||
Context 'sending to the API' {
|
||||
BeforeAll {
|
||||
$stub = Start-ApiStub
|
||||
$live = $config.Clone(); $live.ApiUrl = $stub.Url
|
||||
$first = Invoke-Collector wsb -Config $live -MockArgs '-WithRunningJob'
|
||||
$firstRequests = Get-StubRequests $stub
|
||||
$second = Invoke-Collector wsb -Config $live -WorkDir $first.WorkDir -MockArgs '-RunningJobFinished'
|
||||
$secondRequests = Get-StubRequests $stub
|
||||
}
|
||||
AfterAll { Stop-ApiStub $stub }
|
||||
|
||||
It 'sends every run, remembering the finished ones' {
|
||||
$first.ExitCode | Should -Be 0
|
||||
$firstRequests.Count | Should -Be 5
|
||||
$first.State.Count | Should -Be 4
|
||||
}
|
||||
It 'sends the finished backup once it ends, with the same start time, so the API merges it into the running one' {
|
||||
$running = $firstRequests.Payload.run | Where-Object status -EQ 'running'
|
||||
$secondRequests.Count | Should -Be 1
|
||||
$finished = $secondRequests[0].Payload.run
|
||||
$finished.status | Should -Be 'success'
|
||||
$finished.started_at | Should -Be $running.started_at
|
||||
$second.State.Count | Should -Be 5
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Read-only dump of what the Veeam Backup for Microsoft 365 cmdlets return, for troubleshooting
|
||||
the collector.
|
||||
|
||||
.DESCRIPTION
|
||||
The collector reads properties of Get-VBOJob / Get-VBOJobSession results whose names aren't
|
||||
fully documented. This prints those objects' real structure (property names, types, values and
|
||||
each date's Kind, a few levels deep) for a few jobs and their latest sessions. Nothing is
|
||||
changed. Output is also saved to a text file.
|
||||
|
||||
.EXAMPLE
|
||||
.\Get-Vb365CollectorDiagnostics.ps1
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[int]$Jobs = 3,
|
||||
[int]$SessionsPerJob = 2,
|
||||
[string]$OutFile = (Join-Path $env:TEMP 'vb365-collector-diagnostics.txt')
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$MaxDepth = 4
|
||||
|
||||
if (-not (Get-Command Get-VBOJob -ErrorAction SilentlyContinue)) {
|
||||
if (Get-Module -ListAvailable -Name Veeam.Archiver.PowerShell) {
|
||||
Import-Module Veeam.Archiver.PowerShell
|
||||
} else {
|
||||
Import-Module (Join-Path $env:ProgramFiles 'Veeam\Backup365\Veeam.Archiver.PowerShell\Veeam.Archiver.PowerShell.psd1')
|
||||
}
|
||||
}
|
||||
|
||||
function Test-Scalar($Value) {
|
||||
return $null -eq $Value -or $Value -is [string] -or $Value -is [ValueType] -or $Value -is [guid]
|
||||
}
|
||||
|
||||
# Recursive "name (type) = value" listing. Collections show their first 3 items.
|
||||
function Show-Object($Value, [string]$Indent = ' ', [int]$Depth = 0) {
|
||||
if (Test-Scalar $Value) { return }
|
||||
if ($Depth -ge $MaxDepth) { "$Indent..."; return }
|
||||
if ($Value -is [Collections.IEnumerable]) {
|
||||
$items = @($Value)
|
||||
"$Indent[$($items.Count) item(s)]"
|
||||
$i = 0
|
||||
foreach ($item in $items | Select-Object -First 3) {
|
||||
"$Indent[$i] ($($item.GetType().FullName))$(if (Test-Scalar $item) { " = $item" })"
|
||||
Show-Object $item "$Indent " ($Depth + 1)
|
||||
$i++
|
||||
}
|
||||
return
|
||||
}
|
||||
foreach ($p in $Value.PSObject.Properties) {
|
||||
try { $v = $p.Value } catch { "$Indent$($p.Name) = <error: $($_.Exception.Message)>"; continue }
|
||||
$type = if ($null -eq $v) { 'null' } else { $v.GetType().FullName }
|
||||
if (Test-Scalar $v) {
|
||||
$kind = if ($v -is [datetime]) { " [Kind=$($v.Kind)]" } else { '' }
|
||||
"$Indent$($p.Name) ($type) = $v$kind"
|
||||
} else {
|
||||
$text = [string]$v
|
||||
if ($text.Length -gt 100) { $text = $text.Substring(0, 97) + '...' }
|
||||
"$Indent$($p.Name) ($type) = $text"
|
||||
Show-Object $v "$Indent " ($Depth + 1)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function Show-Section([string]$Title, [scriptblock]$Get) {
|
||||
"== $Title"
|
||||
try {
|
||||
$result = @(& $Get)
|
||||
if ($result.Count -eq 0 -or ($result.Count -eq 1 -and $null -eq $result[0])) { ' (nothing returned)' }
|
||||
$n = 0
|
||||
foreach ($item in $result) {
|
||||
if ($null -eq $item) { continue }
|
||||
" [$n] ($($item.GetType().FullName))"
|
||||
Show-Object $item ' '
|
||||
$n++
|
||||
}
|
||||
} catch {
|
||||
" error: $($_.Exception.Message)"
|
||||
}
|
||||
''
|
||||
}
|
||||
|
||||
$report = & {
|
||||
$module = Get-Module Veeam.Archiver.PowerShell
|
||||
"Generated $(Get-Date -Format o) on $env:COMPUTERNAME, PowerShell $($PSVersionTable.PSVersion), module $($module.Version)"
|
||||
"Local time zone: $([TimeZoneInfo]::Local.Id), UTC offset now $([TimeZoneInfo]::Local.GetUtcOffset((Get-Date))), culture $([Globalization.CultureInfo]::CurrentCulture.Name)"
|
||||
''
|
||||
"== Organizations"
|
||||
try {
|
||||
Get-VBOOrganization | ForEach-Object { " $($_.Name) [$($_.Type)]" }
|
||||
} catch { " error: $($_.Exception.Message)" }
|
||||
''
|
||||
$allJobs = @(Get-VBOJob)
|
||||
"== Jobs: $($allJobs.Count) in total"
|
||||
$allJobs | ForEach-Object { " $($_.Name) [$($_.Id)] enabled=$($_.IsEnabled) next=$($_.NextRun)" }
|
||||
''
|
||||
Show-Section "Get-VBOJob (first $Jobs, full detail)" { $allJobs | Select-Object -First $Jobs }
|
||||
foreach ($j in $allJobs | Select-Object -First $Jobs) {
|
||||
Show-Section "Get-VBOJobSession -Job '$($j.Name)' (latest $SessionsPerJob)" {
|
||||
Get-VBOJobSession -Job $j | Sort-Object CreationTime -Descending | Select-Object -First $SessionsPerJob
|
||||
}
|
||||
}
|
||||
if (Get-Command Get-VBOCopyJob -ErrorAction SilentlyContinue) {
|
||||
Show-Section 'Get-VBOCopyJob (first 2)' { Get-VBOCopyJob | Select-Object -First 2 }
|
||||
if (Get-Command Get-VBOCopyJobSession -ErrorAction SilentlyContinue) {
|
||||
Show-Section 'Get-VBOCopyJobSession (latest 2)' {
|
||||
Get-VBOCopyJobSession | Sort-Object CreationTime -Descending | Select-Object -First 2
|
||||
}
|
||||
}
|
||||
} else {
|
||||
'== Get-VBOCopyJob: not available'
|
||||
}
|
||||
} | Out-String -Width 300
|
||||
|
||||
$report
|
||||
$report | Set-Content -Path $OutFile -Encoding UTF8
|
||||
Write-Host "Saved to $OutFile"
|
||||
@@ -0,0 +1,74 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Installs the Veeam Backup for Microsoft 365 collector as a SYSTEM scheduled task.
|
||||
|
||||
.DESCRIPTION
|
||||
- Copies Send-Vb365BackupStatus.ps1 to Program Files (only admins can modify it, which
|
||||
matters because it runs as SYSTEM).
|
||||
- Writes the config to ProgramData\BackupDashboard and restricts that folder to SYSTEM and
|
||||
Administrators, since it contains the API key.
|
||||
- Registers a scheduled task that runs every IntervalMinutes, under Windows PowerShell 5.1
|
||||
(powershell.exe) unless -PwshPath points at PowerShell 7.
|
||||
Safe to re-run to change settings or update the script. Can sit alongside the other collectors.
|
||||
|
||||
Dashboard company per job: -OrganizationTenants maps Microsoft 365 organization names to
|
||||
companies; jobs of unmapped organizations use -Tenant, or the organization's name if no -Tenant.
|
||||
|
||||
.EXAMPLE
|
||||
.\Install-Vb365Collector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
One client per VB365 server. Prompts for the API key.
|
||||
.EXAMPLE
|
||||
.\Install-Vb365Collector.ps1 -ApiUrl https://backups.example.com -OrganizationTenants @{
|
||||
'acme.onmicrosoft.com' = 'Acme Corp'; 'bluewater.onmicrosoft.com' = 'Bluewater Dental' }
|
||||
A shared VB365 server protecting several organizations.
|
||||
#>
|
||||
#Requires -RunAsAdministrator
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory)][string]$ApiUrl,
|
||||
[string]$Tenant,
|
||||
[hashtable]$OrganizationTenants = @{},
|
||||
[string]$SourceSystem = 'Veeam Backup for M365',
|
||||
[ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
|
||||
[ValidateRange(1, 720)][int]$LookbackHours = 48,
|
||||
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard'),
|
||||
# Run under PowerShell 7 instead of Windows PowerShell, e.g. 'C:\Program Files\PowerShell\7\pwsh.exe'
|
||||
[string]$PwshPath
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$Shell = Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\powershell.exe'
|
||||
if ($PwshPath) {
|
||||
if (-not (Test-Path $PwshPath)) { throw "PowerShell 7 not found at '$PwshPath'." }
|
||||
$Shell = $PwshPath
|
||||
}
|
||||
|
||||
# Check VB365 is here before changing anything
|
||||
$moduleFile = Join-Path $env:ProgramFiles 'Veeam\Backup365\Veeam.Archiver.PowerShell\Veeam.Archiver.PowerShell.psd1'
|
||||
if (-not (Get-Module -ListAvailable -Name Veeam.Archiver.PowerShell) -and -not (Test-Path $moduleFile)) {
|
||||
throw 'The Veeam Backup for Microsoft 365 PowerShell module (Veeam.Archiver.PowerShell) is not installed on this server.'
|
||||
}
|
||||
if (-not $Tenant -and $OrganizationTenants.Count -eq 0) {
|
||||
Write-Warning 'No -Tenant or -OrganizationTenants: jobs will be listed under their Microsoft 365 organization name.'
|
||||
}
|
||||
|
||||
# Shared installer code: in the repo it's in ..\common, so copy the whole collectors folder
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
throw "BackupDashboard.Collector.psm1 not found. Copy the whole collectors folder, including common."
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
$settings = [ordered]@{
|
||||
SourceSystem = $SourceSystem
|
||||
LookbackHours = $LookbackHours
|
||||
OrganizationTenants = $OrganizationTenants
|
||||
}
|
||||
if ($Tenant) { $settings.Tenant = $Tenant }
|
||||
Install-CollectorTask -Name 'vb365' -ScriptPath (Join-Path $PSScriptRoot 'Send-Vb365BackupStatus.ps1') `
|
||||
-TaskName 'Backup Dashboard - VB365 collector' `
|
||||
-TaskDescription 'Sends Veeam Backup for Microsoft 365 results to the Backup Dashboard' `
|
||||
-Shell $Shell -ApiUrl $ApiUrl -IntervalMinutes $IntervalMinutes -InstallDir $InstallDir -Settings $settings
|
||||
@@ -0,0 +1,88 @@
|
||||
# Veeam Backup for Microsoft 365 collector
|
||||
|
||||
A PowerShell script that runs on the Veeam Backup for Microsoft 365 (VB365) server and reports its
|
||||
backup jobs (and backup copy jobs) to the dashboard, one dashboard job per VB365 job.
|
||||
|
||||
> **Checked against:** VB365 8.4 under Windows PowerShell 5.1, a shared server protecting 11
|
||||
> organizations: successful, warning and retried sessions. Not yet seen on a real server: a failed
|
||||
> session, a session in progress, or a backup copy job. If one looks wrong, run
|
||||
> `Get-Vb365CollectorDiagnostics.ps1` while it's in that state and send the output.
|
||||
|
||||
## Which company a job appears under
|
||||
|
||||
One VB365 server can protect several Microsoft 365 organizations, so the company is picked per job
|
||||
from the job's organization:
|
||||
|
||||
1. `OrganizationTenants` maps an organization name (as `Get-VBOOrganization` shows it, e.g.
|
||||
`acme.onmicrosoft.com`; not case-sensitive) to a dashboard company.
|
||||
2. Otherwise `Tenant`, for a VB365 server dedicated to one client.
|
||||
3. Otherwise the organization's own name.
|
||||
|
||||
Changing a job's company later creates a new dashboard job under the new company; delete the old
|
||||
one with `python manage.py delete-job <id>`.
|
||||
|
||||
## Install (on the VB365 server, as an administrator)
|
||||
|
||||
1. Create a key for this server on the dashboard's **Admin** page, or on the dashboard host:
|
||||
```bash
|
||||
python manage.py create-key vb365-01
|
||||
```
|
||||
2. Copy the whole `collectors` folder to the server (the installer needs `common`
|
||||
as well as this folder), then from `collectors\vb365` run one of:
|
||||
```powershell
|
||||
# VB365 server for one client
|
||||
.\Install-Vb365Collector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
|
||||
# Shared VB365 server
|
||||
.\Install-Vb365Collector.ps1 -ApiUrl https://backups.example.com -OrganizationTenants @{
|
||||
'acme.onmicrosoft.com' = 'Acme Corp'
|
||||
'bluewater.onmicrosoft.com' = 'Bluewater Dental'
|
||||
}
|
||||
```
|
||||
It prompts for the key, then:
|
||||
- writes `C:\ProgramData\BackupDashboard\vb365-collector.json` (readable only by SYSTEM and Administrators)
|
||||
- copies the script to `C:\Program Files\BackupDashboard\`
|
||||
- registers the scheduled task **Backup Dashboard - VB365 collector** (SYSTEM, every 60 minutes)
|
||||
|
||||
It runs under Windows PowerShell 5.1. If your VB365 version's module needs PowerShell 7, add
|
||||
`-PwshPath 'C:\Program Files\PowerShell\7\pwsh.exe'`.
|
||||
3. Preview the payloads, then backfill 30 days so the history view is populated straight away:
|
||||
```powershell
|
||||
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-Vb365BackupStatus.ps1' -DryRun
|
||||
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-Vb365BackupStatus.ps1' -LookbackHours 720
|
||||
```
|
||||
|
||||
It can be installed alongside the other collectors: it uses its own config, state, log and task.
|
||||
Re-run the installer to change settings (including the organization mapping) or update the script. It also installs
|
||||
`BackupDashboard.Collector.psm1`, the code shared by all collectors, next to the script.
|
||||
|
||||
## What it sends
|
||||
|
||||
| Dashboard | VB365 |
|
||||
|---|---|
|
||||
| Company | See above |
|
||||
| Job | Each backup job and backup copy job. The job's GUID is the ID, so renaming it in VB365 doesn't duplicate it |
|
||||
| Status | Session status: Success / Warning / Failed. Running or queued → **Running**. Stopped (cancelled) or unrecognised → **Warning** |
|
||||
| Started / finished | Session `CreationTime` / `EndTime` |
|
||||
| Next scheduled | Job's `NextRun`. Empty for disabled jobs |
|
||||
| Message | Warning and error lines from the session log (items titled `[Warning] ...`, `[Failed] ...`), plus "Veeam will retry this job." when a retry is pending |
|
||||
| Retries | VB365 runs a retry as a separate session, so it appears as its own run, marked `details.retry_of` (the session it retried). The retried run is marked `details.will_retry` |
|
||||
| Size | Data transferred (`Statistics.TransferredData`) |
|
||||
| Details | Organization, objects processed, job kind (`backup` / `copy`) |
|
||||
|
||||
Each run covers the last 48 hours (`LookbackHours`). Sessions already delivered are recorded in
|
||||
`vb365-collector.state.json`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Log: `C:\ProgramData\BackupDashboard\vb365-collector.log`. Exit codes: `0` OK, `1` some jobs
|
||||
failed to send (retried next run), `2` setup failed (config or VB365 module).
|
||||
- Wrong status, times or messages: run `.\Get-Vb365CollectorDiagnostics.ps1` (read-only). It prints
|
||||
the real structure of jobs and sessions, including whether each date is local or UTC.
|
||||
|
||||
## Testing without VB365
|
||||
|
||||
```powershell
|
||||
. ./tests/MockVb365.ps1
|
||||
./Send-Vb365BackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||
```
|
||||
@@ -0,0 +1,278 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Reports Veeam Backup for Microsoft 365 (VB365) job results to the Backup Dashboard.
|
||||
|
||||
.DESCRIPTION
|
||||
Runs on the VB365 server, normally as a SYSTEM scheduled task every 60 minutes
|
||||
(see Install-Vb365Collector.ps1). For every backup job (and backup copy job) it sends:
|
||||
- the job's next scheduled run, which drives the dashboard's Overdue flag, and
|
||||
- every session from the last LookbackHours that the dashboard hasn't already received in
|
||||
its final state. Running sessions are re-sent each time until they finish.
|
||||
|
||||
One VB365 server can protect several Microsoft 365 organizations, so the dashboard company is
|
||||
chosen per job from its organization: OrganizationTenants in the config maps an organization
|
||||
name (e.g. "contoso.onmicrosoft.com") to a company; otherwise Tenant is used; otherwise the
|
||||
organization's own name.
|
||||
|
||||
Already-reported sessions are tracked in a state file, and the ingest API is idempotent, so a
|
||||
failed or repeated run is harmless: anything not confirmed is retried next time.
|
||||
|
||||
.PARAMETER ConfigPath
|
||||
JSON config with ApiUrl and ApiKey (optional: Tenant, OrganizationTenants, SourceSystem,
|
||||
LookbackHours).
|
||||
|
||||
.PARAMETER LookbackHours
|
||||
Overrides the config value. Use 720 once to backfill 30 days of history.
|
||||
|
||||
.PARAMETER DryRun
|
||||
Print the payloads instead of sending them. Doesn't touch the state or log files.
|
||||
|
||||
.EXAMPLE
|
||||
.\Send-Vb365BackupStatus.ps1 -DryRun
|
||||
.EXAMPLE
|
||||
.\Send-Vb365BackupStatus.ps1 -LookbackHours 720
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$ConfigPath = (Join-Path $env:ProgramData 'BackupDashboard\vb365-collector.json'),
|
||||
[int]$LookbackHours = 0,
|
||||
[switch]$DryRun
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# Shared collector code: installed next to this script; in the repo, in ..\common
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
Write-Host "BackupDashboard.Collector.psm1 not found next to $PSCommandPath; re-run the installer."
|
||||
exit 2
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
$Invariant = [Globalization.CultureInfo]::InvariantCulture
|
||||
|
||||
# ---------- Helpers ----------
|
||||
|
||||
function Import-Vb365 {
|
||||
if (Get-Command Get-VBOJob -ErrorAction SilentlyContinue) { return } # already loaded
|
||||
if (Get-Module -ListAvailable -Name Veeam.Archiver.PowerShell) {
|
||||
Import-Module Veeam.Archiver.PowerShell
|
||||
return
|
||||
}
|
||||
# Fall back to the default install path
|
||||
Import-Module (Join-Path $env:ProgramFiles 'Veeam\Backup365\Veeam.Archiver.PowerShell\Veeam.Archiver.PowerShell.psd1')
|
||||
}
|
||||
|
||||
function Get-Time($Object, [string]$Field) {
|
||||
$value = Get-FirstValue $Object @($Field)
|
||||
if ($null -eq $value) { return $null }
|
||||
$dt = [datetime]$value
|
||||
if ($dt.Year -le 1900) { return $null } # e.g. EndTime while a session is running
|
||||
return $dt
|
||||
}
|
||||
|
||||
# "1.5 GB", "512 MB", "123456" -> bytes; $null when it can't be read
|
||||
function ConvertTo-Bytes($Value) {
|
||||
if ($null -eq $Value) { return $null }
|
||||
if ($Value -is [ValueType]) { return [long]$Value }
|
||||
$match = [regex]::Match([string]$Value, '^\s*([\d.,]+)\s*([KMGTP]?B)?\s*$', 'IgnoreCase')
|
||||
if (-not $match.Success) { return $null }
|
||||
$number = 0.0
|
||||
if (-not [double]::TryParse(($match.Groups[1].Value -replace ',', '.'), [Globalization.NumberStyles]::Float, $Invariant, [ref]$number)) {
|
||||
return $null
|
||||
}
|
||||
$power = @{ '' = 0; 'B' = 0; 'KB' = 1; 'MB' = 2; 'GB' = 3; 'TB' = 4; 'PB' = 5 }[$match.Groups[2].Value.ToUpperInvariant()]
|
||||
return [long]($number * [math]::Pow(1024, $power))
|
||||
}
|
||||
|
||||
# ---------- Jobs and tenants ----------
|
||||
|
||||
function Get-OrganizationName($Job) {
|
||||
$org = Get-FirstValue $Job @('Organization')
|
||||
if ($null -eq $org) {
|
||||
# Copy jobs point at their source backup job
|
||||
$org = Get-FirstValue (Get-FirstValue $Job @('BackupJob')) @('Organization')
|
||||
}
|
||||
if ($org -is [string]) { return $org }
|
||||
$name = Get-FirstValue $org @('Name', 'OfficeName', 'DisplayName')
|
||||
if ($name) { return [string]$name }
|
||||
return $null
|
||||
}
|
||||
|
||||
function Get-TenantForJob($Job) {
|
||||
$org = Get-OrganizationName $Job
|
||||
if ($org -and $Config.OrganizationTenants) {
|
||||
$mapped = $Config.OrganizationTenants.PSObject.Properties[$org] # case-insensitive lookup
|
||||
if ($mapped -and $mapped.Value) { return [string]$mapped.Value }
|
||||
}
|
||||
if ($Config.Tenant) { return [string]$Config.Tenant }
|
||||
if ($org) { return $org }
|
||||
return $env:COMPUTERNAME
|
||||
}
|
||||
|
||||
function Get-NextRun($Job) {
|
||||
try {
|
||||
$enabled = Get-FirstValue $Job @('IsEnabled', 'Enabled')
|
||||
if ($enabled -eq $false) { return $null }
|
||||
return ConvertTo-IsoTime (Get-Time $Job 'NextRun')
|
||||
} catch {
|
||||
Write-Log "Couldn't read schedule for job '$($Job.Name)': $(Get-ErrorText $_)" 'WARN'
|
||||
return $null
|
||||
}
|
||||
}
|
||||
|
||||
# ---------- Sessions ----------
|
||||
|
||||
function Get-RunStatus($Session) {
|
||||
$status = [string](Get-FirstValue $Session @('Status', 'Result', 'State'))
|
||||
if ($status -match 'Fail|Error') { return 'failed' }
|
||||
if ($status -match 'Warn') { return 'warning' }
|
||||
if ($status -match 'Succe') { return 'success' }
|
||||
if ($status -match 'Run|Queue|Progress|Start|Wait') { return 'running' }
|
||||
if (-not (Get-Time $Session 'EndTime')) { return 'running' }
|
||||
# Stopped (cancelled), Disconnected, or a state we don't recognise: surface it rather than hide it
|
||||
return 'warning'
|
||||
}
|
||||
|
||||
function Test-EmptyGuid($Value) {
|
||||
return (-not $Value) -or [string]$Value -eq [string][guid]::Empty
|
||||
}
|
||||
|
||||
# Warning/error lines from the session log. Each log item's status is a prefix of its Title, e.g.
|
||||
# "[Warning] Item was skipped" (there's no separate status property), so match on that rather than
|
||||
# on words in the text, which would catch e.g. "[Success] Processing site: .../error-reports".
|
||||
function Get-RunMessage($Session, [string]$Status) {
|
||||
if ($Status -eq 'running') { return '' }
|
||||
$lines = New-Object 'System.Collections.Generic.List[string]'
|
||||
$sessionStatus = [string](Get-FirstValue $Session @('Status', 'Result', 'State'))
|
||||
if ($Status -ne 'success') {
|
||||
foreach ($entry in @(Get-FirstValue $Session @('Log', 'Logs') | Where-Object { $_ })) {
|
||||
$title = ([string](Get-FirstValue $entry @('Title', 'Message', 'Text'))).Trim()
|
||||
$prefix = [regex]::Match($title, '^\[(\w+)\]\s*')
|
||||
if ($prefix.Success) {
|
||||
if ($prefix.Groups[1].Value -notmatch 'Warn|Fail|Error') { continue }
|
||||
$title = $title.Substring($prefix.Length)
|
||||
} elseif ([string](Get-FirstValue $entry @('Status', 'Type', 'Severity')) -notmatch 'Warn|Fail|Error') {
|
||||
continue
|
||||
}
|
||||
if ($title -and -not $lines.Contains($title)) { $lines.Add($title) }
|
||||
}
|
||||
}
|
||||
if ($lines.Count -eq 0) {
|
||||
if ($Status -eq 'success') { $lines.Add('Job completed successfully.') }
|
||||
else { $lines.Add("Job finished with status '$sessionStatus'.") }
|
||||
} elseif ($sessionStatus -and $sessionStatus -notmatch 'Fail|Warn') {
|
||||
$lines.Insert(0, "Session status: $sessionStatus")
|
||||
}
|
||||
# A warning/failure VB365 is about to retry; the retry arrives as its own run
|
||||
if ($Status -ne 'success' -and (Get-FirstValue $Session @('JobWillBeRetried')) -eq $true) {
|
||||
$lines.Add('Veeam will retry this job.')
|
||||
}
|
||||
return ($lines -join "`n")
|
||||
}
|
||||
|
||||
function ConvertTo-Run($Session, $Job, [string]$JobKind) {
|
||||
$status = Get-RunStatus $Session
|
||||
$details = [ordered]@{ vb365_server = $env:COMPUTERNAME; job_kind = $JobKind }
|
||||
$org = Get-OrganizationName $Job
|
||||
if ($org) { $details.organization = $org }
|
||||
$sessionStatus = Get-FirstValue $Session @('Status', 'Result', 'State')
|
||||
if ($sessionStatus) { $details.vb365_status = [string]$sessionStatus }
|
||||
# Retries are separate sessions pointing at the session they retry
|
||||
$mainSession = Get-FirstValue $Session @('MainSessionId')
|
||||
if (-not (Test-EmptyGuid $mainSession)) { $details.retry_of = [string]$mainSession }
|
||||
if ((Get-FirstValue $Session @('JobWillBeRetried')) -eq $true) { $details.will_retry = $true }
|
||||
|
||||
$run = [ordered]@{
|
||||
external_id = [string](Get-FirstValue $Session @('Id', 'SessionId'))
|
||||
status = $status
|
||||
started_at = ConvertTo-IsoTime (Get-Time $Session 'CreationTime')
|
||||
message = Get-RunMessage $Session $status
|
||||
details = $details
|
||||
}
|
||||
if ($status -ne 'running') {
|
||||
$finished = ConvertTo-IsoTime (Get-Time $Session 'EndTime')
|
||||
if ($finished) { $run.finished_at = $finished } # otherwise the API uses receipt time
|
||||
$stats = Get-FirstValue $Session @('Statistics')
|
||||
$processed = Get-FirstValue $stats @('ProcessedObjects')
|
||||
if ($null -ne $processed) { $details.processed_objects = $processed }
|
||||
$transferred = ConvertTo-Bytes (Get-FirstValue $stats @('TransferredData', 'TransferredDataSize'))
|
||||
if ($transferred -gt 0) { $run.size_bytes = $transferred }
|
||||
}
|
||||
return $run
|
||||
}
|
||||
|
||||
# ---------- Main ----------
|
||||
|
||||
try {
|
||||
$Config = Initialize-Collector -Name 'vb365' -ConfigPath $ConfigPath -LookbackHours $LookbackHours -DryRun:$DryRun -RequiredKeys 'ApiUrl', 'ApiKey'
|
||||
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Veeam Backup for M365' }
|
||||
$LookbackHours = $Config.LookbackHours
|
||||
|
||||
Import-Vb365
|
||||
|
||||
$since = (Get-Date).AddHours(-$LookbackHours)
|
||||
|
||||
# (job, kind, sessions) for backup jobs, plus backup copy jobs where the cmdlets exist
|
||||
$work = New-Object 'System.Collections.Generic.List[object]'
|
||||
foreach ($job in @(Get-VBOJob)) {
|
||||
$sessions = @(Get-VBOJobSession -Job $job | Where-Object { (Get-Time $_ 'CreationTime') -ge $since })
|
||||
$work.Add([pscustomobject]@{ Job = $job; Kind = 'backup'; Sessions = $sessions })
|
||||
}
|
||||
if ((Get-Command Get-VBOCopyJob -ErrorAction SilentlyContinue) -and (Get-Command Get-VBOCopyJobSession -ErrorAction SilentlyContinue)) {
|
||||
$copyJobs = @(Get-VBOCopyJob)
|
||||
if ($copyJobs.Count) {
|
||||
# Fetch once and match by job ID, since the per-job parameter name isn't consistent
|
||||
$copySessions = @(Get-VBOCopyJobSession | Where-Object { (Get-Time $_ 'CreationTime') -ge $since })
|
||||
foreach ($job in $copyJobs) {
|
||||
$id = [string]$job.Id
|
||||
$work.Add([pscustomobject]@{
|
||||
Job = $job; Kind = 'copy'
|
||||
Sessions = @($copySessions | Where-Object { [string](Get-FirstValue $_ @('JobId')) -eq $id })
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
Write-Log "Collector setup failed: $(Get-ErrorText $_)" 'ERROR'
|
||||
exit 2
|
||||
}
|
||||
|
||||
$failures = 0
|
||||
foreach ($item in $work) {
|
||||
$job = $item.Job
|
||||
$tenant = Get-TenantForJob $job
|
||||
$jobInfo = [ordered]@{
|
||||
external_id = [string]$job.Id
|
||||
name = [string]$job.Name
|
||||
# Always sent (null when disabled/unscheduled) so the dashboard's value tracks VB365's
|
||||
next_run_at = Get-NextRun $job
|
||||
}
|
||||
$pending = @($item.Sessions |
|
||||
Where-Object { $_ -and -not (Test-RunSent ([string](Get-FirstValue $_ @('Id', 'SessionId')))) } |
|
||||
Sort-Object { Get-Time $_ 'CreationTime' })
|
||||
|
||||
try {
|
||||
if ($pending.Count -eq 0) {
|
||||
# Keep next_run_at fresh even when nothing new ran
|
||||
Send-JobReport -Tenant $tenant -SourceSystem $SourceSystem -Job $jobInfo
|
||||
}
|
||||
foreach ($session in $pending) {
|
||||
$run = ConvertTo-Run $session $job $item.Kind
|
||||
Send-JobReport -Tenant $tenant -SourceSystem $SourceSystem -Job $jobInfo -Run $run -SentKey $run.external_id -StartedAt ((Get-Time $session 'CreationTime'))
|
||||
}
|
||||
} catch {
|
||||
$failures++
|
||||
Write-Log "Failed to report job '$($job.Name)': $(Get-ErrorText $_)" 'ERROR'
|
||||
if (Test-AuthRejected $_) {
|
||||
Write-Log 'API key rejected; check ApiKey in the config. Stopping.' 'ERROR'
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$exitCode = Complete-Collector -JobCount $work.Count -Failures $failures
|
||||
if ($exitCode) { exit $exitCode }
|
||||
@@ -0,0 +1,79 @@
|
||||
# Stand-ins for the Veeam Backup for Microsoft 365 cmdlets. Object shapes follow
|
||||
# Get-Vb365CollectorDiagnostics.ps1 output from a real VB365 8.4 server: session and NextRun times
|
||||
# are local, Statistics.TransferredData is a string ("187.2 MB"), log items carry their status as a
|
||||
# Title prefix ("[Warning] ..."), and retries are separate sessions whose MainSessionId points at
|
||||
# the session they retry (which has JobWillBeRetried = True).
|
||||
#
|
||||
# . ./tests/MockVb365.ps1
|
||||
# ./Send-Vb365BackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||
|
||||
# Fixed to the top of the hour, and IDs derived from names/times, so repeated runs see the same data
|
||||
# MOCK_NOW (ISO 8601, set by the test suite) pins the time everything is relative to
|
||||
$now = if ($env:MOCK_NOW) { [datetime]$env:MOCK_NOW } else { Get-Date -Minute 0 -Second 0 -Millisecond 0 }
|
||||
function New-StableGuid([string]$Seed) {
|
||||
[guid]::new([Security.Cryptography.MD5]::Create().ComputeHash([Text.Encoding]::UTF8.GetBytes($Seed)))
|
||||
}
|
||||
|
||||
function New-MockOrg($Name) { [pscustomobject]@{ Name = $Name; OfficeName = $Name; Type = 'Office365'; IsBackedUp = $true } }
|
||||
$acme = New-MockOrg 'acme.onmicrosoft.com'
|
||||
$blue = New-MockOrg 'bluewater.onmicrosoft.com'
|
||||
|
||||
function New-MockJob($Name, $Org, $NextRun, [bool]$Enabled = $true) {
|
||||
[pscustomobject]@{
|
||||
Id = New-StableGuid $Name; Name = $Name; Organization = $Org; JobType = 'Backup'
|
||||
IsEnabled = $Enabled; NextRun = $NextRun; LastStatus = 'Success'
|
||||
LastRun = $(if ($NextRun) { $NextRun.AddDays(-1).ToUniversalTime() }) # UTC, as on the real server
|
||||
}
|
||||
}
|
||||
|
||||
function New-LogItem($Status, $Text, $At) {
|
||||
[pscustomobject]@{ Id = [guid]::NewGuid(); USN = 1; Title = "[$Status] $Text"; CreationTime = $At; EndTime = $At }
|
||||
}
|
||||
|
||||
function New-MockSession($Job, $Start, $End, $Status, $Transferred = '0 B', [long]$Objects = 0, $Log = @(),
|
||||
$MainSessionId = [guid]::Empty, [bool]$WillRetry = $false) {
|
||||
$logItems = @((New-LogItem 'Success' "Job started at: $($Start.ToString('G'))" $Start)) + @($Log)
|
||||
[pscustomobject]@{
|
||||
JobId = $Job.Id; JobName = $Job.Name; Status = $Status
|
||||
Statistics = [pscustomobject]@{ ProcessingRate = '1.6 MB/s (4 items/s)'; TransferredData = $Transferred; ProcessedObjects = $Objects; Bottleneck = 'Target' }
|
||||
JobSessionConfigType = 'Incremental'
|
||||
Id = New-StableGuid "$($Job.Name)|$($Start.Ticks)"
|
||||
CreationTime = $Start
|
||||
EndTime = $(if ($End) { $End } else { [datetime]::MinValue })
|
||||
Progress = 49; MainSessionId = $MainSessionId; JobWillBeRetried = $WillRetry
|
||||
Log = $logItems
|
||||
}
|
||||
}
|
||||
|
||||
$global:MockVboJobs = @(
|
||||
(New-MockJob 'Acme 365 Backup' $acme $now.Date.AddHours(17).AddDays($(if ($now.Hour -ge 17) { 1 } else { 0 }))),
|
||||
(New-MockJob 'Bluewater 365 Backup' $blue $now.Date.AddHours(17.75).AddDays($(if ($now.Hour -ge 17) { 1 } else { 0 }))),
|
||||
(New-MockJob 'Bluewater Old Mailboxes' $blue $now.AddDays(1) $false) # disabled
|
||||
)
|
||||
$acmeJob, $blueJob, $oldJob = $global:MockVboJobs
|
||||
|
||||
$y = $now.Date.AddDays(-1) # yesterday
|
||||
$blueWarn = New-MockSession $blueJob $y.AddHours(17.75) $y.AddHours(17.75).AddMinutes(3.5) 'Warning' '32.9 MB' 47 @(
|
||||
(New-LogItem 'Success' 'Found 94 objects' $y.AddHours(17.75)),
|
||||
# A success line mentioning "error" must not end up in the message
|
||||
(New-LogItem 'Success' 'Processing site: https://bluewater.sharepoint.com/sites/error-reports' $y.AddHours(17.76)),
|
||||
(New-LogItem 'Warning' 'Processing mailbox [email protected] finished with warning: Item "Q3.pst" exceeds the maximum size' $y.AddHours(17.78))
|
||||
) -WillRetry $true
|
||||
|
||||
$global:MockVboSessions = @(
|
||||
(New-MockSession $acmeJob $y.AddHours(17) $y.AddHours(17).AddMinutes(4.8) 'Success' '187.2 MB' 568 @(
|
||||
(New-LogItem 'Success' 'Found 49 objects' $y.AddHours(17)), (New-LogItem 'Success' 'Connected to organization' $y.AddHours(17)))),
|
||||
(New-MockSession $acmeJob $now.AddMinutes(-20) $null 'Running' '0 B' 12),
|
||||
$blueWarn,
|
||||
# Its retry 13 minutes later, which succeeded
|
||||
(New-MockSession $blueJob $y.AddHours(17.75).AddMinutes(13.5) $y.AddHours(17.75).AddMinutes(16.8) 'Success' '0 B' 0 -MainSessionId $blueWarn.Id),
|
||||
(New-MockSession $blueJob $now.AddHours(-30) $now.AddHours(-29.9) 'Failed' '0 B' 0 @(
|
||||
(New-LogItem 'Failed' 'Failed to connect to organization: The remote server returned an error: (401) Unauthorized.' $now.AddHours(-30)))),
|
||||
(New-MockSession $oldJob $now.AddDays(-5) $now.AddDays(-5).AddHours(1) 'Success') # outside 48h lookback
|
||||
)
|
||||
|
||||
function Get-VBOJob { $global:MockVboJobs }
|
||||
function Get-VBOJobSession { param($Job) $global:MockVboSessions | Where-Object { -not $Job -or $_.JobId -eq $Job.Id } }
|
||||
function Get-VBOOrganization { $acme, $blue }
|
||||
function Get-VBOCopyJob { } # none, as on the real server
|
||||
function Get-VBOCopyJobSession { }
|
||||
@@ -0,0 +1,188 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Read-only report of how Veeam jobs and sessions relate, for troubleshooting the collector.
|
||||
|
||||
.DESCRIPTION
|
||||
The collector matches sessions to jobs by Session.JobId = Job.Id. When a job shows as
|
||||
"Unknown" on the dashboard, no session matched. This lists:
|
||||
1. every job from Get-VBRJob and how many sessions matched it
|
||||
2. sessions whose JobId matches no job (the runs that are going missing)
|
||||
3. for jobs with no matching sessions, their type/link-related properties and methods
|
||||
4. what other job cmdlets (e.g. Get-VBRBackupCopyJob) return
|
||||
5. each job's schedule properties, which the collector reads for the next run
|
||||
Nothing is changed in Veeam or the dashboard. Output is also saved to a text file.
|
||||
|
||||
.EXAMPLE
|
||||
.\Get-VeeamCollectorDiagnostics.ps1
|
||||
.EXAMPLE
|
||||
.\Get-VeeamCollectorDiagnostics.ps1 -LookbackHours 168
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[int]$LookbackHours = 48,
|
||||
[string]$OutFile = (Join-Path $env:TEMP 'veeam-collector-diagnostics.txt')
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$LinkPattern = 'Parent|Linked|Worker|Child|Policy|Source|Target|Type|Cloud|Mode|Session'
|
||||
|
||||
if (-not (Get-Command Get-VBRJob -ErrorAction SilentlyContinue)) {
|
||||
if (Get-Module -ListAvailable -Name Veeam.Backup.PowerShell) {
|
||||
Import-Module Veeam.Backup.PowerShell -WarningAction SilentlyContinue
|
||||
} else {
|
||||
Add-PSSnapin VeeamPSSnapin
|
||||
}
|
||||
}
|
||||
|
||||
function Format-Value($Value) {
|
||||
if ($null -eq $Value) { return '<null>' }
|
||||
$text = ($Value | ForEach-Object { [string]$_ }) -join ', '
|
||||
if ($text.Length -gt 120) { $text = $text.Substring(0, 117) + '...' }
|
||||
return $text
|
||||
}
|
||||
|
||||
# Property and method names matching $LinkPattern, with property values. Every read is guarded:
|
||||
# some Veeam properties throw when accessed.
|
||||
function Show-LinkMembers($Object, [string]$Pattern = $LinkPattern, [string]$Indent = ' ') {
|
||||
foreach ($p in $Object.PSObject.Properties | Where-Object { $_.Name -match $Pattern }) {
|
||||
$value = try { Format-Value $p.Value } catch { "<error: $($_.Exception.Message)>" }
|
||||
'{0}{1} = {2}' -f $Indent, $p.Name, $value
|
||||
}
|
||||
$methods = @($Object | Get-Member -MemberType Methods -ErrorAction SilentlyContinue |
|
||||
Where-Object { $_.Name -match $Pattern } | ForEach-Object Name)
|
||||
if ($methods) { $Indent + 'methods: ' + ($methods -join ', ') }
|
||||
}
|
||||
|
||||
function Get-Prop($Object, [string]$Name) {
|
||||
try { Format-Value $Object.$Name } catch { '<error>' }
|
||||
}
|
||||
|
||||
$report = & {
|
||||
$vbrVersion = try {
|
||||
(Get-Item (Join-Path ${env:ProgramFiles} 'Veeam\Backup and Replication\Backup\Veeam.Backup.Core.dll')).VersionInfo.ProductVersion
|
||||
} catch { 'unknown' }
|
||||
"Generated $(Get-Date -Format o) on $env:COMPUTERNAME, VBR $vbrVersion, PowerShell $($PSVersionTable.PSVersion)"
|
||||
"Lookback: $LookbackHours hours"
|
||||
''
|
||||
|
||||
$since = (Get-Date).AddHours(-$LookbackHours)
|
||||
$jobs = @(Get-VBRJob -WarningAction SilentlyContinue)
|
||||
$sessions = @(Get-VBRBackupSession -WarningAction SilentlyContinue | Where-Object { $_.CreationTime -ge $since })
|
||||
$jobIds = @{}
|
||||
foreach ($j in $jobs) { $jobIds[$j.Id.ToString()] = $j }
|
||||
|
||||
'== 1. Jobs from Get-VBRJob'
|
||||
$jobs | ForEach-Object {
|
||||
$id = $_.Id.ToString()
|
||||
[pscustomobject]@{
|
||||
Name = $_.Name
|
||||
JobType = Get-Prop $_ 'JobType'
|
||||
TypeToString = Get-Prop $_ 'TypeToString'
|
||||
Id = $id
|
||||
MatchingSessions = @($sessions | Where-Object { $_.JobId.ToString() -eq $id }).Count
|
||||
}
|
||||
} | Format-Table -AutoSize | Out-String -Width 300
|
||||
|
||||
'== 2. Sessions whose JobId matches no job from Get-VBRJob'
|
||||
$orphans = @($sessions | Where-Object { -not $jobIds.ContainsKey($_.JobId.ToString()) })
|
||||
if ($orphans.Count -eq 0) {
|
||||
' (none)'
|
||||
} else {
|
||||
$orphans | Group-Object { $_.JobId.ToString() } | ForEach-Object {
|
||||
$latest = $_.Group | Sort-Object CreationTime | Select-Object -Last 1
|
||||
[pscustomobject]@{
|
||||
JobId = $_.Name
|
||||
JobName = Get-Prop $latest 'JobName'
|
||||
JobType = Get-Prop $latest 'JobType'
|
||||
Sessions = $_.Count
|
||||
LatestStart = $latest.CreationTime
|
||||
State = Get-Prop $latest 'State'
|
||||
Result = Get-Prop $latest 'Result'
|
||||
}
|
||||
} | Format-Table -AutoSize | Out-String -Width 300
|
||||
|
||||
' Link-related members of one orphan session:'
|
||||
Show-LinkMembers $orphans[0]
|
||||
''
|
||||
' Job object for that session (via GetJob(), if available):'
|
||||
try {
|
||||
$owner = $orphans[0].GetJob()
|
||||
' Name = {0}, Id = {1}' -f (Get-Prop $owner 'Name'), (Get-Prop $owner 'Id')
|
||||
Show-LinkMembers $owner
|
||||
} catch {
|
||||
" not available: $($_.Exception.Message)"
|
||||
}
|
||||
}
|
||||
''
|
||||
|
||||
'== 3. Jobs with no matching sessions in the window'
|
||||
$empty = @($jobs | Where-Object { $id = $_.Id.ToString(); -not ($sessions | Where-Object { $_.JobId.ToString() -eq $id }) })
|
||||
if ($empty.Count -eq 0) { ' (none)' }
|
||||
foreach ($j in $empty) {
|
||||
" $($j.Name) [$($j.Id)]"
|
||||
Show-LinkMembers $j
|
||||
try {
|
||||
$last = $j.FindLastSession()
|
||||
if ($last) {
|
||||
' FindLastSession(): JobId = {0}, JobName = {1}, Start = {2}, Result = {3}' -f `
|
||||
(Get-Prop $last 'JobId'), (Get-Prop $last 'JobName'), $last.CreationTime, (Get-Prop $last 'Result')
|
||||
} else {
|
||||
' FindLastSession(): <null>'
|
||||
}
|
||||
} catch {
|
||||
" FindLastSession(): not available: $($_.Exception.Message)"
|
||||
}
|
||||
''
|
||||
}
|
||||
|
||||
'== 4. Other job cmdlets'
|
||||
foreach ($cmd in 'Get-VBRBackupCopyJob', 'Get-VBRComputerBackupCopyJob', 'Get-VBRComputerBackupJob') {
|
||||
if (-not (Get-Command $cmd -ErrorAction SilentlyContinue)) { " ${cmd}: not available"; continue }
|
||||
try {
|
||||
$found = @(& $cmd -WarningAction SilentlyContinue)
|
||||
" ${cmd}: $($found.Count) job(s)"
|
||||
foreach ($f in $found) {
|
||||
' {0} [{1}] Type = {2}, Mode = {3}' -f (Get-Prop $f 'Name'), (Get-Prop $f 'Id'), (Get-Prop $f 'Type'), (Get-Prop $f 'Mode')
|
||||
}
|
||||
} catch {
|
||||
" ${cmd}: error: $($_.Exception.Message)"
|
||||
}
|
||||
}
|
||||
''
|
||||
|
||||
# The collector uses IsScheduleEnabled and ScheduleOptions.NextRun; show what else is there in
|
||||
# case this version keeps the next run somewhere different
|
||||
"== 5. Schedules (culture: $([Globalization.CultureInfo]::CurrentCulture.Name))"
|
||||
$SchedulePattern = 'Schedul|Next|Latest|Run|Enabled|Daily|Periodic|Monthly|Continuous|After|Option'
|
||||
foreach ($j in $jobs) {
|
||||
" $($j.Name) [$(Get-Prop $j 'JobType')]"
|
||||
Show-LinkMembers $j $SchedulePattern ' '
|
||||
foreach ($source in 'ScheduleOptions', 'GetScheduleOptions()', 'Get-VBRJobScheduleOptions') {
|
||||
try {
|
||||
$opts = switch ($source) {
|
||||
'ScheduleOptions' { $j.ScheduleOptions }
|
||||
'GetScheduleOptions()' { $j.GetScheduleOptions() }
|
||||
'Get-VBRJobScheduleOptions' { Get-VBRJobScheduleOptions -Job $j -WarningAction SilentlyContinue }
|
||||
}
|
||||
if ($null -eq $opts) { " ${source}: <null>"; continue }
|
||||
" ${source} ($($opts.GetType().FullName)):"
|
||||
foreach ($p in $opts.PSObject.Properties) {
|
||||
$value = try { $p.Value } catch { "<error: $($_.Exception.Message)>" }
|
||||
$type = if ($null -eq $value) { 'null' } else { $value.GetType().Name }
|
||||
if ($value -is [string] -or $value -is [ValueType] -or $null -eq $value) {
|
||||
' {0} ({1}) = {2}' -f $p.Name, $type, (Format-Value $value)
|
||||
} else {
|
||||
' {0} ({1})' -f $p.Name, $type
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
" ${source}: not available: $($_.Exception.Message)"
|
||||
}
|
||||
}
|
||||
''
|
||||
}
|
||||
} | Out-String -Width 300
|
||||
|
||||
$report
|
||||
$report | Set-Content -Path $OutFile -Encoding UTF8
|
||||
Write-Host "Saved to $OutFile"
|
||||
@@ -0,0 +1,68 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Installs the Veeam collector on a VBR server as a SYSTEM scheduled task.
|
||||
|
||||
.DESCRIPTION
|
||||
- Copies Send-VeeamBackupStatus.ps1 to Program Files (only admins can modify it, which
|
||||
matters because it runs as SYSTEM).
|
||||
- Writes the config to ProgramData\BackupDashboard and restricts that folder to SYSTEM and
|
||||
Administrators, since it contains the API key.
|
||||
- Registers a scheduled task that runs every IntervalMinutes under PowerShell 7 (pwsh.exe),
|
||||
which the Veeam module on VBR 13 expects. Install PowerShell 7 machine-wide (MSI or winget)
|
||||
first: Microsoft Store installs can't be run by SYSTEM.
|
||||
Safe to re-run to change settings or update the script.
|
||||
|
||||
.EXAMPLE
|
||||
.\Install-VeeamCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
(prompts for the API key)
|
||||
#>
|
||||
#Requires -RunAsAdministrator
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory)][string]$ApiUrl,
|
||||
[Parameter(Mandatory)][string]$Tenant,
|
||||
[string]$SourceSystem = 'Veeam B&R',
|
||||
[ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
|
||||
[ValidateRange(1, 720)][int]$LookbackHours = 48,
|
||||
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard'),
|
||||
# Defaults to the machine-wide PowerShell 7 install
|
||||
[string]$PwshPath
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# Find PowerShell 7 before changing anything, so a missing install fails cleanly
|
||||
if (-not $PwshPath) {
|
||||
$PwshPath = Join-Path $env:ProgramFiles 'PowerShell\7\pwsh.exe'
|
||||
if (-not (Test-Path $PwshPath)) {
|
||||
$onPath = Get-Command pwsh.exe -ErrorAction SilentlyContinue
|
||||
if ($onPath) { $PwshPath = $onPath.Source }
|
||||
}
|
||||
}
|
||||
if (-not (Test-Path $PwshPath)) {
|
||||
throw "PowerShell 7 not found at '$PwshPath'. Install it machine-wide (e.g. winget install Microsoft.PowerShell) or pass -PwshPath."
|
||||
}
|
||||
if ($PwshPath -match '\\WindowsApps\\') {
|
||||
throw "'$PwshPath' is a Microsoft Store install, which the SYSTEM account can't run. Install the MSI or winget version instead."
|
||||
}
|
||||
$pwshVersion = & $PwshPath -NoProfile -NonInteractive -Command '$PSVersionTable.PSVersion.ToString()'
|
||||
$pwshMajor = & $PwshPath -NoProfile -NonInteractive -Command '$PSVersionTable.PSVersion.Major'
|
||||
if ($LASTEXITCODE -or ($pwshMajor -as [int]) -lt 7) {
|
||||
throw "'$PwshPath' isn't a working PowerShell 7 (reported '$pwshVersion')."
|
||||
}
|
||||
|
||||
# Shared installer code: in the repo it's in ..\common, so copy the whole collectors folder
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
throw "BackupDashboard.Collector.psm1 not found. Copy the whole collectors folder, including common."
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
$settings = [ordered]@{ Tenant = $Tenant; SourceSystem = $SourceSystem; LookbackHours = $LookbackHours }
|
||||
Install-CollectorTask -Name 'veeam' -ScriptPath (Join-Path $PSScriptRoot 'Send-VeeamBackupStatus.ps1') `
|
||||
-TaskName 'Backup Dashboard - Veeam collector' `
|
||||
-TaskDescription 'Sends Veeam job results to the Backup Dashboard' `
|
||||
-Shell $PwshPath -ApiUrl $ApiUrl -IntervalMinutes $IntervalMinutes -InstallDir $InstallDir -Settings $settings
|
||||
@@ -0,0 +1,83 @@
|
||||
# Veeam Backup & Replication collector
|
||||
|
||||
A PowerShell script that runs on each client's VBR server and reports job results to the dashboard.
|
||||
It covers jobs returned by `Get-VBRJob`: VM backup, replication and backup copy jobs.
|
||||
|
||||
## Install (on the VBR server, as an administrator)
|
||||
|
||||
Requires PowerShell 7 installed machine-wide (`winget install Microsoft.PowerShell` or the MSI),
|
||||
which VBR 13's Veeam module expects. The Microsoft Store version won't work, because the SYSTEM
|
||||
account can't run it. The installer checks for this before changing anything.
|
||||
|
||||
1. Create a key for this server on the dashboard's **Admin** page, or on the dashboard host:
|
||||
```bash
|
||||
python manage.py create-key veeam-acme
|
||||
```
|
||||
2. Copy the whole `collectors` folder to the VBR server (the installer needs `common`
|
||||
as well as this folder), then from `collectors\veeam` run, from PowerShell 7 (`pwsh`):
|
||||
```powershell
|
||||
.\Install-VeeamCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
```
|
||||
It prompts for the key, then:
|
||||
- writes `C:\ProgramData\BackupDashboard\veeam-collector.json` (readable only by SYSTEM and Administrators)
|
||||
- copies the script to `C:\Program Files\BackupDashboard\`
|
||||
- registers the scheduled task **Backup Dashboard - Veeam collector** (SYSTEM, every 60 minutes,
|
||||
run by `C:\Program Files\PowerShell\7\pwsh.exe`; use `-PwshPath` if it's installed elsewhere)
|
||||
3. Preview the payloads, then backfill 30 days so the history view is populated straight away
|
||||
(from `pwsh`):
|
||||
```powershell
|
||||
& 'C:\Program Files\BackupDashboard\Send-VeeamBackupStatus.ps1' -DryRun
|
||||
& 'C:\Program Files\BackupDashboard\Send-VeeamBackupStatus.ps1' -LookbackHours 720
|
||||
```
|
||||
|
||||
Re-run the installer to change settings or update the script. It also installs
|
||||
`BackupDashboard.Collector.psm1`, the code shared by all collectors, next to the script. To remove the collector,
|
||||
delete the scheduled task and both `BackupDashboard` folders.
|
||||
|
||||
## What it sends
|
||||
|
||||
| Dashboard | Veeam |
|
||||
|---|---|
|
||||
| Company | `Tenant` from the config |
|
||||
| Job | Job name. The job's GUID is the ID, so renaming a job in Veeam doesn't duplicate it |
|
||||
| Status | Session result (Success / Warning / Failed). Any state other than Stopped is **Running**. A session that stopped with no result (e.g. cancelled) is reported as **Warning** |
|
||||
| Started / finished | Session `CreationTime` / `EndTime` |
|
||||
| Next scheduled | Job's next run, from the live schedule (`GetScheduleOptions()`). Sent as empty, so never flagged Overdue, for disabled jobs, chained ("run after") jobs, and immediate-mode backup copy jobs, which run whenever their source does |
|
||||
| Message | Session reason, each VM or object that didn't succeed, and warning/error lines from the session log |
|
||||
| Size | `BackupStats.BackupSize` |
|
||||
|
||||
**Backup copy jobs (VBR 12+):** Veeam records their sessions under hidden per-source "worker" jobs.
|
||||
The collector files each worker session under its copy job (via the worker's `PolicyJobId`), so a
|
||||
copy job gets one run per source job it copies, with the source in `copied_from`.
|
||||
|
||||
Each run covers the last 48 hours (`LookbackHours`), so a few missed runs (server rebooting, API
|
||||
down) fill in automatically. Sessions already delivered are recorded in
|
||||
`veeam-collector.state.json`. Running sessions are re-sent until they finish.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Log: `C:\ProgramData\BackupDashboard\veeam-collector.log` (rotates at 5 MB).
|
||||
- Exit codes: `0` OK, `1` some jobs failed to send (they're retried next run), `2` setup failed
|
||||
(config, Veeam module, or Veeam connection).
|
||||
- `API key rejected`: the key was revoked or mistyped. Re-run the installer with a new key.
|
||||
- A job shows **Unknown** on the dashboard: the collector sent the job but matched no sessions to it.
|
||||
Run `.\Get-VeeamCollectorDiagnostics.ps1` (read-only). It lists sessions whose job ID matches no
|
||||
job, plus the type and link properties of the jobs left without sessions.
|
||||
- `Get-VBRBackupSession` loads every session in the VBR database. On a server with years of history,
|
||||
this can make each run take a minute or more. The task's 30-minute limit and "don't start if
|
||||
already running" setting keep that safe.
|
||||
|
||||
## Testing without Veeam
|
||||
|
||||
`tests/MockVeeam.ps1` defines stand-in `Get-VBRJob` / `Get-VBRBackupSession` cmdlets with a mix of
|
||||
success, warning, failed, running and disabled jobs:
|
||||
|
||||
```powershell
|
||||
. ./tests/MockVeeam.ps1
|
||||
./Send-VeeamBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||
```
|
||||
|
||||
## Not covered yet
|
||||
|
||||
- Veeam Agent jobs managed by VBR (`Get-VBRComputerBackupJob`), tape jobs, and SureBackup.
|
||||
- Veeam Backup for Microsoft 365, which has its own PowerShell module.
|
||||
@@ -0,0 +1,266 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Reports Veeam Backup & Replication job results to the Backup Dashboard.
|
||||
|
||||
.DESCRIPTION
|
||||
Runs on the VBR server, normally as a SYSTEM scheduled task every 60 minutes
|
||||
(see Install-VeeamCollector.ps1). For every backup, replica and backup copy job it sends:
|
||||
- the job's next scheduled run (drives the dashboard's Overdue flag), and
|
||||
- every session from the last LookbackHours that the dashboard hasn't already received
|
||||
in its final state. Running sessions are re-sent each time until they finish.
|
||||
|
||||
Already-reported sessions are tracked in a state file, and the ingest API is idempotent,
|
||||
so a failed or repeated run is harmless: anything not confirmed is retried next time.
|
||||
|
||||
Compatible with Windows PowerShell 5.1 and PowerShell 7.
|
||||
|
||||
.PARAMETER ConfigPath
|
||||
JSON config with ApiUrl, ApiKey and Tenant (optional: SourceSystem, LookbackHours).
|
||||
|
||||
.PARAMETER LookbackHours
|
||||
Overrides the config value. Use 720 once to backfill 30 days of history.
|
||||
|
||||
.PARAMETER DryRun
|
||||
Print the payloads instead of sending them. Doesn't touch the state or log files.
|
||||
|
||||
.EXAMPLE
|
||||
.\Send-VeeamBackupStatus.ps1 -DryRun
|
||||
.EXAMPLE
|
||||
.\Send-VeeamBackupStatus.ps1 -LookbackHours 720
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$ConfigPath = (Join-Path $env:ProgramData 'BackupDashboard\veeam-collector.json'),
|
||||
[int]$LookbackHours = 0,
|
||||
[switch]$DryRun
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# Shared collector code: installed next to this script; in the repo, in ..\common
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
Write-Host "BackupDashboard.Collector.psm1 not found next to $PSCommandPath; re-run the installer."
|
||||
exit 2
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
# ---------- Helpers ----------
|
||||
|
||||
function Import-Veeam {
|
||||
if (Get-Command Get-VBRJob -ErrorAction SilentlyContinue) { return } # already loaded
|
||||
if (Get-Module -ListAvailable -Name Veeam.Backup.PowerShell) {
|
||||
Import-Module Veeam.Backup.PowerShell -WarningAction SilentlyContinue # VBR 11+
|
||||
} else {
|
||||
Add-PSSnapin VeeamPSSnapin # VBR 10 and earlier
|
||||
}
|
||||
}
|
||||
|
||||
function Get-NextRun($Job) {
|
||||
try {
|
||||
if (-not $Job.IsScheduleEnabled) { return $null }
|
||||
# Immediate-mode backup copy jobs run whenever their source job does. Their schedule still
|
||||
# carries a NextRun, but it's a stale date (months old on VBR 13) that would show as Overdue.
|
||||
if ($Job.IsContinuous) { return $null }
|
||||
|
||||
# The ScheduleOptions property is a stale snapshot on VBR 13 (NextRun empty, last run years
|
||||
# old); GetScheduleOptions() reads the live schedule. Fall back for versions without it.
|
||||
$options = $null
|
||||
try { $options = $Job.GetScheduleOptions() } catch { }
|
||||
if (-not $options) { $options = $Job.ScheduleOptions }
|
||||
$raw = $options.NextRun
|
||||
if (-not $raw) { return $null } # e.g. chained jobs that run "after" another job
|
||||
if ($raw -is [datetime]) { return ConvertTo-IsoTime $raw }
|
||||
# A string formatted by VBR ("2026-10-09 12:15:00 AM" on VBR 13, en-CA). Try the server's
|
||||
# culture first, then the invariant one
|
||||
foreach ($culture in [Globalization.CultureInfo]::CurrentCulture, [Globalization.CultureInfo]::InvariantCulture) {
|
||||
$parsed = [datetime]::MinValue
|
||||
if ([datetime]::TryParse([string]$raw, $culture, [Globalization.DateTimeStyles]::None, [ref]$parsed)) {
|
||||
return ConvertTo-IsoTime $parsed
|
||||
}
|
||||
}
|
||||
Write-Log "Couldn't parse next run '$raw' for job '$($Job.Name)'" 'WARN'
|
||||
} catch {
|
||||
Write-Log "Couldn't read schedule for job '$($Job.Name)': $(Get-ErrorText $_)" 'WARN'
|
||||
}
|
||||
return $null
|
||||
}
|
||||
|
||||
function Get-RunStatus($Session) {
|
||||
if ([string]$Session.State -ne 'Stopped') { return 'running' }
|
||||
switch ([string]$Session.Result) {
|
||||
'Success' { return 'success' }
|
||||
'Warning' { return 'warning' }
|
||||
'Failed' { return 'failed' }
|
||||
# Stopped without a result (e.g. cancelled): surface it rather than hide it
|
||||
default { return 'warning' }
|
||||
}
|
||||
}
|
||||
|
||||
# Builds a readable summary: the session's reason, any task (VM/object) that didn't succeed,
|
||||
# and warning/error lines from the session log.
|
||||
function Get-RunMessage($Session, [string]$Status, $Tasks) {
|
||||
$lines = New-Object 'System.Collections.Generic.List[string]'
|
||||
try {
|
||||
$reason = [string]$Session.Info.Reason
|
||||
if ($reason.Trim()) { $lines.Add($reason.Trim()) }
|
||||
} catch { }
|
||||
|
||||
foreach ($task in $Tasks) {
|
||||
$taskStatus = [string]$task.Status
|
||||
if ($taskStatus -and $taskStatus -ne 'Success' -and $taskStatus -ne 'InProgress' -and $taskStatus -ne 'Pending') {
|
||||
$taskReason = ''
|
||||
try { $taskReason = ([string]$task.Info.Reason).Trim() } catch { }
|
||||
$line = "$($task.Name): $taskStatus"
|
||||
if ($taskReason) { $line += " - $taskReason" }
|
||||
$lines.Add($line)
|
||||
}
|
||||
}
|
||||
|
||||
if ($Status -eq 'warning' -or $Status -eq 'failed') {
|
||||
try {
|
||||
foreach ($record in $Session.Logger.GetLog().UpdatedRecords) {
|
||||
$recordStatus = [string]$record.Status
|
||||
if ($recordStatus -eq 'EWarning' -or $recordStatus -eq 'EFailed') {
|
||||
$title = ([string]$record.Title).Trim()
|
||||
if ($title -and -not $lines.Contains($title)) { $lines.Add($title) }
|
||||
}
|
||||
}
|
||||
} catch { }
|
||||
}
|
||||
|
||||
if ($lines.Count -eq 0) {
|
||||
switch ($Status) {
|
||||
'success' { return 'Job completed successfully.' }
|
||||
'running' { return '' }
|
||||
default { return "Job finished with result '$($Session.Result)'." }
|
||||
}
|
||||
}
|
||||
return ($lines -join "`n")
|
||||
}
|
||||
|
||||
# VBR 12+ backup copy jobs: Get-VBRJob returns the copy (policy) job, but Get-VBRBackupSession
|
||||
# returns only the sessions of its hidden worker jobs, one per source job, never the policy's own.
|
||||
# Each worker job records its policy in PolicyJobId, so those sessions can be filed under it.
|
||||
function Get-PolicyJobId($Session, [hashtable]$KnownJobIds) {
|
||||
try {
|
||||
$policyId = [string]$Session.GetJob().PolicyJobId
|
||||
if ($policyId -and $KnownJobIds.ContainsKey($policyId)) { return $policyId }
|
||||
} catch { }
|
||||
return ''
|
||||
}
|
||||
|
||||
function ConvertTo-Run($Session, $Job) {
|
||||
$status = Get-RunStatus $Session
|
||||
$tasks = @()
|
||||
try { $tasks = @($Session.GetTaskSessions()) } catch { }
|
||||
|
||||
$details = [ordered]@{
|
||||
job_type = [string]$Job.JobType
|
||||
vbr_server = $env:COMPUTERNAME
|
||||
objects = $tasks.Count
|
||||
}
|
||||
$failedTasks = @($tasks | Where-Object { [string]$_.Status -eq 'Failed' }).Count
|
||||
$warningTasks = @($tasks | Where-Object { [string]$_.Status -eq 'Warning' }).Count
|
||||
if ($failedTasks) { $details.objects_failed = $failedTasks }
|
||||
if ($warningTasks) { $details.objects_warning = $warningTasks }
|
||||
try { if ($Session.IsRetryMode) { $details.retry = $true } } catch { }
|
||||
# Worker session of a backup copy job: say which source job it copied
|
||||
# (worker sessions are named "<copy job>\<source job>")
|
||||
if ($Session.JobId.ToString() -ne $Job.Id.ToString()) {
|
||||
$details.copied_from = ([string]$Session.JobName -split '\\')[-1]
|
||||
}
|
||||
|
||||
$run = [ordered]@{
|
||||
external_id = $Session.Id.ToString()
|
||||
status = $status
|
||||
started_at = ConvertTo-IsoTime $Session.CreationTime
|
||||
message = Get-RunMessage $Session $status $tasks
|
||||
details = $details
|
||||
}
|
||||
if ($status -ne 'running') {
|
||||
$finished = ConvertTo-IsoTime $Session.EndTime
|
||||
if ($finished) { $run.finished_at = $finished } # otherwise the API uses receipt time
|
||||
try {
|
||||
$size = [long]$Session.BackupStats.BackupSize
|
||||
if ($size -gt 0) { $run.size_bytes = $size }
|
||||
$dataSize = [long]$Session.BackupStats.DataSize
|
||||
if ($dataSize -gt 0) { $details.data_size_bytes = $dataSize }
|
||||
} catch { }
|
||||
}
|
||||
return $run
|
||||
}
|
||||
|
||||
# ---------- Main ----------
|
||||
|
||||
try {
|
||||
$Config = Initialize-Collector -Name 'veeam' -ConfigPath $ConfigPath -LookbackHours $LookbackHours -DryRun:$DryRun
|
||||
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Veeam B&R' }
|
||||
$LookbackHours = $Config.LookbackHours
|
||||
|
||||
Import-Veeam
|
||||
|
||||
$since = (Get-Date).AddHours(-$LookbackHours)
|
||||
$jobs = @(Get-VBRJob -WarningAction SilentlyContinue)
|
||||
|
||||
$jobIds = @{}
|
||||
foreach ($j in $jobs) { $jobIds[$j.Id.ToString()] = $true }
|
||||
|
||||
# Get-VBRBackupSession returns every session in the database, so fetch once and bucket by job
|
||||
$sessionsByJob = @{}
|
||||
$policyByWorker = @{} # worker job ID -> policy job ID ('' if it isn't a worker of a listed job)
|
||||
foreach ($s in Get-VBRBackupSession -WarningAction SilentlyContinue) {
|
||||
if ($s.CreationTime -lt $since) { continue }
|
||||
$key = $s.JobId.ToString()
|
||||
if (-not $jobIds.ContainsKey($key)) {
|
||||
if (-not $policyByWorker.ContainsKey($key)) { $policyByWorker[$key] = Get-PolicyJobId $s $jobIds }
|
||||
if (-not $policyByWorker[$key]) { continue }
|
||||
$key = $policyByWorker[$key]
|
||||
}
|
||||
if (-not $sessionsByJob.ContainsKey($key)) { $sessionsByJob[$key] = New-Object 'System.Collections.Generic.List[object]' }
|
||||
$sessionsByJob[$key].Add($s)
|
||||
}
|
||||
} catch {
|
||||
Write-Log "Collector setup failed: $(Get-ErrorText $_)" 'ERROR'
|
||||
exit 2
|
||||
}
|
||||
|
||||
$failures = 0
|
||||
foreach ($job in $jobs) {
|
||||
$jobInfo = [ordered]@{
|
||||
external_id = $job.Id.ToString()
|
||||
name = $job.Name
|
||||
# Always sent (null when unscheduled/disabled) so the dashboard's value tracks Veeam's
|
||||
next_run_at = Get-NextRun $job
|
||||
}
|
||||
$pending = @()
|
||||
if ($sessionsByJob.ContainsKey($jobInfo.external_id)) {
|
||||
$pending = @($sessionsByJob[$jobInfo.external_id] |
|
||||
Where-Object { -not (Test-RunSent ($_.Id.ToString())) } |
|
||||
Sort-Object CreationTime)
|
||||
}
|
||||
|
||||
try {
|
||||
if ($pending.Count -eq 0) {
|
||||
# Keep next_run_at fresh even when nothing new ran
|
||||
Send-JobReport -Tenant $Config.Tenant -SourceSystem $SourceSystem -Job $jobInfo
|
||||
}
|
||||
foreach ($session in $pending) {
|
||||
$run = ConvertTo-Run $session $job
|
||||
Send-JobReport -Tenant $Config.Tenant -SourceSystem $SourceSystem -Job $jobInfo -Run $run -SentKey $run.external_id -StartedAt ([datetime]$session.CreationTime)
|
||||
}
|
||||
} catch {
|
||||
$failures++
|
||||
Write-Log "Failed to report job '$($job.Name)': $(Get-ErrorText $_)" 'ERROR'
|
||||
if (Test-AuthRejected $_) {
|
||||
Write-Log 'API key rejected; check ApiKey in the config. Stopping.' 'ERROR'
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$exitCode = Complete-Collector -JobCount $jobs.Count -Failures $failures
|
||||
if ($exitCode) { exit $exitCode }
|
||||
@@ -0,0 +1,94 @@
|
||||
# Stand-ins for the Veeam cmdlets, shaped like the real objects' properties the collector reads.
|
||||
# Dot-source this before running the collector to try it without a VBR server:
|
||||
#
|
||||
# . ./tests/MockVeeam.ps1
|
||||
# ./Send-VeeamBackupStatus.ps1 -ConfigPath ./tests/config.json -DryRun
|
||||
|
||||
# Fixed to the top of the hour, and IDs derived from names/times, so repeated runs see the same
|
||||
# jobs and sessions, like a real server
|
||||
# MOCK_NOW (ISO 8601, set by the test suite) pins the time everything is relative to
|
||||
$now = if ($env:MOCK_NOW) { [datetime]$env:MOCK_NOW } else { Get-Date -Minute 0 -Second 0 -Millisecond 0 }
|
||||
function New-StableGuid([string]$Seed) {
|
||||
[guid]::new([Security.Cryptography.MD5]::Create().ComputeHash([Text.Encoding]::UTF8.GetBytes($Seed)))
|
||||
}
|
||||
$notSet = Get-Date '1900-01-01'
|
||||
|
||||
# As on VBR 13: the ScheduleOptions property is a stale snapshot with an empty NextRun, and the live
|
||||
# schedule (NextRun as a "2026-10-09 12:15:00 AM" string) comes from GetScheduleOptions().
|
||||
# -Continuous: an immediate-mode copy job, whose live NextRun is a stale date in the past.
|
||||
function New-MockJob($Name, $Type, $NextRun, [bool]$Enabled = $true, [switch]$Continuous) {
|
||||
$job = [pscustomobject]@{
|
||||
Id = New-StableGuid $Name; Name = $Name; JobType = $Type
|
||||
IsScheduleEnabled = $Enabled
|
||||
IsContinuous = [bool]$Continuous
|
||||
ScheduleOptions = [pscustomobject]@{ NextRun = '' }
|
||||
LiveNextRun = $NextRun
|
||||
}
|
||||
$job | Add-Member ScriptMethod GetScheduleOptions { [pscustomobject]@{ NextRun = $this.LiveNextRun } }
|
||||
$job
|
||||
}
|
||||
function Format-VbrTime([datetime]$Value) {
|
||||
$Value.ToString('yyyy-MM-dd h:mm:ss tt', [Globalization.CultureInfo]::InvariantCulture)
|
||||
}
|
||||
|
||||
function New-MockSession($Job, $Start, $End, $State, $Result, $Reason = '', $Tasks = @(), $LogRecords = @(), $Size = 0) {
|
||||
$s = [pscustomobject]@{
|
||||
Id = New-StableGuid "$($Job.Name)|$($Start.Ticks)"; JobId = $Job.Id; JobName = $Job.Name
|
||||
CreationTime = $Start; EndTime = $End; State = $State; Result = $Result
|
||||
IsRetryMode = $false
|
||||
Info = [pscustomobject]@{ Reason = $Reason }
|
||||
BackupStats = [pscustomobject]@{ BackupSize = $Size; DataSize = $Size * 3 }
|
||||
Logger = [pscustomobject]@{ Records = $LogRecords }
|
||||
Tasks = $Tasks
|
||||
}
|
||||
$s.Logger | Add-Member ScriptMethod GetLog { [pscustomobject]@{ UpdatedRecords = $this.Records } }
|
||||
$s | Add-Member ScriptMethod GetTaskSessions { $this.Tasks }
|
||||
$s
|
||||
}
|
||||
|
||||
function New-MockTask($Name, $Status, $Reason = '') {
|
||||
[pscustomobject]@{ Name = $Name; Status = $Status; Info = [pscustomobject]@{ Reason = $Reason } }
|
||||
}
|
||||
|
||||
$global:MockJobs = @(
|
||||
(New-MockJob 'File Server – Nightly' 'Backup' (Format-VbrTime $now.Date.AddDays(1).AddMinutes(15))),
|
||||
# Some versions return NextRun as a DateTime
|
||||
(New-MockJob 'SQL01 – Logs' 'Backup' $now.AddMinutes(40)),
|
||||
(New-MockJob 'Offsite Copy' 'BackupSync' $null),
|
||||
(New-MockJob 'Old Exchange' 'Backup' $now.AddDays(1) $false),
|
||||
# VBR 12+ backup copy job (immediate mode, no schedule): the "policy" job Get-VBRJob returns
|
||||
(New-MockJob 'Cloud Copy' 'SimpleBackupCopyPolicy' (Format-VbrTime $now.AddMonths(-6)) -Continuous)
|
||||
)
|
||||
$fs, $sql, $copy, $old, $cloud = $global:MockJobs
|
||||
|
||||
# Its hidden worker job, one per source job. Get-VBRJob doesn't return it, and Get-VBRBackupSession
|
||||
# returns only the worker's sessions. Shape taken from a real VBR 13 server.
|
||||
$cloudWorker = [pscustomobject]@{
|
||||
Id = New-StableGuid 'Cloud Copy\File Server – Nightly'; Name = 'Cloud Copy\File Server – Nightly'
|
||||
JobType = 'SimpleBackupCopyWorker'; PolicyJobId = $cloud.Id
|
||||
}
|
||||
function New-MockWorkerSession($Start, $End, $Result) {
|
||||
$s = New-MockSession $cloudWorker $Start $End 'Stopped' $Result -Size 40GB
|
||||
$s | Add-Member ScriptMethod GetJob { $cloudWorker }
|
||||
$s
|
||||
}
|
||||
|
||||
$global:MockSessions = @(
|
||||
(New-MockSession $fs $now.AddDays(-1).AddHours(-2) $now.AddDays(-1).AddHours(-1) 'Stopped' 'Success' -Size 128GB -Tasks @(
|
||||
(New-MockTask 'FS01' 'Success'), (New-MockTask 'FS02' 'Success'))),
|
||||
(New-MockSession $fs $now.AddMinutes(-20) $notSet 'Working' 'None' -Tasks @((New-MockTask 'FS01' 'InProgress'))),
|
||||
(New-MockSession $sql $now.AddHours(-1) $now.AddMinutes(-55) 'Stopped' 'Warning' -Size 2GB `
|
||||
-Tasks @((New-MockTask 'SQL01' 'Warning' 'Failed to truncate transaction logs for database "ERP_PROD"'))),
|
||||
(New-MockSession $copy $now.AddHours(-5) $now.AddHours(-4) 'Stopped' 'Failed' 'Error: Unable to connect to repository NAS02' `
|
||||
-LogRecords @(
|
||||
[pscustomobject]@{ Title = 'Preparing backup chain'; Status = 'ESucceeded' },
|
||||
[pscustomobject]@{ Title = 'Network path not found (0x80070035)'; Status = 'EFailed' })),
|
||||
(New-MockSession $old $now.AddDays(-10) $now.AddDays(-10).AddHours(1) 'Stopped' 'Success'), # outside 48h lookback
|
||||
(New-MockWorkerSession $now.AddDays(-1).AddMinutes(-55) $now.AddDays(-1).AddMinutes(-30) 'Success'),
|
||||
(New-MockWorkerSession $now.AddMinutes(-55) $now.AddMinutes(-35) 'Warning'),
|
||||
# A session from a job the collector doesn't know (e.g. deleted): must be ignored, not crash
|
||||
(New-MockSession ([pscustomobject]@{ Id = New-StableGuid 'gone'; Name = 'Deleted job' }) $now.AddHours(-2) $now.AddHours(-1) 'Stopped' 'Success')
|
||||
)
|
||||
|
||||
function Get-VBRJob { [CmdletBinding()] param() $global:MockJobs }
|
||||
function Get-VBRBackupSession { [CmdletBinding()] param() $global:MockSessions }
|
||||
@@ -0,0 +1,103 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Read-only dump of what the Windows Server Backup cmdlets return, for troubleshooting the collector.
|
||||
|
||||
.DESCRIPTION
|
||||
The collector reads properties of Get-WBJob / Get-WBSummary results whose names and time zones
|
||||
aren't fully documented. This prints those objects' real structure (property names, types and
|
||||
values, a few levels deep), plus recent events from the Microsoft-Windows-Backup log, whose
|
||||
timestamps are reliable local times to compare against. Nothing is changed. Output is also
|
||||
saved to a text file.
|
||||
|
||||
.EXAMPLE
|
||||
.\Get-WsbCollectorDiagnostics.ps1
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[int]$Jobs = 3,
|
||||
[int]$Events = 15,
|
||||
[string]$OutFile = (Join-Path $env:TEMP 'wsb-collector-diagnostics.txt')
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$MaxDepth = 6
|
||||
|
||||
if (-not (Get-Command Get-WBJob -ErrorAction SilentlyContinue)) {
|
||||
Import-Module WindowsServerBackup
|
||||
}
|
||||
|
||||
function Test-Scalar($Value) {
|
||||
return $null -eq $Value -or $Value -is [string] -or $Value -is [ValueType] -or $Value -is [guid]
|
||||
}
|
||||
|
||||
# Recursive "name (type) = value" listing. Collections show their first 3 items.
|
||||
function Show-Object($Value, [string]$Indent = ' ', [int]$Depth = 0) {
|
||||
if (Test-Scalar $Value) { return }
|
||||
if ($Depth -ge $MaxDepth) { "$Indent..."; return }
|
||||
if ($Value -is [Collections.IEnumerable]) {
|
||||
$items = @($Value)
|
||||
"$Indent[$($items.Count) item(s)]"
|
||||
$i = 0
|
||||
foreach ($item in $items | Select-Object -First 3) {
|
||||
"$Indent[$i] ($($item.GetType().FullName))$(if (Test-Scalar $item) { " = $item" })"
|
||||
Show-Object $item "$Indent " ($Depth + 1)
|
||||
$i++
|
||||
}
|
||||
return
|
||||
}
|
||||
foreach ($p in $Value.PSObject.Properties) {
|
||||
try { $v = $p.Value } catch { "$Indent$($p.Name) = <error: $($_.Exception.Message)>"; continue }
|
||||
$type = if ($null -eq $v) { 'null' } else { $v.GetType().FullName }
|
||||
if (Test-Scalar $v) {
|
||||
"$Indent$($p.Name) ($type) = $v"
|
||||
} else {
|
||||
$text = [string]$v
|
||||
if ($text.Length -gt 100) { $text = $text.Substring(0, 97) + '...' }
|
||||
"$Indent$($p.Name) ($type) = $text"
|
||||
Show-Object $v "$Indent " ($Depth + 1)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function Show-Section([string]$Title, [scriptblock]$Get) {
|
||||
"== $Title"
|
||||
try {
|
||||
$result = @(& $Get)
|
||||
if ($result.Count -eq 0 -or ($result.Count -eq 1 -and $null -eq $result[0])) { ' (nothing returned)' }
|
||||
$n = 0
|
||||
foreach ($item in $result) {
|
||||
if ($null -eq $item) { continue }
|
||||
" [$n] ($($item.GetType().FullName))"
|
||||
Show-Object $item ' '
|
||||
$n++
|
||||
}
|
||||
} catch {
|
||||
" error: $($_.Exception.Message)"
|
||||
}
|
||||
''
|
||||
}
|
||||
|
||||
$report = & {
|
||||
"Generated $(Get-Date -Format o) on $env:COMPUTERNAME, PowerShell $($PSVersionTable.PSVersion)"
|
||||
"Local time zone: $([TimeZoneInfo]::Local.Id), UTC offset now $([TimeZoneInfo]::Local.GetUtcOffset((Get-Date)))"
|
||||
''
|
||||
Show-Section 'Get-WBSummary' { Get-WBSummary }
|
||||
Show-Section 'Get-WBJob (in progress)' { Get-WBJob }
|
||||
Show-Section "Get-WBJob -Previous $Jobs" { Get-WBJob -Previous $Jobs }
|
||||
Show-Section 'Get-WBPolicy | Get-WBSchedule' {
|
||||
$policy = Get-WBPolicy
|
||||
if ($policy) { Get-WBSchedule -Policy $policy } else { '(no scheduled backup policy)' }
|
||||
}
|
||||
"== Microsoft-Windows-Backup event log (last $Events, local times)"
|
||||
try {
|
||||
Get-WinEvent -LogName 'Microsoft-Windows-Backup' -MaxEvents $Events |
|
||||
Select-Object TimeCreated, Id, LevelDisplayName, @{ n = 'Message'; e = { ($_.Message -split "`r?`n")[0] } } |
|
||||
Format-Table -AutoSize -Wrap | Out-String -Width 300
|
||||
} catch {
|
||||
" error: $($_.Exception.Message)"
|
||||
}
|
||||
} | Out-String -Width 300
|
||||
|
||||
$report
|
||||
$report | Set-Content -Path $OutFile -Encoding UTF8
|
||||
Write-Host "Saved to $OutFile"
|
||||
@@ -0,0 +1,54 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Installs the Windows Server Backup collector as a SYSTEM scheduled task.
|
||||
|
||||
.DESCRIPTION
|
||||
- Copies Send-WsbBackupStatus.ps1 to Program Files (only admins can modify it, which
|
||||
matters because it runs as SYSTEM).
|
||||
- Writes the config to ProgramData\BackupDashboard and restricts that folder to SYSTEM and
|
||||
Administrators, since it contains the API key.
|
||||
- Registers a scheduled task that runs every IntervalMinutes under Windows PowerShell 5.1
|
||||
(powershell.exe), which the WindowsServerBackup module targets.
|
||||
Safe to re-run to change settings or update the script. Can sit alongside the other collectors.
|
||||
|
||||
.EXAMPLE
|
||||
.\Install-WsbCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
(prompts for the API key)
|
||||
#>
|
||||
#Requires -RunAsAdministrator
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory)][string]$ApiUrl,
|
||||
[Parameter(Mandatory)][string]$Tenant,
|
||||
# Name shown on the dashboard; defaults to "<server> - Windows Server Backup"
|
||||
[string]$JobName,
|
||||
[string]$SourceSystem = 'Windows Server Backup',
|
||||
[ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
|
||||
[ValidateRange(1, 720)][int]$LookbackHours = 48,
|
||||
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard')
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$WindowsPowerShell = Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\powershell.exe'
|
||||
|
||||
# Check Windows Server Backup is here before changing anything
|
||||
if (-not (Get-Module -ListAvailable -Name WindowsServerBackup)) {
|
||||
throw 'The WindowsServerBackup PowerShell module is not available. Install the Windows Server Backup feature (Install-WindowsFeature Windows-Server-Backup).'
|
||||
}
|
||||
|
||||
# Shared installer code: in the repo it's in ..\common, so copy the whole collectors folder
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
throw "BackupDashboard.Collector.psm1 not found. Copy the whole collectors folder, including common."
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
$settings = [ordered]@{ Tenant = $Tenant; SourceSystem = $SourceSystem; LookbackHours = $LookbackHours }
|
||||
if ($JobName) { $settings.JobName = $JobName }
|
||||
Install-CollectorTask -Name 'wsb' -ScriptPath (Join-Path $PSScriptRoot 'Send-WsbBackupStatus.ps1') `
|
||||
-TaskName 'Backup Dashboard - WSB collector' `
|
||||
-TaskDescription 'Sends Windows Server Backup results to the Backup Dashboard' `
|
||||
-Shell $WindowsPowerShell -ApiUrl $ApiUrl -IntervalMinutes $IntervalMinutes -InstallDir $InstallDir -Settings $settings
|
||||
@@ -0,0 +1,71 @@
|
||||
# Windows Server Backup collector
|
||||
|
||||
A PowerShell script that runs on each server using Windows Server Backup (WSB) and reports its
|
||||
backups to the dashboard. Each server appears as one job, named `<SERVER> - Windows Server Backup`
|
||||
by default.
|
||||
|
||||
> **Checked against:** Server 2019 Hyper-V host backing up VMs, successful backups only.
|
||||
> Not yet seen on a real server: a failed backup, a failed VM or volume, or a backup in progress.
|
||||
> Those paths are written to the same object model, but if one looks wrong, run
|
||||
> `Get-WsbCollectorDiagnostics.ps1` while it's in that state and send the output.
|
||||
|
||||
## Install (on the server, as an administrator)
|
||||
|
||||
Requires the Windows Server Backup feature with its PowerShell module
|
||||
(`Install-WindowsFeature Windows-Server-Backup`). Runs under Windows PowerShell 5.1
|
||||
(`powershell.exe`), which that module targets.
|
||||
|
||||
1. Create a key for this server on the dashboard's **Admin** page, or on the dashboard host:
|
||||
```bash
|
||||
python manage.py create-key wsb-acme-dc01
|
||||
```
|
||||
2. Copy the whole `collectors` folder to the server (the installer needs `common`
|
||||
as well as this folder), then from `collectors\wsb` run:
|
||||
```powershell
|
||||
.\Install-WsbCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
```
|
||||
Add `-JobName "Domain Controller"` to choose the name shown on the dashboard. It prompts for the
|
||||
key, then:
|
||||
- writes `C:\ProgramData\BackupDashboard\wsb-collector.json` (readable only by SYSTEM and Administrators)
|
||||
- copies the script to `C:\Program Files\BackupDashboard\`
|
||||
- registers the scheduled task **Backup Dashboard - WSB collector** (SYSTEM, every 60 minutes)
|
||||
3. Preview the payloads, then backfill 30 days so the history view is populated straight away:
|
||||
```powershell
|
||||
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-WsbBackupStatus.ps1' -DryRun
|
||||
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-WsbBackupStatus.ps1' -LookbackHours 720
|
||||
```
|
||||
|
||||
It can be installed alongside the other collectors: it uses its own config, state, log and task.
|
||||
Re-run the installer to change settings or update the script. It also installs
|
||||
`BackupDashboard.Collector.psm1`, the code shared by all collectors, next to the script.
|
||||
|
||||
## What it sends
|
||||
|
||||
| Dashboard | Windows Server Backup |
|
||||
|---|---|
|
||||
| Company | `Tenant` from the config |
|
||||
| Job | One per server, ID `wsb:<computer name>`, so changing `JobName` renames rather than duplicates it |
|
||||
| Run | One per backup job from `Get-WBJob`, identified by its start time (WSB jobs have no ID). Restores are ignored. `details.items` lists what was backed up: the VMs for a Hyper-V backup, otherwise volumes |
|
||||
| Status | Error HRESULT → **Failed**. Success, but a detailed error or a failed item (e.g. one volume) → **Warning**. Cancelled or unrecognised → **Warning**. In progress → **Running** |
|
||||
| Started / finished | Job start / end time. WSB gives these as local-time text to the minute, in the server's date format; the collector parses them with that format |
|
||||
| Next scheduled | `Get-WBSummary`'s `NextBackupTime`. Empty when there's no scheduled backup |
|
||||
| Message | Error description, error codes (as `0x8XXXXXXX`), failed VMs/volumes, and the failure log path |
|
||||
| Size | Bytes processed across the backup's items |
|
||||
|
||||
Each run covers the last 48 hours (`LookbackHours`). Runs already delivered are recorded in
|
||||
`wsb-collector.state.json`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Log: `C:\ProgramData\BackupDashboard\wsb-collector.log`. Exit codes: `0` OK, `1` some runs
|
||||
failed to send (retried next run), `2` setup failed (config or WindowsServerBackup module).
|
||||
- Wrong status, times or messages: run `.\Get-WsbCollectorDiagnostics.ps1` (read-only). It prints
|
||||
the real structure of what the cmdlets return, plus recent `Microsoft-Windows-Backup` events to
|
||||
compare times against.
|
||||
|
||||
## Testing without Windows Server Backup
|
||||
|
||||
```powershell
|
||||
. ./tests/MockWsb.ps1 # or: . ./tests/MockWsb.ps1 -WithRunningJob
|
||||
./Send-WsbBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||
```
|
||||
@@ -0,0 +1,302 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Reports Windows Server Backup (WSB) results to the Backup Dashboard.
|
||||
|
||||
.DESCRIPTION
|
||||
Runs on a server with the Windows Server Backup feature, normally as a SYSTEM scheduled task
|
||||
every 60 minutes (see Install-WsbCollector.ps1). The server appears on the dashboard as one job.
|
||||
Each run sends:
|
||||
- the next scheduled backup (Get-WBSummary's NextBackupTime), which drives the dashboard's
|
||||
Overdue flag, and
|
||||
- every backup from the last LookbackHours that the dashboard hasn't already received in its
|
||||
final state. A backup in progress is sent as Running until it finishes.
|
||||
|
||||
WSB jobs have no ID, so a run is identified by its start time. Already-reported runs are
|
||||
tracked in a state file, and the ingest API is idempotent, so a failed or repeated run is
|
||||
harmless: anything not confirmed is retried next time.
|
||||
|
||||
Written for Windows PowerShell 5.1, which the WindowsServerBackup module targets.
|
||||
|
||||
.PARAMETER ConfigPath
|
||||
JSON config with ApiUrl, ApiKey and Tenant (optional: JobName, SourceSystem, LookbackHours).
|
||||
|
||||
.PARAMETER LookbackHours
|
||||
Overrides the config value. Use 720 once to backfill 30 days of history.
|
||||
|
||||
.PARAMETER DryRun
|
||||
Print the payloads instead of sending them. Doesn't touch the state or log files.
|
||||
|
||||
.EXAMPLE
|
||||
.\Send-WsbBackupStatus.ps1 -DryRun
|
||||
.EXAMPLE
|
||||
.\Send-WsbBackupStatus.ps1 -LookbackHours 720
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$ConfigPath = (Join-Path $env:ProgramData 'BackupDashboard\wsb-collector.json'),
|
||||
[int]$LookbackHours = 0,
|
||||
[switch]$DryRun
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# Shared collector code: installed next to this script; in the repo, in ..\common
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
Write-Host "BackupDashboard.Collector.psm1 not found next to $PSCommandPath; re-run the installer."
|
||||
exit 2
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
# Get-WBJob -Previous accepts at most 100. That covers a 30-day backfill at up to 3 backups a day;
|
||||
# servers backing up more often only get their latest 100 runs backfilled.
|
||||
$MaxPreviousJobs = 100
|
||||
|
||||
# ---------- Helpers ----------
|
||||
|
||||
function Import-Wsb {
|
||||
if (Get-Command Get-WBJob -ErrorAction SilentlyContinue) { return } # already loaded
|
||||
# Present only when the Windows Server Backup feature (with its command-line tools) is installed
|
||||
Import-Module WindowsServerBackup
|
||||
}
|
||||
|
||||
# ---------- Jobs ----------
|
||||
|
||||
# WBJob times are strings in local time, to the minute, formatted with the current culture
|
||||
# ("10/6/2026 9:00 PM" on en-US). Parse with that same culture: PowerShell's [datetime] cast would
|
||||
# use the invariant (US) format and misread dd/MM dates. Empty while not set (e.g. EndTime of a
|
||||
# running backup).
|
||||
function Get-JobTime($Job, [string]$Field) {
|
||||
$value = Get-FirstValue $Job @($Field)
|
||||
if ($null -eq $value) { return $null }
|
||||
if ($value -is [datetime]) {
|
||||
$dt = $value
|
||||
} else {
|
||||
$dt = [datetime]::MinValue
|
||||
if (-not [datetime]::TryParse([string]$value, [Globalization.CultureInfo]::CurrentCulture,
|
||||
[Globalization.DateTimeStyles]::AssumeLocal, [ref]$dt)) {
|
||||
Write-Log "Couldn't parse $Field '$value'" 'WARN'
|
||||
return $null
|
||||
}
|
||||
}
|
||||
if ($dt.Year -le 1900) { return $null }
|
||||
return $dt
|
||||
}
|
||||
|
||||
# HRESULTs as numbers; 0 = OK. $null when the object doesn't have the property. WSB returns them
|
||||
# as strings ("0"); PowerShell's conversion also accepts hex ("0x80042302") and negative decimals.
|
||||
function Get-HResult($Object, [string]$Field) {
|
||||
$value = Get-FirstValue $Object @($Field)
|
||||
if ($null -eq $value) { return $null }
|
||||
try { return [long]$value } catch { return $null }
|
||||
}
|
||||
|
||||
function Format-HResult([long]$Value) {
|
||||
if ($Value -lt 0) { $Value += 4294967296 } # show as the familiar unsigned 0x8XXXXXXX
|
||||
return '0x{0:X8}' -f $Value
|
||||
}
|
||||
|
||||
function Get-JobItems($Job) {
|
||||
return @(Get-FirstValue $Job @('JobItems', 'Items') | Where-Object { $_ })
|
||||
}
|
||||
|
||||
function Get-ItemName($Item) {
|
||||
$name = Get-FirstValue $Item @('Name', 'VolumeLabel', 'MountPath', 'Path', 'Type')
|
||||
if ($name) { return [string]$name }
|
||||
return 'Item'
|
||||
}
|
||||
|
||||
# The things actually backed up. Items can nest: a Hyper-V backup has one VirtualMachineList item
|
||||
# whose SubItemList holds each VM (and "Host Component").
|
||||
function Get-LeafItems($Items) {
|
||||
foreach ($item in $Items) {
|
||||
$subItems = @(Get-FirstValue $item @('SubItemList') | Where-Object { $_ })
|
||||
if ($subItems.Count) { Get-LeafItems $subItems } else { $item }
|
||||
}
|
||||
}
|
||||
|
||||
function Test-ItemFailed($Item) {
|
||||
$hr = Get-HResult $Item 'HResult'
|
||||
$state = [string](Get-FirstValue $Item @('State', 'JobState'))
|
||||
return ($hr -and $hr -ne 0) -or $state -match 'Fail'
|
||||
}
|
||||
|
||||
# Items that reported an error: failed leaves, or a parent that counts failed sub-items without
|
||||
# saying which
|
||||
function Get-FailedItems($Job) {
|
||||
$items = Get-JobItems $Job
|
||||
$failed = @(Get-LeafItems $items | Where-Object { Test-ItemFailed $_ })
|
||||
if ($failed.Count) { return $failed }
|
||||
return @($items | Where-Object {
|
||||
(Test-ItemFailed $_) -or ((Get-FirstValue $_ @('SubItemFailed')) -and [long](Get-FirstValue $_ @('SubItemFailed')) -gt 0)
|
||||
})
|
||||
}
|
||||
|
||||
# Bytes processed, from the top-level items (sub-items are already counted in their parent)
|
||||
function Get-BytesProcessed($Job) {
|
||||
$sum = [long]0
|
||||
foreach ($item in Get-JobItems $Job) {
|
||||
$value = Get-FirstValue $item @('BytesProcessed')
|
||||
if ($value) { $sum += [long]$value }
|
||||
}
|
||||
return $sum
|
||||
}
|
||||
|
||||
function Get-RunStatus($Job) {
|
||||
$state = [string](Get-FirstValue $Job @('JobState', 'State'))
|
||||
if (-not (Get-JobTime $Job 'EndTime') -and $state -notmatch 'Complet|Fail|Cancel|Abort|Stop|Succe') {
|
||||
return 'running'
|
||||
}
|
||||
if ($state -match 'Fail') { return 'failed' }
|
||||
if ($state -match 'Cancel|Abort|Stop') { return 'warning' }
|
||||
$hr = Get-HResult $Job 'HResult'
|
||||
if ($hr -and $hr -ne 0) { return 'failed' }
|
||||
# Overall success but something underneath went wrong (e.g. a volume or files skipped)
|
||||
$detailed = Get-HResult $Job 'DetailedHResult'
|
||||
if (($detailed -and $detailed -ne 0) -or (Get-FailedItems $Job).Count) { return 'warning' }
|
||||
if (-not $state -or $state -match 'Complet|Succe') { return 'success' }
|
||||
return 'warning' # finished in a state we don't recognise: surface it rather than hide it
|
||||
}
|
||||
|
||||
function Get-RunMessage($Job, [string]$Status) {
|
||||
if ($Status -eq 'running') { return '' }
|
||||
$lines = New-Object 'System.Collections.Generic.List[string]'
|
||||
$state = [string](Get-FirstValue $Job @('JobState', 'State'))
|
||||
$description = Get-FirstValue $Job @('ErrorDescription', 'DetailedMessage')
|
||||
if ($description) { $lines.Add(([string]$description).Trim()) }
|
||||
|
||||
$hr = Get-HResult $Job 'HResult'
|
||||
$detailed = Get-HResult $Job 'DetailedHResult'
|
||||
if (($hr -and $hr -ne 0) -or ($detailed -and $detailed -ne 0)) {
|
||||
$codes = @()
|
||||
if ($hr -and $hr -ne 0) { $codes += "error $(Format-HResult $hr)" }
|
||||
if ($detailed -and $detailed -ne 0) { $codes += "detailed error $(Format-HResult $detailed)" }
|
||||
$lines.Add('Windows Server Backup reported ' + ($codes -join ', ') + '.')
|
||||
}
|
||||
|
||||
foreach ($item in Get-FailedItems $Job) {
|
||||
$itemError = Get-FirstValue $item @('ErrorDescription', 'DetailedMessage')
|
||||
$itemHr = Get-HResult $item 'HResult'
|
||||
$subFailed = Get-FirstValue $item @('SubItemFailed')
|
||||
$line = "$(Get-ItemName $item) - " # names are often volumes like "D:", so no colon
|
||||
if ($itemError) { $line += ([string]$itemError).Trim() }
|
||||
elseif ($itemHr) { $line += "error $(Format-HResult $itemHr)" }
|
||||
elseif ($subFailed -and [long]$subFailed -gt 0) { $line += "$subFailed sub-item(s) failed" }
|
||||
else { $line += [string](Get-FirstValue $item @('State', 'JobState')) }
|
||||
$lines.Add($line)
|
||||
}
|
||||
|
||||
# WSB fills in FailureLogPath even for successful backups, so only point at it when relevant
|
||||
if ($Status -ne 'success') {
|
||||
$failureLog = Get-FirstValue $Job @('FailureLogPath')
|
||||
if ($failureLog) { $lines.Add("Failure log: $failureLog") }
|
||||
}
|
||||
|
||||
if ($lines.Count -eq 0) {
|
||||
if ($Status -eq 'success') { return 'Backup completed successfully.' }
|
||||
return "Backup finished with state '$state'."
|
||||
}
|
||||
# WSB reports "Completed" even for a backup that failed with an error code, so only mention
|
||||
# the state when it says something else (e.g. cancelled)
|
||||
if ($state -and $Status -ne 'success' -and $state -notmatch 'Complet') { $lines.Insert(0, "Job state: $state") }
|
||||
return ($lines -join "`n")
|
||||
}
|
||||
|
||||
function ConvertTo-Run($Job) {
|
||||
$status = Get-RunStatus $Job
|
||||
$details = [ordered]@{ server = $env:COMPUTERNAME }
|
||||
$items = @(Get-LeafItems (Get-JobItems $Job) | ForEach-Object { Get-ItemName $_ })
|
||||
if ($items.Count) { $details.items = $items }
|
||||
$state = Get-FirstValue $Job @('JobState', 'State')
|
||||
if ($state) { $details.wsb_state = [string]$state }
|
||||
$version = Get-FirstValue $Job @('VersionId')
|
||||
if ($version) { $details.version_id = [string]$version }
|
||||
|
||||
# No external_id: WSB jobs have no ID, so the API keys the run on started_at
|
||||
$run = [ordered]@{
|
||||
status = $status
|
||||
started_at = ConvertTo-IsoTime (Get-JobTime $Job 'StartTime')
|
||||
message = Get-RunMessage $Job $status
|
||||
details = $details
|
||||
}
|
||||
if ($status -ne 'running') {
|
||||
$finished = ConvertTo-IsoTime (Get-JobTime $Job 'EndTime')
|
||||
if ($finished) { $run.finished_at = $finished } # otherwise the API uses receipt time
|
||||
$bytes = Get-BytesProcessed $Job
|
||||
if ($bytes -gt 0) { $run.size_bytes = $bytes }
|
||||
}
|
||||
return $run
|
||||
}
|
||||
|
||||
function Test-IsBackupJob($Job) {
|
||||
# Get-WBJob also lists recoveries; only backups belong on the dashboard
|
||||
$type = [string](Get-FirstValue $Job @('JobType', 'Type'))
|
||||
return (-not $type) -or $type -match 'Backup'
|
||||
}
|
||||
|
||||
function Get-NextRun {
|
||||
try {
|
||||
$summary = Get-WBSummary
|
||||
return ConvertTo-IsoTime (Get-FirstValue $summary @('NextBackupTime'))
|
||||
} catch {
|
||||
Write-Log "Couldn't read the backup summary: $(Get-ErrorText $_)" 'WARN'
|
||||
return $null
|
||||
}
|
||||
}
|
||||
|
||||
# ---------- Main ----------
|
||||
|
||||
try {
|
||||
$Config = Initialize-Collector -Name 'wsb' -ConfigPath $ConfigPath -LookbackHours $LookbackHours -DryRun:$DryRun
|
||||
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Windows Server Backup' }
|
||||
$JobName = if ($Config.JobName) { [string]$Config.JobName } else { "$env:COMPUTERNAME - Windows Server Backup" }
|
||||
$LookbackHours = $Config.LookbackHours
|
||||
|
||||
Import-Wsb
|
||||
|
||||
$since = (Get-Date).AddHours(-$LookbackHours)
|
||||
# Get-WBJob with no arguments returns the job in progress (if any); -Previous lists finished ones
|
||||
$jobs = @(@(Get-WBJob) + @(Get-WBJob -Previous $MaxPreviousJobs) | Where-Object {
|
||||
$_ -and (Test-IsBackupJob $_) -and (Get-JobTime $_ 'StartTime')
|
||||
})
|
||||
$byStart = [ordered]@{}
|
||||
foreach ($job in $jobs) {
|
||||
$start = Get-JobTime $job 'StartTime'
|
||||
if ($start -lt $since) { continue }
|
||||
$byStart[(ConvertTo-IsoTime $start)] = $job # a job in both lists: the finished entry (later) wins
|
||||
}
|
||||
$pending = @($byStart.Keys | Sort-Object | Where-Object { -not (Test-RunSent ($_)) } |
|
||||
ForEach-Object { $byStart[$_] })
|
||||
|
||||
$jobInfo = [ordered]@{
|
||||
# Stable per server, so renaming the dashboard job via JobName doesn't create a new one
|
||||
external_id = "wsb:$($env:COMPUTERNAME.ToLowerInvariant())"
|
||||
name = $JobName
|
||||
# Always sent (null when unscheduled) so the dashboard's value tracks WSB's
|
||||
next_run_at = Get-NextRun
|
||||
}
|
||||
} catch {
|
||||
Write-Log "Collector setup failed: $(Get-ErrorText $_)" 'ERROR'
|
||||
exit 2
|
||||
}
|
||||
|
||||
$failures = 0
|
||||
try {
|
||||
if ($pending.Count -eq 0) {
|
||||
# Keep next_run_at fresh even when nothing new ran
|
||||
Send-JobReport -Tenant $Config.Tenant -SourceSystem $SourceSystem -Job $jobInfo
|
||||
}
|
||||
foreach ($job in $pending) {
|
||||
$run = ConvertTo-Run $job
|
||||
Send-JobReport -Tenant $Config.Tenant -SourceSystem $SourceSystem -Job $jobInfo -Run $run -SentKey $run.started_at -StartedAt ((Get-JobTime $job 'StartTime'))
|
||||
}
|
||||
} catch {
|
||||
$failures++
|
||||
Write-Log "Failed to report: $(Get-ErrorText $_)" 'ERROR'
|
||||
}
|
||||
|
||||
$exitCode = Complete-Collector -Failures $failures
|
||||
if ($exitCode) { exit $exitCode }
|
||||
@@ -0,0 +1,104 @@
|
||||
# Stand-ins for the WindowsServerBackup cmdlets. Object shapes follow Get-WsbCollectorDiagnostics.ps1
|
||||
# output from a real Hyper-V host (Server 2019): job times are culture-formatted local-time strings
|
||||
# to the minute ("10/6/2026 9:00 PM" on en-US), HRESULTs are strings, and a Hyper-V backup has one
|
||||
# VirtualMachineList item whose SubItemList holds the VMs. When nothing is running, Get-WBJob
|
||||
# returns an empty job (JobType None).
|
||||
#
|
||||
# . ./tests/MockWsb.ps1 # nothing running
|
||||
# . ./tests/MockWsb.ps1 -WithRunningJob # a backup in progress
|
||||
# . ./tests/MockWsb.ps1 -RunningJobFinished # that same backup, now finished
|
||||
# ./Send-WsbBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||
param([switch]$WithRunningJob, [switch]$RunningJobFinished)
|
||||
|
||||
# Fixed to the top of the hour so repeated runs see the same jobs
|
||||
# MOCK_NOW (ISO 8601, set by the test suite) pins the time everything is relative to
|
||||
$now = if ($env:MOCK_NOW) { [datetime]$env:MOCK_NOW } else { Get-Date -Minute 0 -Second 0 -Millisecond 0 }
|
||||
|
||||
# Same format as the real cmdlet: short date + short time in the current culture
|
||||
function Format-WbTime($Value) {
|
||||
if ($null -eq $Value) { return '' }
|
||||
return ([datetime]$Value).ToString('g', [Globalization.CultureInfo]::CurrentCulture)
|
||||
}
|
||||
|
||||
function New-MockItem($Name, $Type = 'Volume', [long]$Bytes = 0, [string]$HResult = '0', $ErrorDescription = '', $SubItems = $null) {
|
||||
$failedSubs = @($SubItems | Where-Object { $_ -and $_.HResult -ne '0' }).Count
|
||||
[pscustomobject]@{
|
||||
Name = $Name
|
||||
Type = $Type
|
||||
State = $(if ($HResult -ne '0') { 'Failed' } else { 'Completed' })
|
||||
HResult = $HResult
|
||||
DetailedHResult = '0'
|
||||
ErrorDescription = $ErrorDescription
|
||||
BytesProcessed = $Bytes
|
||||
TotalBytes = $Bytes
|
||||
CurrentItem = $null
|
||||
SubItemProcessed = [long]@($SubItems | Where-Object { $_ }).Count
|
||||
SubItemFailed = [long]$failedSubs
|
||||
TotalSubItem = [long]@($SubItems | Where-Object { $_ }).Count
|
||||
SubItemList = $SubItems
|
||||
}
|
||||
}
|
||||
|
||||
function New-VmList([long]$VmBytes, [string]$VmHResult = '0', $VmError = '') {
|
||||
$vms = @(
|
||||
(New-MockItem 'AAOG-EDM-AD1' 'Component' $VmBytes $VmHResult $VmError),
|
||||
(New-MockItem 'Host Component' 'Component' 447729)
|
||||
)
|
||||
New-MockItem 'VirtualMachineList' 'VirtualmachineList' ($VmBytes + 447729) -SubItems $vms
|
||||
}
|
||||
|
||||
function New-MockJob($Type, $Start, $End, $State, [string]$HResult = '0', [string]$DetailedHResult = '0', $ErrorDescription = '', $Items = $null) {
|
||||
[pscustomobject]@{
|
||||
JobType = $Type
|
||||
StartTime = Format-WbTime $Start
|
||||
EndTime = Format-WbTime $End
|
||||
JobState = $State
|
||||
CurrentOperation = ''
|
||||
HResult = $HResult
|
||||
DetailedHResult = $DetailedHResult
|
||||
ErrorDescription = $ErrorDescription
|
||||
JobItems = $Items
|
||||
VersionId = $(if ($End) { $Start.ToUniversalTime().ToString('MM/dd/yyyy-HH:mm', [Globalization.CultureInfo]::InvariantCulture) } else { '' })
|
||||
SuccessLogPath = 'C:\Windows\Logs\WindowsServerBackup\Backup.log'
|
||||
# Populated even for successful backups, as on the real server
|
||||
FailureLogPath = 'C:\Windows\Logs\WindowsServerBackup\Backup_Error.log'
|
||||
}
|
||||
}
|
||||
|
||||
$global:MockWsbRunning = if ($WithRunningJob) {
|
||||
@(New-MockJob 'Backup' $now.AddMinutes(-10) $null 'Running' -Items @(New-VmList 10GB))
|
||||
} else {
|
||||
@(New-MockJob 'None' $null $null 'Unknown')
|
||||
}
|
||||
|
||||
$global:MockWsbPrevious = @(
|
||||
(New-MockJob 'Backup' $now.AddHours(-12) $now.AddHours(-11.6) 'Completed' -Items @(New-VmList 124756706036)),
|
||||
# Completed overall, but one VM failed
|
||||
(New-MockJob 'Backup' $now.AddHours(-36) $now.AddHours(-35.6) 'Completed' '0' '0' '' @(
|
||||
New-VmList 0 '-2147024891' 'Access is denied.')),
|
||||
# Failed outright: backup target not found (0x80042302)
|
||||
(New-MockJob 'Backup' $now.AddHours(-30) $now.AddHours(-29.9) 'Completed' '-2147212542' '0' `
|
||||
'The backup storage location could not be found.'),
|
||||
# A volume-based backup, for servers that aren't Hyper-V hosts
|
||||
(New-MockJob 'Backup' $now.AddHours(-20) $now.AddHours(-19.5) 'Completed' -Items @(
|
||||
(New-MockItem 'C:' 'Volume' 40GB), (New-MockItem 'D:' 'Volume' 200GB))),
|
||||
(New-MockJob 'Recovery' $now.AddHours(-5) $now.AddHours(-4) 'Completed'), # a restore: must be ignored
|
||||
(New-MockJob 'Backup' $now.AddDays(-5) $now.AddDays(-5).AddHours(1) 'Completed') # outside 48h lookback
|
||||
)
|
||||
if ($RunningJobFinished) {
|
||||
$global:MockWsbPrevious += New-MockJob 'Backup' $now.AddMinutes(-10) $now.AddMinutes(-2) 'Completed' -Items @(New-VmList 10GB)
|
||||
}
|
||||
|
||||
$global:MockWsbSummary = [pscustomobject]@{
|
||||
NextBackupTime = $now.Date.AddHours(21).AddDays($(if ($now.Hour -ge 21) { 1 } else { 0 }))
|
||||
NumberOfVersions = 1
|
||||
LastSuccessfulBackupTime = $now.AddHours(-12)
|
||||
LastBackupResultHR = [uint32]0
|
||||
CurrentOperationStatus = $(if ($WithRunningJob) { 'BackupInProgress' } else { 'NoOperationInProgress' })
|
||||
}
|
||||
|
||||
# Same limit as the real cmdlet
|
||||
function Get-WBJob { param([ValidateRange(1, 100)][int]$Previous) if ($PSBoundParameters.ContainsKey('Previous')) { $global:MockWsbPrevious } else { $global:MockWsbRunning } }
|
||||
function Get-WBSummary { $global:MockWsbSummary }
|
||||
function Get-WBPolicy { [pscustomobject]@{ PolicyName = 'Scheduled' } }
|
||||
function Get-WBSchedule { param($Policy) @($global:MockWsbSummary.NextBackupTime) }
|
||||
@@ -0,0 +1,23 @@
|
||||
[Unit]
|
||||
Description=Backup Dashboard API
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
User=backupdash
|
||||
Group=backupdash
|
||||
WorkingDirectory=/opt/backup-dashboard/backend
|
||||
# Settings live in this file so manage.py reads exactly the same ones (see deploy/backup-dashboard.env)
|
||||
EnvironmentFile=/etc/backup-dashboard/backup-dashboard.env
|
||||
# Bound to localhost: only nginx can reach it, so nginx's auth can't be bypassed from outside.
|
||||
# One worker is plenty at this scale and keeps SQLite writes simple.
|
||||
ExecStart=/opt/backup-dashboard/venv/bin/uvicorn --factory app.main:create_app --host 127.0.0.1 --port 8000 --proxy-headers
|
||||
Restart=on-failure
|
||||
|
||||
StateDirectory=backup-dashboard
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,6 @@
|
||||
# Install as /etc/backup-dashboard/backup-dashboard.env
|
||||
# Read by both the API service and manage.py, so they always use the same database.
|
||||
BACKUP_DASH_DB=/var/lib/backup-dashboard/backup-dashboard.db
|
||||
BACKUP_DASH_RETENTION_DAYS=90
|
||||
# Microsoft 365 sign-in names (comma-separated) allowed into the admin pages (API key management)
|
||||
BACKUP_DASH_ADMINS=[email protected]
|
||||
@@ -0,0 +1,98 @@
|
||||
server {
|
||||
listen 80;
|
||||
server_name backups.example.com;
|
||||
# Session cookies and API keys must never cross plain HTTP
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl;
|
||||
http2 on;
|
||||
server_name backups.example.com;
|
||||
|
||||
# Paths as created by `certbot certonly --nginx -d backups.example.com`
|
||||
ssl_certificate /etc/letsencrypt/live/backups.example.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/backups.example.com/privkey.pem;
|
||||
|
||||
root /var/www/backup-dashboard/public;
|
||||
index index.html;
|
||||
|
||||
# Strict CSP works because there is no inline JS/CSS
|
||||
add_header Content-Security-Policy "default-src 'self'; object-src 'none'; base-uri 'self'; frame-ancestors 'none'" always;
|
||||
add_header X-Content-Type-Options "nosniff" always;
|
||||
add_header Referrer-Policy "no-referrer" always;
|
||||
add_header Strict-Transport-Security "max-age=31536000" always;
|
||||
|
||||
# ---------- Microsoft 365 sign-in ----------
|
||||
# oauth2-proxy (deploy/oauth2-proxy.cfg) runs on 127.0.0.1:4180 and does the sign-in with
|
||||
# Entra ID. Every request below asks it "is this browser signed in?" via auth_request.
|
||||
|
||||
# Its own pages: /oauth2/start (go to Microsoft), /oauth2/callback, /oauth2/sign_out
|
||||
location /oauth2/ {
|
||||
proxy_pass http://127.0.0.1:4180;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
# Where to return after signing in: the page that was asked for
|
||||
proxy_set_header X-Auth-Request-Redirect $request_uri;
|
||||
# Entra ID session cookies are large
|
||||
proxy_buffer_size 16k;
|
||||
proxy_buffers 8 16k;
|
||||
}
|
||||
|
||||
# The sign-in check itself, only reachable as an auth_request subrequest
|
||||
location = /oauth2/auth {
|
||||
internal;
|
||||
proxy_pass http://127.0.0.1:4180;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-Uri $request_uri;
|
||||
proxy_set_header Content-Length "";
|
||||
proxy_pass_request_body off;
|
||||
proxy_buffer_size 16k;
|
||||
proxy_buffers 8 16k;
|
||||
}
|
||||
|
||||
# ---------- Dashboard ----------
|
||||
|
||||
location / {
|
||||
auth_request /oauth2/auth;
|
||||
# Not signed in: straight to Microsoft's sign-in, then back to this page
|
||||
error_page 401 = /oauth2/start;
|
||||
# Make browsers check for a newer page/script on every load (a cheap 304 when unchanged),
|
||||
# so an update shows up at once. (expires, not add_header: an add_header here would drop
|
||||
# the security headers above.)
|
||||
expires -1;
|
||||
try_files $uri $uri/ /index.html;
|
||||
}
|
||||
|
||||
location /api/ {
|
||||
auth_request /oauth2/auth;
|
||||
# No redirect here: API calls from the page get a plain 401, and the page asks the user to
|
||||
# reload, which goes through sign-in above.
|
||||
|
||||
# Who's signed in, for the dashboard's "signed in as" (proxy_set_header replaces any value
|
||||
# the browser sent, so these can't be spoofed)
|
||||
auth_request_set $auth_email $upstream_http_x_auth_request_email;
|
||||
auth_request_set $auth_user $upstream_http_x_auth_request_user;
|
||||
auth_request_set $auth_name $upstream_http_x_auth_request_preferred_username;
|
||||
proxy_set_header X-Forwarded-Email $auth_email;
|
||||
proxy_set_header X-Forwarded-User $auth_user;
|
||||
proxy_set_header X-Forwarded-Preferred-Username $auth_name;
|
||||
|
||||
proxy_pass http://127.0.0.1:8000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
|
||||
# Collectors authenticate with their own API key (Authorization: Bearer …), checked by the app,
|
||||
# so no Microsoft sign-in here
|
||||
location = /api/v1/ingest {
|
||||
client_max_body_size 256k;
|
||||
proxy_pass http://127.0.0.1:8000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
# oauth2-proxy config for Backup Dashboard's Microsoft 365 (Entra ID) sign-in.
|
||||
# Install as /etc/oauth2-proxy/backup-dashboard.cfg, readable only by root and the oauth2-proxy
|
||||
# user (it holds the cookie secret). Needs oauth2-proxy 7.8 or newer for the entra-id provider.
|
||||
# Replace the four <...> values; see the README's "Microsoft 365 sign-in" section.
|
||||
|
||||
## Where it listens: only nginx talks to it
|
||||
http_address = "127.0.0.1:4180"
|
||||
reverse_proxy = true
|
||||
# Only nginx (on this machine) may supply X-Forwarded-* headers
|
||||
trusted_proxy_ips = ["127.0.0.1/32", "::1/128"]
|
||||
|
||||
## Entra ID app registration (single tenant)
|
||||
provider = "entra-id"
|
||||
oidc_issuer_url = "https://login.microsoftonline.com/<TENANT_ID>/v2.0"
|
||||
client_id = "<APPLICATION_CLIENT_ID>"
|
||||
# The client secret's *value*, in a file of its own, root:oauth2-proxy, mode 640
|
||||
client_secret_file = "/etc/oauth2-proxy/client-secret"
|
||||
redirect_url = "https://<DASHBOARD_HOST>/oauth2/callback"
|
||||
|
||||
## Who may sign in
|
||||
# Anyone in the tenant that Entra lets through. Limit it in Entra itself: Enterprise
|
||||
# applications -> this app -> Properties -> "Assignment required" = Yes, then assign users or a
|
||||
# group. Entra then refuses everyone else before they ever reach the dashboard.
|
||||
email_domains = ["*"]
|
||||
# The user's sign-in name (UPN), which every work account has. The "email" claim is only present
|
||||
# when the account has a mailbox address.
|
||||
oidc_email_claim = "preferred_username"
|
||||
|
||||
## Session cookie
|
||||
# Generate with: openssl rand -base64 32 | tr -- '+/' '-_'
|
||||
cookie_secret = "<COOKIE_SECRET>"
|
||||
cookie_name = "_backup_dashboard"
|
||||
cookie_secure = true
|
||||
cookie_samesite = "lax"
|
||||
# Sign in again (usually silently, via the Microsoft session) after this long
|
||||
cookie_expire = "12h"
|
||||
|
||||
## Behaviour with nginx's auth_request
|
||||
# Auth-only: nginx serves the dashboard; oauth2-proxy just answers "signed in or not"
|
||||
upstreams = ["static://202"]
|
||||
# Put the user in X-Auth-Request-* headers on /oauth2/auth responses, for nginx to pass on
|
||||
set_xauthrequest = true
|
||||
# Go straight to Microsoft instead of showing a "Sign in with..." page first
|
||||
skip_provider_button = true
|
||||
# Allow "Sign out" to continue to Microsoft's sign-out page
|
||||
whitelist_domains = ["login.microsoftonline.com"]
|
||||
@@ -0,0 +1,18 @@
|
||||
[Unit]
|
||||
Description=oauth2-proxy (Microsoft 365 sign-in for Backup Dashboard)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
User=oauth2-proxy
|
||||
Group=oauth2-proxy
|
||||
ExecStart=/usr/local/bin/oauth2-proxy --config=/etc/oauth2-proxy/backup-dashboard.cfg
|
||||
Restart=on-failure
|
||||
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,103 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>API keys · Backup Monitor</title>
|
||||
<link rel="stylesheet" href="/css/styles.css">
|
||||
<script type="module" src="/js/admin.js"></script>
|
||||
</head>
|
||||
<body>
|
||||
<header class="topbar">
|
||||
<div class="topbar__nav">
|
||||
<a id="back-link" class="btn" href="/">← Dashboard</a>
|
||||
<h1 class="topbar__title">API keys</h1>
|
||||
</div>
|
||||
<div class="topbar__meta">
|
||||
<span id="signed-in" class="topbar__user" hidden>
|
||||
<span id="signed-in-email" class="muted"></span>
|
||||
<a id="sign-out" class="btn" href="#">Sign out</a>
|
||||
</span>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main class="container container--narrow">
|
||||
<div id="error-banner" class="banner banner--error" role="alert" hidden></div>
|
||||
<p id="not-admin" class="banner banner--info" hidden>
|
||||
Your account isn't an admin. Admins are the sign-in names listed in <code>BACKUP_DASH_ADMINS</code>
|
||||
in the server's settings file.
|
||||
</p>
|
||||
|
||||
<div id="admin-content" hidden>
|
||||
<section class="panel" aria-labelledby="new-key-title">
|
||||
<h2 id="new-key-title" class="panel__title">New key</h2>
|
||||
<p class="muted">Create one key per collector install, so each can be revoked on its own.</p>
|
||||
<form id="new-key-form" class="new-key">
|
||||
<div class="field">
|
||||
<label for="key-name">Name</label>
|
||||
<span id="key-name-hint" class="hint">For example <code>veeam-acme</code> or <code>mars-acme-fs01</code>:
|
||||
letters, digits, <code>.</code>, <code>_</code> and <code>-</code>.</span>
|
||||
<input id="key-name" name="name" class="input" required maxlength="64"
|
||||
pattern="[A-Za-z0-9][A-Za-z0-9._\-]{0,63}" autocomplete="off" spellcheck="false"
|
||||
aria-describedby="key-name-hint" aria-errormessage="key-name-error">
|
||||
<div id="key-name-error" class="error-msg">
|
||||
Use up to 64 letters, digits, <code>.</code>, <code>_</code> or <code>-</code>, starting with a letter or digit.
|
||||
</div>
|
||||
</div>
|
||||
<button id="create-btn" class="btn btn--primary" type="submit">Create key</button>
|
||||
<p id="create-error" class="form-error" role="alert" hidden></p>
|
||||
</form>
|
||||
|
||||
<div id="new-key-result" class="secret" hidden>
|
||||
<h3 id="new-key-result-title" class="secret__title" tabindex="-1">Copy the key for <span id="new-key-name"></span> now</h3>
|
||||
<p>It won't be shown again: the dashboard only keeps a hash of it. Paste it when the collector's
|
||||
installer asks for the API key.</p>
|
||||
<div class="secret__row">
|
||||
<code id="new-key-secret" class="secret__value"></code>
|
||||
<button id="copy-btn" class="btn" type="button">Copy</button>
|
||||
</div>
|
||||
<button id="done-btn" class="btn" type="button">Done, hide it</button>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section aria-labelledby="keys-title">
|
||||
<h2 id="keys-title" class="panel__title" tabindex="-1">Keys</h2>
|
||||
<div class="table-wrap">
|
||||
<table class="jobs keys">
|
||||
<thead>
|
||||
<tr>
|
||||
<th scope="col">Name</th>
|
||||
<th scope="col">Status</th>
|
||||
<th scope="col">Last used</th>
|
||||
<th scope="col">Created</th>
|
||||
<th scope="col"><span class="visually-hidden">Actions</span></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="keys-body"></tbody>
|
||||
</table>
|
||||
<p id="keys-empty" class="empty" hidden>No keys yet. Create one above for each collector.</p>
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<!-- Announces results (created, copied, revoked) to screen readers -->
|
||||
<p id="status" class="visually-hidden" role="status"></p>
|
||||
</main>
|
||||
|
||||
<dialog id="revoke-dialog" class="modal modal--small" aria-labelledby="revoke-title">
|
||||
<form method="dialog">
|
||||
<div class="modal__header">
|
||||
<h2 id="revoke-title" class="modal__title">Revoke <span id="revoke-name"></span>?</h2>
|
||||
</div>
|
||||
<div class="modal__body">
|
||||
<p>Collectors using this key will be refused from their next run. This can't be undone: to
|
||||
reconnect them, create a new key and re-run their installer.</p>
|
||||
<div class="dialog-actions">
|
||||
<button class="btn" value="cancel" autofocus>Cancel</button>
|
||||
<button class="btn btn--danger" value="revoke">Revoke key</button>
|
||||
</div>
|
||||
</div>
|
||||
</form>
|
||||
</dialog>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,525 @@
|
||||
:root {
|
||||
color-scheme: light dark;
|
||||
--bg: #f6f7f9;
|
||||
--surface: #ffffff;
|
||||
--surface-alt: #f0f2f5;
|
||||
--border: #d9dde3;
|
||||
--text: #1f2328;
|
||||
--muted: #656d76;
|
||||
--accent: #0969da;
|
||||
|
||||
/* Status palette: each has fg/bg/border so badges stay legible in both themes */
|
||||
--success-fg: #116329; --success-bg: #dafbe1; --success-bd: #4ac26b;
|
||||
--warning-fg: #7d4e00; --warning-bg: #fff8c5; --warning-bd: #d4a72c;
|
||||
--failed-fg: #a40e26; --failed-bg: #ffebe9; --failed-bd: #ff8182;
|
||||
--unknown-fg: #57606a; --unknown-bg: #eaeef2; --unknown-bd: #afb8c1;
|
||||
--running-fg: #0550ae; --running-bg: #ddf4ff; --running-bd: #54aeff;
|
||||
|
||||
--radius: 8px;
|
||||
--shadow: 0 1px 2px rgb(0 0 0 / 6%), 0 4px 12px rgb(0 0 0 / 4%);
|
||||
font-family: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
|
||||
font-size: 15px;
|
||||
line-height: 1.45;
|
||||
}
|
||||
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root {
|
||||
--bg: #0d1117;
|
||||
--surface: #161b22;
|
||||
--surface-alt: #1c2128;
|
||||
--border: #30363d;
|
||||
--text: #e6edf3;
|
||||
--muted: #8d96a0;
|
||||
--accent: #4493f8;
|
||||
--success-fg: #6fdd8b; --success-bg: #12261e; --success-bd: #2ea043;
|
||||
--warning-fg: #f2cc60; --warning-bg: #272115; --warning-bd: #9e6a03;
|
||||
--failed-fg: #ffa198; --failed-bg: #2d1214; --failed-bd: #da3633;
|
||||
--unknown-fg: #9da7b3; --unknown-bg: #21262d; --unknown-bd: #484f58;
|
||||
--running-fg: #79c0ff; --running-bg: #0c2d4b; --running-bd: #1f6feb;
|
||||
--shadow: none;
|
||||
}
|
||||
}
|
||||
|
||||
*, *::before, *::after { box-sizing: border-box; }
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.muted { color: var(--muted); }
|
||||
|
||||
.visually-hidden {
|
||||
position: absolute !important;
|
||||
width: 1px; height: 1px;
|
||||
overflow: hidden;
|
||||
clip: rect(0 0 0 0);
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
/* ---------- Layout ---------- */
|
||||
.topbar {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 1rem;
|
||||
flex-wrap: wrap;
|
||||
padding: 0.75rem 1.5rem;
|
||||
background: var(--surface);
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
.topbar__title { margin: 0; font-size: 1.25rem; }
|
||||
.topbar__meta { display: flex; align-items: center; gap: 0.75rem; font-size: 0.875rem; }
|
||||
/* Only when shown: display:flex on the bare class would override the hidden attribute */
|
||||
.topbar__user:not([hidden]) {
|
||||
display: flex; align-items: center; gap: 0.75rem;
|
||||
padding-left: 0.75rem; border-left: 1px solid var(--border);
|
||||
}
|
||||
a.btn { text-decoration: none; }
|
||||
|
||||
.container {
|
||||
max-width: 1400px;
|
||||
margin: 0 auto;
|
||||
padding: 1.25rem 1.5rem 3rem;
|
||||
}
|
||||
|
||||
/* ---------- Controls ---------- */
|
||||
.btn {
|
||||
font: inherit;
|
||||
padding: 0.4rem 0.9rem;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
background: var(--surface-alt);
|
||||
color: var(--text);
|
||||
cursor: pointer;
|
||||
}
|
||||
.btn:hover { border-color: var(--muted); }
|
||||
.btn:disabled { opacity: 0.6; cursor: progress; }
|
||||
.btn--icon { padding: 0.25rem 0.55rem; line-height: 1; }
|
||||
|
||||
.input {
|
||||
font: inherit;
|
||||
width: min(100%, 360px);
|
||||
padding: 0.45rem 0.75rem;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
background: var(--surface);
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
|
||||
|
||||
.toolbar { margin: 1rem 0; }
|
||||
|
||||
/* ---------- Banner ---------- */
|
||||
.banner {
|
||||
padding: 0.75rem 1rem;
|
||||
border-radius: var(--radius);
|
||||
margin-bottom: 1rem;
|
||||
border: 1px solid;
|
||||
}
|
||||
.banner--error { background: var(--failed-bg); color: var(--failed-fg); border-color: var(--failed-bd); }
|
||||
|
||||
/* ---------- Summary cards ---------- */
|
||||
.summary {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
|
||||
gap: 0.75rem;
|
||||
}
|
||||
.summary-card {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: flex-start;
|
||||
gap: 0.15rem;
|
||||
padding: 0.9rem 1rem;
|
||||
font: inherit;
|
||||
text-align: left;
|
||||
color: var(--text);
|
||||
background: var(--surface);
|
||||
border: 1px solid var(--border);
|
||||
border-left-width: 4px;
|
||||
border-radius: var(--radius);
|
||||
box-shadow: var(--shadow);
|
||||
cursor: pointer;
|
||||
}
|
||||
.summary-card[aria-pressed="true"] { outline: 2px solid var(--accent); outline-offset: -1px; }
|
||||
.summary-card__count { font-size: 1.75rem; font-weight: 600; font-variant-numeric: tabular-nums; }
|
||||
.summary-card__label { font-size: 0.85rem; color: var(--muted); }
|
||||
.summary-card--failed { border-left-color: var(--failed-bd); }
|
||||
.summary-card--warning { border-left-color: var(--warning-bd); }
|
||||
.summary-card--running { border-left-color: var(--running-bd); }
|
||||
.summary-card--success { border-left-color: var(--success-bd); }
|
||||
.summary-card--overdue { border-left-color: var(--unknown-bd); }
|
||||
|
||||
/* ---------- Table ---------- */
|
||||
.table-wrap {
|
||||
overflow-x: auto;
|
||||
background: var(--surface);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
box-shadow: var(--shadow);
|
||||
}
|
||||
.jobs { width: 100%; border-collapse: collapse; }
|
||||
.jobs th, .jobs td {
|
||||
padding: 0.65rem 1rem;
|
||||
text-align: left;
|
||||
border-bottom: 1px solid var(--border);
|
||||
white-space: nowrap;
|
||||
}
|
||||
.jobs tbody tr:last-child td { border-bottom: none; }
|
||||
.jobs tbody tr:hover { background: var(--surface-alt); }
|
||||
.jobs th { background: var(--surface-alt); font-size: 0.8rem; text-transform: uppercase; letter-spacing: 0.03em; }
|
||||
|
||||
.sort {
|
||||
all: unset;
|
||||
cursor: pointer;
|
||||
color: var(--muted);
|
||||
font-weight: 600;
|
||||
}
|
||||
.sort::after { content: " ↕"; opacity: 0.4; }
|
||||
th[aria-sort="ascending"] .sort::after { content: " ↑"; opacity: 1; }
|
||||
th[aria-sort="descending"] .sort::after { content: " ↓"; opacity: 1; }
|
||||
th[aria-sort="ascending"] .sort,
|
||||
th[aria-sort="descending"] .sort { color: var(--text); }
|
||||
|
||||
.time { display: flex; flex-direction: column; }
|
||||
.time__rel { font-size: 0.8rem; color: var(--muted); }
|
||||
|
||||
.empty { padding: 2rem; text-align: center; color: var(--muted); margin: 0; }
|
||||
|
||||
/* ---------- Status badges ---------- */
|
||||
.badge {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.35rem;
|
||||
padding: 0.2rem 0.65rem;
|
||||
font: inherit;
|
||||
font-size: 0.85rem;
|
||||
font-weight: 600;
|
||||
border: 1px solid;
|
||||
border-radius: 999px;
|
||||
}
|
||||
button.badge { cursor: pointer; }
|
||||
button.badge:hover { filter: brightness(0.95); }
|
||||
/* Icon + text, not colour alone, so it's readable for colour-blind users */
|
||||
.badge__icon { font-size: 0.75rem; }
|
||||
.badge--success { color: var(--success-fg); background: var(--success-bg); border-color: var(--success-bd); }
|
||||
.badge--warning { color: var(--warning-fg); background: var(--warning-bg); border-color: var(--warning-bd); }
|
||||
.badge--failed { color: var(--failed-fg); background: var(--failed-bg); border-color: var(--failed-bd); }
|
||||
.badge--unknown { color: var(--unknown-fg); background: var(--unknown-bg); border-color: var(--unknown-bd); }
|
||||
.badge--running { color: var(--running-fg); background: var(--running-bg); border-color: var(--running-bd); }
|
||||
@media (prefers-reduced-motion: no-preference) {
|
||||
.badge--running .badge__icon { animation: spin 2s linear infinite; }
|
||||
}
|
||||
@keyframes spin { to { transform: rotate(360deg); } }
|
||||
|
||||
.tag-overdue {
|
||||
margin-left: 0.5rem;
|
||||
padding: 0.05rem 0.45rem;
|
||||
font-size: 0.75rem;
|
||||
font-weight: 600;
|
||||
border-radius: 4px;
|
||||
color: var(--failed-fg);
|
||||
background: var(--failed-bg);
|
||||
}
|
||||
|
||||
/* ---------- Modal ---------- */
|
||||
.modal {
|
||||
width: min(720px, calc(100vw - 2rem));
|
||||
max-height: calc(100vh - 4rem);
|
||||
padding: 0;
|
||||
color: var(--text);
|
||||
background: var(--surface);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 12px;
|
||||
}
|
||||
.modal::backdrop { background: rgb(0 0 0 / 45%); }
|
||||
.modal__header {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: flex-start;
|
||||
gap: 1rem;
|
||||
padding: 1rem 1.25rem;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
.modal__title { margin: 0; font-size: 1.15rem; }
|
||||
.modal__header p { margin: 0.15rem 0 0; }
|
||||
.modal__body { padding: 1rem 1.25rem 1.25rem; }
|
||||
|
||||
.detail-grid {
|
||||
display: grid;
|
||||
grid-template-columns: max-content 1fr;
|
||||
gap: 0.4rem 1.25rem;
|
||||
margin: 1rem 0;
|
||||
}
|
||||
.detail-grid dt { color: var(--muted); }
|
||||
.detail-grid dd { margin: 0; overflow-wrap: anywhere; }
|
||||
|
||||
.detail-section-title { font-size: 0.85rem; text-transform: uppercase; color: var(--muted); margin: 1.25rem 0 0.5rem; }
|
||||
|
||||
.message {
|
||||
margin: 0;
|
||||
padding: 0.75rem 1rem;
|
||||
max-height: 320px;
|
||||
overflow: auto;
|
||||
white-space: pre-wrap; /* preserve backup-software line breaks */
|
||||
overflow-wrap: anywhere;
|
||||
font-family: ui-monospace, SFMono-Regular, Consolas, monospace;
|
||||
font-size: 0.85rem;
|
||||
background: var(--surface-alt);
|
||||
border: 1px solid var(--border);
|
||||
border-left-width: 4px;
|
||||
border-radius: 6px;
|
||||
}
|
||||
.message--failed { border-left-color: var(--failed-bd); }
|
||||
.message--warning { border-left-color: var(--warning-bd); }
|
||||
.message--success { border-left-color: var(--success-bd); }
|
||||
.message--running { border-left-color: var(--running-bd); }
|
||||
|
||||
.detail-status { display: flex; align-items: center; gap: 0.6rem; flex-wrap: wrap; }
|
||||
|
||||
.link {
|
||||
all: unset;
|
||||
color: var(--accent);
|
||||
cursor: pointer;
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
/* ---------- Run history ---------- */
|
||||
.history-stats { margin: 0 0 0.6rem; font-size: 0.9rem; }
|
||||
|
||||
/* One cell per day, oldest → today. Colour + tooltip; the runs table below is the accessible detail. */
|
||||
.day-strip {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(30, 1fr);
|
||||
gap: 3px;
|
||||
}
|
||||
.day {
|
||||
height: 28px;
|
||||
padding: 0;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 3px;
|
||||
background: var(--surface-alt);
|
||||
cursor: pointer;
|
||||
}
|
||||
.day:disabled { cursor: default; opacity: 0.6; }
|
||||
.day:not(:disabled):hover { filter: brightness(0.9); }
|
||||
.day[aria-pressed="true"] { outline: 2px solid var(--accent); outline-offset: 1px; }
|
||||
.day--success { background: var(--success-bd); border-color: var(--success-bd); }
|
||||
.day--warning { background: var(--warning-bd); border-color: var(--warning-bd); }
|
||||
.day--failed { background: var(--failed-bd); border-color: var(--failed-bd); }
|
||||
.day--running { background: var(--running-bd); border-color: var(--running-bd); }
|
||||
.day--unknown { background: var(--unknown-bd); border-color: var(--unknown-bd); }
|
||||
.day-strip__ticks {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
margin-top: 0.25rem;
|
||||
font-size: 0.75rem;
|
||||
}
|
||||
|
||||
.history-filter { margin: 0.75rem 0 0; font-size: 0.85rem; }
|
||||
|
||||
.runs-wrap {
|
||||
margin-top: 0.75rem;
|
||||
max-height: 300px;
|
||||
overflow: auto;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
}
|
||||
.runs { width: 100%; border-collapse: collapse; font-size: 0.85rem; }
|
||||
.runs th, .runs td {
|
||||
padding: 0.4rem 0.75rem;
|
||||
text-align: left;
|
||||
white-space: nowrap;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
.runs th {
|
||||
position: sticky;
|
||||
top: 0;
|
||||
background: var(--surface-alt);
|
||||
color: var(--muted);
|
||||
font-size: 0.75rem;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
.runs tbody tr:last-child td { border-bottom: none; }
|
||||
.runs tr.is-selected td { background: color-mix(in srgb, var(--accent) 16%, transparent); }
|
||||
.runs .badge { font-size: 0.75rem; padding: 0.1rem 0.5rem; }
|
||||
/* ---------- Admin page ---------- */
|
||||
.container--narrow { max-width: 1100px; }
|
||||
.topbar__nav { display: flex; align-items: center; gap: 1rem; }
|
||||
|
||||
.banner--info { background: var(--surface-alt); color: var(--text); border-color: var(--border); }
|
||||
|
||||
.panel {
|
||||
margin-bottom: 1.5rem;
|
||||
padding: 1rem 1.25rem 1.25rem;
|
||||
background: var(--surface);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
box-shadow: var(--shadow);
|
||||
}
|
||||
.panel__title { font-size: 1.05rem; margin: 0 0 0.25rem; }
|
||||
section > .panel__title { margin: 0 0 0.75rem; }
|
||||
.panel__title:focus { outline: none; } /* focused from script after a change, not by the user */
|
||||
.panel p { margin: 0 0 0.75rem; }
|
||||
|
||||
code {
|
||||
font-family: ui-monospace, SFMono-Regular, Consolas, monospace;
|
||||
font-size: 0.9em;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
/* New key form: label, hint above the input (so autocomplete can't cover it), error below */
|
||||
.new-key { display: flex; flex-wrap: wrap; align-items: flex-end; gap: 0.75rem; }
|
||||
.field { display: flex; flex-direction: column; gap: 0.25rem; flex: 1 1 22rem; max-width: 30rem; }
|
||||
.field label { font-weight: 600; }
|
||||
.hint { font-size: 0.85rem; color: var(--muted); }
|
||||
.field .input { width: 100%; }
|
||||
.error-msg { display: none; font-size: 0.85rem; color: var(--failed-fg); }
|
||||
/* Only after the user has interacted with the field (or tried to submit) */
|
||||
.field .input:user-invalid { border-color: var(--failed-bd); background: var(--failed-bg); }
|
||||
.field .input:user-invalid + .error-msg { display: block; }
|
||||
.form-error { flex-basis: 100%; margin: 0; color: var(--failed-fg); }
|
||||
|
||||
.btn--primary { background: var(--accent); border-color: var(--accent); color: #fff; font-weight: 600; }
|
||||
.btn--primary:hover { filter: brightness(1.08); border-color: var(--accent); }
|
||||
.btn--danger { background: var(--failed-bd); border-color: var(--failed-bd); color: #fff; font-weight: 600; }
|
||||
.btn--danger-outline { color: var(--failed-fg); border-color: var(--failed-bd); background: transparent; }
|
||||
.btn--danger-outline:hover { background: var(--failed-bg); border-color: var(--failed-bd); }
|
||||
|
||||
/* The one-time key */
|
||||
.secret {
|
||||
margin-top: 1rem;
|
||||
padding: 1rem;
|
||||
border: 1px solid var(--warning-bd);
|
||||
border-left-width: 4px;
|
||||
border-radius: 6px;
|
||||
background: var(--warning-bg);
|
||||
}
|
||||
.secret__title { margin: 0 0 0.5rem; font-size: 1rem; }
|
||||
.secret__title:focus { outline: none; }
|
||||
.secret__row { display: flex; gap: 0.5rem; align-items: stretch; margin-bottom: 0.75rem; flex-wrap: wrap; }
|
||||
.secret__value {
|
||||
flex: 1 1 20rem;
|
||||
padding: 0.45rem 0.75rem;
|
||||
background: var(--surface);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
user-select: all; /* one click selects the whole key */
|
||||
}
|
||||
|
||||
/* Key table */
|
||||
.keys td { vertical-align: top; }
|
||||
.keys td .badge { margin-bottom: 0.25rem; }
|
||||
.keys td > .time__rel { display: block; }
|
||||
.keys .actions { text-align: right; width: 1%; }
|
||||
.keys tr.is-revoked td { color: var(--muted); }
|
||||
.keys tr.is-revoked code { text-decoration: line-through; }
|
||||
.stale-note { display: block; font-size: 0.8rem; font-weight: 600; color: var(--warning-fg); }
|
||||
|
||||
/* Confirm dialog */
|
||||
.modal--small { width: min(460px, calc(100vw - 2rem)); }
|
||||
.modal--small .modal__body p { margin-top: 0; }
|
||||
.dialog-actions { display: flex; justify-content: flex-end; gap: 0.5rem; margin-top: 1rem; }
|
||||
|
||||
/* ---------- Report page ---------- */
|
||||
.report-controls {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
align-items: flex-end;
|
||||
gap: 0.75rem 1rem;
|
||||
margin-bottom: 1.25rem;
|
||||
}
|
||||
.report-controls__field { display: flex; flex-direction: column; gap: 0.25rem; }
|
||||
.report-controls__field label { font-weight: 600; font-size: 0.9rem; }
|
||||
.report-controls__field .input { width: auto; min-width: 14rem; }
|
||||
.report-controls__actions { display: flex; gap: 0.5rem; margin-left: auto; }
|
||||
.btn[aria-disabled="true"] { opacity: 0.55; pointer-events: none; }
|
||||
|
||||
.report__header { margin-bottom: 1rem; }
|
||||
.report__title { margin: 0; font-size: 1.4rem; }
|
||||
.report__period { margin: 0.25rem 0 0; color: var(--muted); }
|
||||
.report__section-title { font-size: 1.05rem; margin: 1.75rem 0 0.75rem; }
|
||||
.report__footer { margin-top: 1.5rem; font-size: 0.85rem; }
|
||||
|
||||
.report-summary {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(9rem, 1fr));
|
||||
gap: 0.75rem;
|
||||
margin: 0;
|
||||
}
|
||||
.report-summary__item {
|
||||
padding: 0.75rem 1rem;
|
||||
background: var(--surface);
|
||||
border: 1px solid var(--border);
|
||||
border-left-width: 4px;
|
||||
border-radius: var(--radius);
|
||||
}
|
||||
.report-summary__item dt { font-size: 0.85rem; color: var(--muted); }
|
||||
.report-summary__item dd { margin: 0; font-size: 1.5rem; font-weight: 600; font-variant-numeric: tabular-nums; }
|
||||
.report-summary__item--warning { border-left-color: var(--warning-bd); }
|
||||
.report-summary__item--failed { border-left-color: var(--failed-bd); }
|
||||
|
||||
/* Row headers (job names) are <th> for accessibility, but shouldn't look like the column headings */
|
||||
.report-table tbody th {
|
||||
background: none; color: var(--text); font-size: inherit; font-weight: 600;
|
||||
text-transform: none; letter-spacing: normal;
|
||||
}
|
||||
.report-table .num { text-align: right; font-variant-numeric: tabular-nums; }
|
||||
.report-table td > .time__rel { display: block; }
|
||||
.report-table .badge { font-size: 0.75rem; padding: 0.1rem 0.5rem; }
|
||||
.report-alert { color: var(--failed-fg); font-weight: 600; }
|
||||
.report-message { white-space: pre-wrap; overflow-wrap: anywhere; min-width: 20rem; font-size: 0.85rem; }
|
||||
.nowrap { white-space: nowrap; }
|
||||
|
||||
.report-legend {
|
||||
display: flex; flex-wrap: wrap; gap: 0.25rem 1rem;
|
||||
margin: 0 0 0.75rem; padding: 0; list-style: none; font-size: 0.85rem; color: var(--muted);
|
||||
}
|
||||
.report-legend li { display: flex; align-items: center; gap: 0.35rem; }
|
||||
.report-legend .day { display: inline-block; width: 0.9rem; height: 0.9rem; }
|
||||
|
||||
.report-daily__job { margin-bottom: 0.75rem; }
|
||||
.report-daily__name { font-size: 0.9rem; font-weight: 600; margin-bottom: 0.25rem; }
|
||||
/* One cell per day of the report's period */
|
||||
.report-strip { grid-template-columns: repeat(var(--days, 30), 1fr); gap: 2px; }
|
||||
.report-strip .day { display: block; height: 18px; cursor: default; }
|
||||
/* Before the dashboard knew of the job: outlined only, so it doesn't read as a missed backup */
|
||||
.day--untracked { background: transparent; border: 1px dashed var(--muted); opacity: 0.5; }
|
||||
|
||||
/* ---------- Printing ---------- */
|
||||
@media print {
|
||||
/* Always print light, whatever the screen's theme (same values as the light tokens above) */
|
||||
:root {
|
||||
color-scheme: light;
|
||||
--bg: #ffffff;
|
||||
--surface: #ffffff;
|
||||
--surface-alt: #f0f2f5;
|
||||
--border: #c9ced6;
|
||||
--text: #1f2328;
|
||||
--muted: #57606a;
|
||||
--accent: #0969da;
|
||||
--success-fg: #116329; --success-bg: #dafbe1; --success-bd: #4ac26b;
|
||||
--warning-fg: #7d4e00; --warning-bg: #fff8c5; --warning-bd: #d4a72c;
|
||||
--failed-fg: #a40e26; --failed-bg: #ffebe9; --failed-bd: #ff8182;
|
||||
--unknown-fg: #57606a; --unknown-bg: #eaeef2; --unknown-bd: #afb8c1;
|
||||
--running-fg: #0550ae; --running-bg: #ddf4ff; --running-bd: #54aeff;
|
||||
--shadow: none;
|
||||
font-size: 11pt;
|
||||
}
|
||||
@page { margin: 14mm; }
|
||||
body { background: #fff; color: var(--text); }
|
||||
.no-print, .topbar, .banner { display: none !important; }
|
||||
.container { max-width: none; padding: 0; }
|
||||
.table-wrap { overflow: visible; box-shadow: none; }
|
||||
.jobs th, .jobs td { white-space: normal; padding: 0.35rem 0.5rem; }
|
||||
.jobs tbody tr:hover { background: none; }
|
||||
.report-message { min-width: 0; }
|
||||
/* Keep things whole across page breaks */
|
||||
tr, .report-daily__job, .report-summary__item, .report-legend { break-inside: avoid; }
|
||||
.report__section-title { break-after: avoid; }
|
||||
thead { display: table-header-group; } /* repeat column headings on each page */
|
||||
/* Browsers drop backgrounds when printing unless told they carry meaning */
|
||||
.day, .badge, .report-summary__item { print-color-adjust: exact; -webkit-print-color-adjust: exact; }
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Backup Monitor</title>
|
||||
<link rel="stylesheet" href="/css/styles.css">
|
||||
<!-- type="module" is deferred by default; no inline JS so a strict CSP works -->
|
||||
<script type="module" src="/js/app.js"></script>
|
||||
</head>
|
||||
<body>
|
||||
<header class="topbar">
|
||||
<h1 class="topbar__title">Backup Monitor</h1>
|
||||
<div class="topbar__meta">
|
||||
<span id="last-updated" class="muted" aria-live="polite">Loading…</span>
|
||||
<button id="refresh-btn" class="btn" type="button">Refresh</button>
|
||||
<a id="reports-link" class="btn" href="/report.html">Reports</a>
|
||||
<a id="admin-link" class="btn" href="/admin.html" hidden>Admin</a>
|
||||
<!-- Filled in from /api/v1/me when signed in with Microsoft 365 -->
|
||||
<span id="signed-in" class="topbar__user" hidden>
|
||||
<span id="signed-in-email" class="muted"></span>
|
||||
<a id="sign-out" class="btn" href="#">Sign out</a>
|
||||
</span>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main class="container">
|
||||
<div id="error-banner" class="banner banner--error" role="alert" hidden></div>
|
||||
|
||||
<!-- Summary cards double as status filters -->
|
||||
<section class="summary" aria-label="Status summary and filters">
|
||||
<button type="button" class="summary-card" data-filter="all" aria-pressed="true">
|
||||
<span class="summary-card__count" data-count="all">–</span>
|
||||
<span class="summary-card__label">All jobs</span>
|
||||
</button>
|
||||
<button type="button" class="summary-card summary-card--failed" data-filter="failed" aria-pressed="false">
|
||||
<span class="summary-card__count" data-count="failed">–</span>
|
||||
<span class="summary-card__label">Failed</span>
|
||||
</button>
|
||||
<button type="button" class="summary-card summary-card--warning" data-filter="warning" aria-pressed="false">
|
||||
<span class="summary-card__count" data-count="warning">–</span>
|
||||
<span class="summary-card__label">Warning</span>
|
||||
</button>
|
||||
<button type="button" class="summary-card summary-card--running" data-filter="running" aria-pressed="false">
|
||||
<span class="summary-card__count" data-count="running">–</span>
|
||||
<span class="summary-card__label">Running</span>
|
||||
</button>
|
||||
<button type="button" class="summary-card summary-card--success" data-filter="success" aria-pressed="false">
|
||||
<span class="summary-card__count" data-count="success">–</span>
|
||||
<span class="summary-card__label">Success</span>
|
||||
</button>
|
||||
<button type="button" class="summary-card summary-card--overdue" data-filter="overdue" aria-pressed="false">
|
||||
<span class="summary-card__count" data-count="overdue">–</span>
|
||||
<span class="summary-card__label">Overdue</span>
|
||||
</button>
|
||||
</section>
|
||||
|
||||
<div class="toolbar">
|
||||
<label for="search" class="visually-hidden">Search jobs</label>
|
||||
<input id="search" class="input" type="search" placeholder="Search company or job…" autocomplete="off">
|
||||
</div>
|
||||
|
||||
<div class="table-wrap">
|
||||
<table class="jobs">
|
||||
<thead>
|
||||
<tr>
|
||||
<th scope="col" aria-sort="none"><button type="button" class="sort" data-sort="company">Company</button></th>
|
||||
<th scope="col" aria-sort="none"><button type="button" class="sort" data-sort="name">Backup Job</button></th>
|
||||
<th scope="col" aria-sort="ascending"><button type="button" class="sort" data-sort="status">Status</button></th>
|
||||
<th scope="col" aria-sort="none"><button type="button" class="sort" data-sort="lastRun">Last Run</button></th>
|
||||
<th scope="col" aria-sort="none"><button type="button" class="sort" data-sort="nextRun">Next Scheduled</button></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="jobs-body"></tbody>
|
||||
</table>
|
||||
<p id="empty-state" class="empty" hidden>No backup jobs match the current filter.</p>
|
||||
</div>
|
||||
</main>
|
||||
|
||||
<!-- Native <dialog>: built-in focus trap, Esc-to-close, and top-layer rendering -->
|
||||
<dialog id="detail-dialog" class="modal" aria-labelledby="detail-title">
|
||||
<div class="modal__header">
|
||||
<div>
|
||||
<h2 id="detail-title" class="modal__title"></h2>
|
||||
<p id="detail-subtitle" class="muted"></p>
|
||||
</div>
|
||||
<button type="button" class="btn btn--icon" data-close aria-label="Close">✕</button>
|
||||
</div>
|
||||
<div id="detail-body" class="modal__body"></div>
|
||||
</dialog>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,228 @@
|
||||
// Admin page: create, list and revoke the API keys collectors use.
|
||||
import { getApi } from './api.js';
|
||||
import { el } from './dom.js';
|
||||
import { parseDate, formatDateTime, formatRelative } from './format.js';
|
||||
import { showSignedInUser } from './signed-in.js';
|
||||
|
||||
const api = await getApi();
|
||||
|
||||
// A collector runs hourly; an active key quiet for longer than this probably has a problem
|
||||
const STALE_HOURS = 24;
|
||||
|
||||
const $ = (sel) => document.querySelector(sel);
|
||||
const dom = {
|
||||
backLink: $('#back-link'),
|
||||
signedIn: $('#signed-in'),
|
||||
signedInEmail: $('#signed-in-email'),
|
||||
signOut: $('#sign-out'),
|
||||
errorBanner: $('#error-banner'),
|
||||
notAdmin: $('#not-admin'),
|
||||
content: $('#admin-content'),
|
||||
form: $('#new-key-form'),
|
||||
name: $('#key-name'),
|
||||
createBtn: $('#create-btn'),
|
||||
createError: $('#create-error'),
|
||||
result: $('#new-key-result'),
|
||||
resultTitle: $('#new-key-result-title'),
|
||||
resultName: $('#new-key-name'),
|
||||
secret: $('#new-key-secret'),
|
||||
copyBtn: $('#copy-btn'),
|
||||
doneBtn: $('#done-btn'),
|
||||
keysTitle: $('#keys-title'),
|
||||
tbody: $('#keys-body'),
|
||||
empty: $('#keys-empty'),
|
||||
status: $('#status'),
|
||||
dialog: $('#revoke-dialog'),
|
||||
revokeName: $('#revoke-name'),
|
||||
};
|
||||
|
||||
let keys = [];
|
||||
let pendingRevoke = null;
|
||||
|
||||
// ---------- Feedback ----------
|
||||
|
||||
// Screen-reader announcement. Cleared first so repeating the same message is announced again.
|
||||
// (A timer rather than requestAnimationFrame, which doesn't run while the tab is in the background.)
|
||||
function announce(message) {
|
||||
dom.status.textContent = '';
|
||||
setTimeout(() => { dom.status.textContent = message; }, 50);
|
||||
}
|
||||
|
||||
function showError(message) {
|
||||
dom.errorBanner.textContent = message;
|
||||
dom.errorBanner.hidden = false;
|
||||
}
|
||||
|
||||
// ---------- Key table ----------
|
||||
|
||||
function timeBlock(date, ...extra) {
|
||||
return el('time', { class: 'time', datetime: date.toISOString(), title: date.toString() },
|
||||
el('span', {}, formatDateTime(date)),
|
||||
el('span', { class: 'time__rel' }, formatRelative(date), ...extra));
|
||||
}
|
||||
|
||||
function statusCell(key) {
|
||||
if (key.revoked_at) {
|
||||
return el('td', {},
|
||||
el('span', { class: 'badge badge--unknown' }, 'Revoked'),
|
||||
el('span', { class: 'time__rel' }, formatDateTime(parseDate(key.revoked_at)),
|
||||
key.revoked_by ? ` by ${key.revoked_by}` : null));
|
||||
}
|
||||
return el('td', {}, el('span', { class: 'badge badge--success' }, 'Active'));
|
||||
}
|
||||
|
||||
function lastUsedCell(key) {
|
||||
const lastUsed = parseDate(key.last_used_at);
|
||||
if (!lastUsed) {
|
||||
return el('td', {}, el('span', { class: 'muted' }, key.revoked_at ? 'Never' : 'Never: not installed yet?'));
|
||||
}
|
||||
const stale = !key.revoked_at && Date.now() - lastUsed.getTime() > STALE_HOURS * 3_600_000;
|
||||
return el('td', { class: stale ? 'is-stale' : null },
|
||||
timeBlock(lastUsed, key.last_used_tenant ? ` · ${key.last_used_tenant}` : null),
|
||||
stale ? el('span', { class: 'stale-note' }, `Quiet for over ${STALE_HOURS} hours`) : null);
|
||||
}
|
||||
|
||||
function createdCell(key) {
|
||||
const created = parseDate(key.created_at);
|
||||
return el('td', {},
|
||||
created ? timeBlock(created) : el('span', { class: 'muted' }, 'Unknown'),
|
||||
key.created_by ? el('span', { class: 'time__rel' }, `by ${key.created_by}`) : null);
|
||||
}
|
||||
|
||||
function renderKeys() {
|
||||
dom.tbody.replaceChildren(...keys.map((key) => el('tr', { class: key.revoked_at ? 'is-revoked' : null },
|
||||
el('td', {}, el('code', {}, key.name)),
|
||||
statusCell(key),
|
||||
lastUsedCell(key),
|
||||
createdCell(key),
|
||||
el('td', { class: 'actions' }, key.revoked_at ? null : el('button', {
|
||||
type: 'button',
|
||||
class: 'btn btn--danger-outline',
|
||||
'aria-haspopup': 'dialog',
|
||||
'aria-label': `Revoke ${key.name}`,
|
||||
dataset: { keyId: String(key.id) },
|
||||
}, 'Revoke')))));
|
||||
dom.empty.hidden = keys.length > 0;
|
||||
}
|
||||
|
||||
async function loadKeys() {
|
||||
try {
|
||||
({ keys } = await api.listKeys());
|
||||
dom.errorBanner.hidden = true;
|
||||
renderKeys();
|
||||
} catch (err) {
|
||||
showError(`Couldn't load the keys: ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- Creating a key ----------
|
||||
|
||||
// Keep aria-invalid in step with :user-invalid (the browser doesn't sync them)
|
||||
function syncInvalid() {
|
||||
dom.name.setAttribute('aria-invalid', dom.name.matches(':user-invalid') ? 'true' : 'false');
|
||||
}
|
||||
dom.name.addEventListener('blur', syncInvalid);
|
||||
dom.name.addEventListener('invalid', syncInvalid); // a submit attempt with a bad name
|
||||
dom.name.addEventListener('input', () => { if (dom.name.hasAttribute('aria-invalid')) syncInvalid(); });
|
||||
|
||||
function showSecret(name, secret) {
|
||||
dom.resultName.textContent = name;
|
||||
dom.secret.textContent = secret;
|
||||
dom.copyBtn.textContent = 'Copy';
|
||||
dom.result.hidden = false;
|
||||
dom.resultTitle.focus();
|
||||
}
|
||||
|
||||
function hideSecret() {
|
||||
dom.result.hidden = true;
|
||||
dom.secret.textContent = '';
|
||||
}
|
||||
|
||||
// Only fires once the browser's own validation (required, pattern) has passed
|
||||
dom.form.addEventListener('submit', async (e) => {
|
||||
e.preventDefault();
|
||||
dom.createBtn.disabled = true;
|
||||
dom.createError.hidden = true;
|
||||
hideSecret();
|
||||
try {
|
||||
const { key, secret } = await api.createKey(dom.name.value.trim());
|
||||
dom.form.reset();
|
||||
dom.name.removeAttribute('aria-invalid');
|
||||
showSecret(key.name, secret);
|
||||
announce(`Key ${key.name} created. Copy it now: it won't be shown again.`);
|
||||
await loadKeys();
|
||||
} catch (err) {
|
||||
dom.createError.textContent = err.message;
|
||||
dom.createError.hidden = false;
|
||||
} finally {
|
||||
dom.createBtn.disabled = false;
|
||||
}
|
||||
});
|
||||
|
||||
dom.copyBtn.addEventListener('click', async () => {
|
||||
try {
|
||||
await navigator.clipboard.writeText(dom.secret.textContent);
|
||||
dom.copyBtn.textContent = 'Copied';
|
||||
announce('Key copied to the clipboard.');
|
||||
} catch {
|
||||
// Clipboard access can be refused (permissions, non-secure context): select it for Ctrl+C instead
|
||||
getSelection().selectAllChildren(dom.secret);
|
||||
dom.copyBtn.textContent = 'Selected: press Ctrl+C';
|
||||
announce('Copy blocked by the browser. The key is selected: press Ctrl+C to copy it.');
|
||||
}
|
||||
});
|
||||
|
||||
dom.doneBtn.addEventListener('click', () => {
|
||||
hideSecret();
|
||||
dom.name.focus();
|
||||
});
|
||||
|
||||
// ---------- Revoking a key ----------
|
||||
|
||||
dom.tbody.addEventListener('click', (e) => {
|
||||
const button = e.target.closest('button[data-key-id]');
|
||||
if (!button) return;
|
||||
pendingRevoke = keys.find((k) => String(k.id) === button.dataset.keyId) ?? null;
|
||||
if (!pendingRevoke) return;
|
||||
dom.revokeName.textContent = pendingRevoke.name;
|
||||
dom.dialog.returnValue = '';
|
||||
dom.dialog.showModal();
|
||||
});
|
||||
|
||||
// returnValue is the clicked button's value; Esc leaves it empty, which cancels
|
||||
dom.dialog.addEventListener('close', async () => {
|
||||
const key = pendingRevoke;
|
||||
pendingRevoke = null;
|
||||
if (dom.dialog.returnValue !== 'revoke' || !key) return;
|
||||
try {
|
||||
await api.revokeKey(key.id);
|
||||
announce(`Key ${key.name} revoked.`);
|
||||
await loadKeys();
|
||||
} catch (err) {
|
||||
showError(`Couldn't revoke ${key.name}: ${err.message}`);
|
||||
}
|
||||
// Its Revoke button is gone, so focus would otherwise fall back to the page
|
||||
dom.keysTitle.focus();
|
||||
});
|
||||
|
||||
// ---------- Start ----------
|
||||
|
||||
async function main() {
|
||||
dom.backLink.href = `/${location.search}`; // keeps ?mock=1
|
||||
let me;
|
||||
try {
|
||||
me = await api.getMe();
|
||||
} catch (err) {
|
||||
showError(`Couldn't check who's signed in: ${err.message}`);
|
||||
return;
|
||||
}
|
||||
showSignedInUser(me, dom);
|
||||
if (!me?.is_admin) {
|
||||
dom.notAdmin.hidden = false;
|
||||
return;
|
||||
}
|
||||
dom.content.hidden = false;
|
||||
await loadKeys();
|
||||
}
|
||||
|
||||
main();
|
||||
@@ -0,0 +1,117 @@
|
||||
import { CONFIG } from './config.js';
|
||||
|
||||
export class ApiError extends Error {
|
||||
constructor(message, status = null) {
|
||||
super(message);
|
||||
this.name = 'ApiError';
|
||||
this.status = status;
|
||||
}
|
||||
}
|
||||
|
||||
// The API's explanation from an error response: FastAPI's {"detail": "..."}, or for validation
|
||||
// errors {"detail": [{"msg": "..."}, ...]}
|
||||
async function errorDetail(res) {
|
||||
try {
|
||||
const { detail } = await res.json();
|
||||
if (typeof detail === 'string') return detail;
|
||||
if (Array.isArray(detail)) return detail.map((d) => d.msg).filter(Boolean).join('; ');
|
||||
} catch {
|
||||
// not JSON: fall back to the status code
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
async function request(path, { method = 'GET', body } = {}) {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), CONFIG.requestTimeoutMs);
|
||||
const headers = { Accept: 'application/json' };
|
||||
if (method !== 'GET') {
|
||||
// Required by the API for changes: another site's page can't add it to a request here
|
||||
headers['X-Requested-By'] = 'backup-dashboard';
|
||||
}
|
||||
if (body !== undefined) headers['Content-Type'] = 'application/json';
|
||||
try {
|
||||
const res = await fetch(`${CONFIG.apiBaseUrl}${path}`, {
|
||||
method,
|
||||
headers,
|
||||
body: body === undefined ? undefined : JSON.stringify(body),
|
||||
credentials: 'same-origin', // the sign-in session cookie
|
||||
signal: controller.signal,
|
||||
});
|
||||
// 401 means the nginx/SSO session lapsed; a reload sends the user back through sign-in
|
||||
if (res.status === 401) throw new ApiError('Your session has expired. Reload the page to sign in again', 401);
|
||||
if (!res.ok) {
|
||||
throw new ApiError((await errorDetail(res)) ?? `Server returned HTTP ${res.status}`, res.status);
|
||||
}
|
||||
return await res.json();
|
||||
} catch (err) {
|
||||
if (err.name === 'AbortError') throw new ApiError('Request timed out');
|
||||
if (err instanceof ApiError) throw err;
|
||||
throw new ApiError(`Network error: ${err.message}`);
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
const httpApi = {
|
||||
/** GET /jobs → { generated_at, jobs: Job[] } */
|
||||
async listJobs() {
|
||||
const data = await request('/jobs');
|
||||
if (!Array.isArray(data?.jobs)) throw new ApiError('Unexpected response shape from /jobs');
|
||||
return data;
|
||||
},
|
||||
/** GET /jobs/:id → JobDetail */
|
||||
getJob(id) {
|
||||
return request(`/jobs/${encodeURIComponent(id)}`);
|
||||
},
|
||||
/** GET /jobs/:id/runs?days=N → { job_id, since, runs: RunSummary[] } (newest first, no messages) */
|
||||
listRuns(id, days) {
|
||||
return request(`/jobs/${encodeURIComponent(id)}/runs?days=${encodeURIComponent(days)}`);
|
||||
},
|
||||
/** GET /runs/:id → Run (with message and details) */
|
||||
getRun(id) {
|
||||
return request(`/runs/${encodeURIComponent(id)}`);
|
||||
},
|
||||
/** GET /me → { email, name, is_admin } of the Microsoft 365 user; email/name null without sign-in */
|
||||
getMe() {
|
||||
return request('/me');
|
||||
},
|
||||
|
||||
// ----- Reports -----
|
||||
/** GET /tenants → { tenants: [{ name, jobs }] } */
|
||||
listTenants() {
|
||||
return request('/tenants');
|
||||
},
|
||||
/** GET /reports/tenant → { tenant, days, since, until, generated_at, jobs: [job + runs] } */
|
||||
tenantReport(tenant, days) {
|
||||
return request(`/reports/tenant?tenant=${encodeURIComponent(tenant)}&days=${encodeURIComponent(days)}`);
|
||||
},
|
||||
/** Link to the CSV of every run in the period, with times in time zone `tz` */
|
||||
async tenantReportCsvUrl(tenant, days, tz) {
|
||||
const params = new URLSearchParams({ tenant, days: String(days), tz });
|
||||
return `${CONFIG.apiBaseUrl}/reports/tenant.csv?${params}`;
|
||||
},
|
||||
|
||||
// ----- Admin (403 for non-admins) -----
|
||||
/** GET /admin/keys → { keys: ApiKey[] } (active first) */
|
||||
listKeys() {
|
||||
return request('/admin/keys');
|
||||
},
|
||||
/** POST /admin/keys → { key: ApiKey, secret } — the secret is only ever returned here */
|
||||
createKey(name) {
|
||||
return request('/admin/keys', { method: 'POST', body: { name } });
|
||||
},
|
||||
/** POST /admin/keys/:id/revoke → { key: ApiKey } */
|
||||
revokeKey(id) {
|
||||
return request(`/admin/keys/${encodeURIComponent(id)}/revoke`, { method: 'POST' });
|
||||
},
|
||||
};
|
||||
|
||||
// Mock is loaded lazily so it never ships in the production request path.
|
||||
export async function getApi() {
|
||||
if (CONFIG.useMock) {
|
||||
const { mockApi } = await import('./mock.js');
|
||||
return mockApi;
|
||||
}
|
||||
return httpApi;
|
||||
}
|
||||
@@ -0,0 +1,466 @@
|
||||
import { CONFIG } from './config.js';
|
||||
import { getApi } from './api.js';
|
||||
import { el } from './dom.js';
|
||||
import { showSignedInUser } from './signed-in.js';
|
||||
import { STATUS, normalizeStatus, statusBadge } from './status.js';
|
||||
import {
|
||||
localDayKey, dayLabelFmt, dayTickFmt, normalizeRun, buildDays, describeCounts,
|
||||
} from './history.js';
|
||||
import {
|
||||
parseDate, formatDateTime, formatTime, formatRelative, formatDuration, formatBytes,
|
||||
} from './format.js';
|
||||
|
||||
const api = await getApi();
|
||||
|
||||
// ---------- Domain ----------
|
||||
|
||||
function normalizeJob(j) {
|
||||
return {
|
||||
id: String(j.id),
|
||||
company: j.company_name ?? '',
|
||||
name: j.job_name ?? '',
|
||||
status: normalizeStatus(j.status),
|
||||
message: j.message ?? '',
|
||||
runningSince: parseDate(j.running_since),
|
||||
// When the most recent run *finished*; a job that's running keeps showing its previous finish
|
||||
lastRun: parseDate(j.last_run_at),
|
||||
nextRun: parseDate(j.next_run_at),
|
||||
};
|
||||
}
|
||||
|
||||
// A running job isn't silently dead, so it's never overdue (a long run can pass its next slot)
|
||||
const isOverdue = (job, now = Date.now()) =>
|
||||
job.status !== 'running' && !!job.nextRun && job.nextRun.getTime() + CONFIG.overdueGraceMinutes * 60_000 < now;
|
||||
|
||||
// ---------- State ----------
|
||||
const state = {
|
||||
jobs: [],
|
||||
filter: 'all',
|
||||
search: '',
|
||||
sort: { key: 'status', dir: 1 },
|
||||
lastSuccess: null,
|
||||
};
|
||||
|
||||
// ---------- DOM ----------
|
||||
const $ = (sel) => document.querySelector(sel);
|
||||
const dom = {
|
||||
tbody: $('#jobs-body'),
|
||||
empty: $('#empty-state'),
|
||||
search: $('#search'),
|
||||
refreshBtn: $('#refresh-btn'),
|
||||
signedIn: $('#signed-in'),
|
||||
signedInEmail: $('#signed-in-email'),
|
||||
signOut: $('#sign-out'),
|
||||
adminLink: $('#admin-link'),
|
||||
reportsLink: $('#reports-link'),
|
||||
lastUpdated: $('#last-updated'),
|
||||
errorBanner: $('#error-banner'),
|
||||
dialog: $('#detail-dialog'),
|
||||
detailTitle: $('#detail-title'),
|
||||
detailSubtitle: $('#detail-subtitle'),
|
||||
detailBody: $('#detail-body'),
|
||||
};
|
||||
|
||||
// ---------- Rendering ----------
|
||||
|
||||
const runningTitle = (since) => (since ? `Running since ${formatDateTime(since)}` : null);
|
||||
|
||||
function timeCell(date, { extra = null, emptyLabel = 'Never' } = {}) {
|
||||
if (!date) return el('td', {}, el('span', { class: 'muted' }, emptyLabel));
|
||||
return el('td', {},
|
||||
el('time', { class: 'time', datetime: date.toISOString(), title: date.toString() },
|
||||
el('span', {}, formatDateTime(date), extra),
|
||||
el('span', { class: 'time__rel' }, formatRelative(date))));
|
||||
}
|
||||
|
||||
function renderRow(job) {
|
||||
const overdueTag = isOverdue(job) ? el('span', { class: 'tag-overdue' }, 'Overdue') : null;
|
||||
return el('tr', {},
|
||||
el('td', {}, job.company),
|
||||
el('td', {}, job.name),
|
||||
el('td', {}, statusBadge(job.status, {
|
||||
asButton: true, opensDialog: true, dataset: { jobId: job.id }, title: runningTitle(job.runningSince),
|
||||
})),
|
||||
timeCell(job.lastRun),
|
||||
// No next run: unscheduled, or e.g. a Veeam copy job in immediate mode
|
||||
timeCell(job.nextRun, { extra: overdueTag, emptyLabel: 'Not scheduled' }));
|
||||
}
|
||||
|
||||
const collator = new Intl.Collator(undefined, { sensitivity: 'base', numeric: true });
|
||||
|
||||
function cmpDate(a, b) {
|
||||
if (!a && !b) return 0;
|
||||
if (!a) return 1;
|
||||
if (!b) return -1;
|
||||
return a - b;
|
||||
}
|
||||
|
||||
const SORTERS = {
|
||||
company: (a, b) => collator.compare(a.company, b.company),
|
||||
name: (a, b) => collator.compare(a.name, b.name),
|
||||
status: (a, b) => STATUS[a.status].rank - STATUS[b.status].rank,
|
||||
lastRun: (a, b) => cmpDate(a.lastRun, b.lastRun),
|
||||
nextRun: (a, b) => cmpDate(a.nextRun, b.nextRun),
|
||||
};
|
||||
|
||||
function visibleJobs() {
|
||||
const q = state.search.trim().toLowerCase();
|
||||
const now = Date.now();
|
||||
const { key, dir } = state.sort;
|
||||
return state.jobs
|
||||
.filter((j) => {
|
||||
if (state.filter === 'overdue') return isOverdue(j, now);
|
||||
return state.filter === 'all' || j.status === state.filter;
|
||||
})
|
||||
.filter((j) => !q || j.company.toLowerCase().includes(q) || j.name.toLowerCase().includes(q))
|
||||
// Stable tie-breakers keep row order from jumping around on each refresh
|
||||
.sort((a, b) => SORTERS[key](a, b) * dir || SORTERS.company(a, b) || SORTERS.name(a, b));
|
||||
}
|
||||
|
||||
function renderSummary() {
|
||||
const now = Date.now();
|
||||
const counts = { all: state.jobs.length, failed: 0, warning: 0, running: 0, success: 0, overdue: 0 };
|
||||
for (const j of state.jobs) {
|
||||
if (j.status in counts) counts[j.status]++;
|
||||
if (isOverdue(j, now)) counts.overdue++;
|
||||
}
|
||||
for (const [k, v] of Object.entries(counts)) {
|
||||
const node = document.querySelector(`[data-count="${k}"]`);
|
||||
if (node) node.textContent = v;
|
||||
}
|
||||
document.querySelectorAll('.summary-card').forEach((card) => {
|
||||
card.setAttribute('aria-pressed', String(card.dataset.filter === state.filter));
|
||||
});
|
||||
}
|
||||
|
||||
function renderSortHeaders() {
|
||||
document.querySelectorAll('th[aria-sort]').forEach((th) => {
|
||||
const key = th.querySelector('.sort').dataset.sort;
|
||||
th.setAttribute('aria-sort',
|
||||
key === state.sort.key ? (state.sort.dir === 1 ? 'ascending' : 'descending') : 'none');
|
||||
});
|
||||
}
|
||||
|
||||
function render() {
|
||||
const rows = visibleJobs();
|
||||
dom.tbody.replaceChildren(...rows.map(renderRow));
|
||||
dom.empty.hidden = rows.length > 0;
|
||||
renderSummary();
|
||||
renderSortHeaders();
|
||||
}
|
||||
|
||||
// ---------- Data loading ----------
|
||||
let inFlight = null;
|
||||
let pollTimer = null;
|
||||
|
||||
function refresh() {
|
||||
if (inFlight) return inFlight; // de-dupe overlapping manual + scheduled refreshes
|
||||
dom.refreshBtn.disabled = true;
|
||||
inFlight = (async () => {
|
||||
try {
|
||||
const data = await api.listJobs();
|
||||
state.jobs = data.jobs.map(normalizeJob);
|
||||
state.lastSuccess = new Date();
|
||||
dom.errorBanner.hidden = true;
|
||||
dom.lastUpdated.textContent = `Updated ${formatTime(state.lastSuccess)}`;
|
||||
} catch (err) {
|
||||
// Keep showing stale data, but make it obvious it's stale
|
||||
dom.errorBanner.textContent = state.lastSuccess
|
||||
? `Couldn't refresh (${err.message}). Showing data from ${formatTime(state.lastSuccess)}.`
|
||||
: `Couldn't load backup status (${err.message}).`;
|
||||
dom.errorBanner.hidden = false;
|
||||
dom.lastUpdated.textContent = 'Update failed';
|
||||
console.error(err);
|
||||
} finally {
|
||||
render();
|
||||
dom.refreshBtn.disabled = false;
|
||||
inFlight = null;
|
||||
}
|
||||
})();
|
||||
return inFlight;
|
||||
}
|
||||
|
||||
function schedulePoll() {
|
||||
clearTimeout(pollTimer);
|
||||
pollTimer = setTimeout(async () => {
|
||||
await refresh();
|
||||
schedulePoll();
|
||||
}, CONFIG.refreshIntervalMs);
|
||||
}
|
||||
|
||||
// Don't poll from background tabs (e.g. a wall display that's been switched away); catch up on return
|
||||
document.addEventListener('visibilitychange', () => {
|
||||
if (document.hidden) clearTimeout(pollTimer);
|
||||
else refresh().then(schedulePoll);
|
||||
});
|
||||
|
||||
// ---------- Detail modal ----------
|
||||
// Every async result checks reqId first, so a slow response for job A can never paint over job B.
|
||||
const modal = {
|
||||
reqId: 0,
|
||||
job: null,
|
||||
detail: null,
|
||||
runs: null, // null = loading, [] = none
|
||||
runsError: null,
|
||||
dayFilter: null, // local YYYY-MM-DD, or null for all days
|
||||
selectedRunId: null, // null = the job's latest finished run (from the detail call)
|
||||
runMessage: null, // { status, text, startedAt } for the selected run
|
||||
runMessageError: null,
|
||||
};
|
||||
|
||||
function detailRow(label, value) {
|
||||
if (value == null || value === '') return [];
|
||||
return [el('dt', {}, label), el('dd', {}, value)];
|
||||
}
|
||||
|
||||
function renderSummarySection() {
|
||||
const { job, detail } = modal;
|
||||
const status = detail ? normalizeStatus(detail.status) : job.status;
|
||||
const runningSince = detail ? parseDate(detail.running_since) : job.runningSince;
|
||||
const lastStatus = detail?.last_status ? normalizeStatus(detail.last_status) : null;
|
||||
|
||||
dom.detailTitle.textContent = detail?.job_name ?? job.name;
|
||||
const company = detail?.company_name ?? job.company;
|
||||
dom.detailSubtitle.replaceChildren(el('a', { href: reportUrl(company), title: `Backup report for ${company}` }, company));
|
||||
|
||||
return el('section', {},
|
||||
el('div', { class: 'detail-status' },
|
||||
statusBadge(status),
|
||||
runningSince ? el('span', { class: 'muted' }, `since ${formatDateTime(runningSince)}`) : null),
|
||||
el('dl', { class: 'detail-grid' },
|
||||
// While a job is running, its "last run" is the previous one — say how that went
|
||||
status === 'running' && lastStatus ? detailRow('Previous result', STATUS[lastStatus].label) : [],
|
||||
detailRow('Last finished', formatDateTime(detail ? parseDate(detail.last_run_at) : job.lastRun)),
|
||||
detailRow('Next scheduled', formatDateTime(detail ? parseDate(detail.next_run_at) : job.nextRun, 'Not scheduled')),
|
||||
detailRow('Duration', formatDuration(detail?.duration_seconds)),
|
||||
detailRow('Size', formatBytes(detail?.size_bytes)),
|
||||
detailRow('Source', detail?.source_system)));
|
||||
}
|
||||
|
||||
function renderHistorySection() {
|
||||
const section = el('section', { id: 'detail-history' },
|
||||
el('h3', { class: 'detail-section-title' }, `Last ${CONFIG.historyDays} days`));
|
||||
|
||||
if (modal.runsError) {
|
||||
section.append(el('p', { class: 'banner banner--error' }, `Couldn't load run history: ${modal.runsError}`));
|
||||
return section;
|
||||
}
|
||||
if (!modal.runs) {
|
||||
section.append(el('p', { class: 'muted' }, 'Loading history…'));
|
||||
return section;
|
||||
}
|
||||
|
||||
const days = buildDays(modal.runs, CONFIG.historyDays);
|
||||
const finished = modal.runs.filter((r) => r.finishedAt);
|
||||
const ok = finished.filter((r) => r.status === 'success').length;
|
||||
const totals = {};
|
||||
for (const r of modal.runs) totals[r.status] = (totals[r.status] ?? 0) + 1;
|
||||
|
||||
section.append(
|
||||
el('p', { class: 'history-stats' },
|
||||
modal.runs.length
|
||||
? `${modal.runs.length} runs · ${finished.length ? Math.round((ok / finished.length) * 100) : 0}% successful · ${describeCounts(totals)}`
|
||||
: 'No runs recorded in this period.'),
|
||||
el('div', { class: 'day-strip', role: 'group', 'aria-label': 'Daily results, oldest to newest. Select a day to filter runs.' },
|
||||
days.map((d) => {
|
||||
const label = `${dayLabelFmt.format(d.date)}: ${d.total ? describeCounts(d.counts) : 'no runs'}`;
|
||||
return el('button', {
|
||||
type: 'button',
|
||||
class: `day day--${d.status}`,
|
||||
title: label,
|
||||
'aria-label': label,
|
||||
'aria-pressed': String(modal.dayFilter === d.key),
|
||||
disabled: !d.total,
|
||||
dataset: { day: d.key },
|
||||
});
|
||||
})),
|
||||
el('div', { class: 'day-strip__ticks muted', 'aria-hidden': 'true' },
|
||||
el('span', {}, dayTickFmt.format(days[0].date)),
|
||||
el('span', {}, 'Today')),
|
||||
);
|
||||
|
||||
const rows = modal.dayFilter
|
||||
? modal.runs.filter((r) => localDayKey(r.startedAt) === modal.dayFilter)
|
||||
: modal.runs;
|
||||
if (!rows.length) return section;
|
||||
|
||||
if (modal.dayFilter) {
|
||||
const date = days.find((d) => d.key === modal.dayFilter)?.date;
|
||||
section.append(el('p', { class: 'muted history-filter' },
|
||||
`Showing ${dayLabelFmt.format(date)} · `,
|
||||
el('button', { type: 'button', class: 'link', dataset: { day: modal.dayFilter } }, 'Show all days')));
|
||||
}
|
||||
|
||||
section.append(el('div', { class: 'runs-wrap' },
|
||||
el('table', { class: 'runs' },
|
||||
el('thead', {}, el('tr', {},
|
||||
['Status', 'Started', 'Finished', 'Duration', 'Size'].map((h) => el('th', { scope: 'col' }, h)))),
|
||||
el('tbody', {}, rows.map((r) => el('tr', { class: r.id === modal.selectedRunId ? 'is-selected' : null },
|
||||
el('td', {}, statusBadge(r.status, { asButton: true, dataset: { runId: r.id } })),
|
||||
el('td', {}, formatDateTime(r.startedAt)),
|
||||
el('td', {}, r.finishedAt ? formatDateTime(r.finishedAt) : el('span', { class: 'muted' }, 'In progress')),
|
||||
el('td', {}, formatDuration(r.duration) ?? '—'),
|
||||
el('td', {}, formatBytes(r.size) ?? '—')))))));
|
||||
return section;
|
||||
}
|
||||
|
||||
function renderMessageSection() {
|
||||
const { detail, job } = modal;
|
||||
let status;
|
||||
let text;
|
||||
let heading = 'Message';
|
||||
if (modal.selectedRunId) {
|
||||
const run = modal.runs?.find((r) => r.id === modal.selectedRunId);
|
||||
heading = `Message · run started ${formatDateTime(run?.startedAt)}`;
|
||||
status = run?.status ?? 'unknown';
|
||||
text = modal.runMessageError
|
||||
? `Couldn't load this run: ${modal.runMessageError}`
|
||||
: modal.runMessage?.text ?? 'Loading…';
|
||||
} else {
|
||||
status = detail?.last_status ? normalizeStatus(detail.last_status) : job.status;
|
||||
text = detail?.message ?? job.message;
|
||||
}
|
||||
|
||||
const extras = !modal.selectedRunId && detail?.details && typeof detail.details === 'object'
|
||||
? Object.entries(detail.details).flatMap(([k, v]) =>
|
||||
detailRow(k.replace(/_/g, ' '), typeof v === 'object' ? JSON.stringify(v) : String(v)))
|
||||
: modal.runMessage?.extras ?? [];
|
||||
|
||||
return el('section', { id: 'detail-message' },
|
||||
el('h3', { class: 'detail-section-title' }, heading),
|
||||
el('pre', { class: `message message--${status}` }, text || 'No message reported.'),
|
||||
extras.length ? el('h3', { class: 'detail-section-title' }, 'Additional details') : null,
|
||||
extras.length ? el('dl', { class: 'detail-grid' }, extras) : null);
|
||||
}
|
||||
|
||||
function renderModal({ loading = false, error = null } = {}) {
|
||||
dom.detailBody.replaceChildren(...[
|
||||
renderSummarySection(),
|
||||
loading ? el('p', { class: 'muted' }, 'Loading full details…') : null,
|
||||
error ? el('p', { class: 'banner banner--error' }, `Couldn't load full details: ${error}`) : null,
|
||||
renderMessageSection(),
|
||||
renderHistorySection(),
|
||||
].filter(Boolean));
|
||||
}
|
||||
|
||||
// Re-render just the history section, keeping the runs table's scroll position
|
||||
function renderHistory() {
|
||||
const wrap = dom.detailBody.querySelector('.runs-wrap');
|
||||
const scrollTop = wrap?.scrollTop ?? 0;
|
||||
dom.detailBody.querySelector('#detail-history')?.replaceWith(renderHistorySection());
|
||||
const newWrap = dom.detailBody.querySelector('.runs-wrap');
|
||||
if (newWrap) newWrap.scrollTop = scrollTop;
|
||||
}
|
||||
|
||||
function renderMessage() {
|
||||
dom.detailBody.querySelector('#detail-message')?.replaceWith(renderMessageSection());
|
||||
}
|
||||
|
||||
async function openDetail(jobId) {
|
||||
const job = state.jobs.find((j) => j.id === jobId);
|
||||
if (!job) return;
|
||||
const reqId = ++modal.reqId;
|
||||
Object.assign(modal, {
|
||||
job, detail: null, runs: null, runsError: null, dayFilter: null,
|
||||
selectedRunId: null, runMessage: null, runMessageError: null,
|
||||
});
|
||||
renderModal({ loading: true }); // show what we have immediately
|
||||
if (!dom.dialog.open) dom.dialog.showModal();
|
||||
|
||||
const current = () => reqId === modal.reqId && dom.dialog.open;
|
||||
const detailP = api.getJob(jobId).then(
|
||||
(detail) => { if (current()) { modal.detail = detail; renderModal(); } },
|
||||
(err) => { if (current()) renderModal({ error: err.message }); },
|
||||
);
|
||||
const runsP = api.listRuns(jobId, CONFIG.historyDays).then(
|
||||
(data) => {
|
||||
if (!current()) return;
|
||||
modal.runs = (data.runs ?? []).map(normalizeRun).filter((r) => r.startedAt);
|
||||
renderHistory();
|
||||
},
|
||||
(err) => { if (current()) { modal.runsError = err.message; renderHistory(); } },
|
||||
);
|
||||
await Promise.all([detailP, runsP]);
|
||||
}
|
||||
|
||||
async function selectRun(runId) {
|
||||
const reqId = modal.reqId;
|
||||
modal.selectedRunId = runId;
|
||||
modal.runMessage = null;
|
||||
modal.runMessageError = null;
|
||||
renderHistory();
|
||||
renderMessage();
|
||||
try {
|
||||
const run = await api.getRun(runId);
|
||||
if (reqId !== modal.reqId || modal.selectedRunId !== runId) return;
|
||||
modal.runMessage = {
|
||||
text: run.message,
|
||||
extras: Object.entries(run.details ?? {}).flatMap(([k, v]) =>
|
||||
detailRow(k.replace(/_/g, ' '), typeof v === 'object' ? JSON.stringify(v) : String(v))),
|
||||
};
|
||||
} catch (err) {
|
||||
if (reqId !== modal.reqId || modal.selectedRunId !== runId) return;
|
||||
modal.runMessageError = err.message;
|
||||
}
|
||||
renderMessage();
|
||||
dom.detailBody.querySelector('#detail-message')?.scrollIntoView({ block: 'nearest', behavior: 'smooth' });
|
||||
}
|
||||
|
||||
// ---------- Events ----------
|
||||
dom.tbody.addEventListener('click', (e) => {
|
||||
const btn = e.target.closest('button[data-job-id]');
|
||||
if (btn) openDetail(btn.dataset.jobId);
|
||||
});
|
||||
|
||||
dom.dialog.addEventListener('click', (e) => {
|
||||
// Click on the backdrop (the dialog element itself, outside its content) or the close button
|
||||
if (e.target === dom.dialog || e.target.closest('[data-close]')) dom.dialog.close();
|
||||
});
|
||||
|
||||
dom.detailBody.addEventListener('click', (e) => {
|
||||
const day = e.target.closest('button[data-day]');
|
||||
if (day) {
|
||||
modal.dayFilter = modal.dayFilter === day.dataset.day ? null : day.dataset.day;
|
||||
renderHistory();
|
||||
return;
|
||||
}
|
||||
const run = e.target.closest('button[data-run-id]');
|
||||
if (run) selectRun(run.dataset.runId);
|
||||
});
|
||||
|
||||
document.querySelector('.summary').addEventListener('click', (e) => {
|
||||
const card = e.target.closest('.summary-card');
|
||||
if (!card) return;
|
||||
state.filter = card.dataset.filter;
|
||||
render();
|
||||
});
|
||||
|
||||
document.querySelector('.jobs thead').addEventListener('click', (e) => {
|
||||
const btn = e.target.closest('.sort');
|
||||
if (!btn) return;
|
||||
const key = btn.dataset.sort;
|
||||
state.sort = state.sort.key === key ? { key, dir: -state.sort.dir } : { key, dir: 1 };
|
||||
render();
|
||||
});
|
||||
|
||||
dom.search.addEventListener('input', () => {
|
||||
state.search = dom.search.value;
|
||||
render();
|
||||
});
|
||||
|
||||
dom.refreshBtn.addEventListener('click', () => refresh().then(schedulePoll));
|
||||
|
||||
// A company's report page (keeps ?mock=1)
|
||||
function reportUrl(company) {
|
||||
const params = new URLSearchParams({ tenant: company, days: '30' });
|
||||
if (CONFIG.useMock) params.set('mock', '1');
|
||||
return `/report.html?${params}`;
|
||||
}
|
||||
|
||||
// ---------- Boot ----------
|
||||
dom.reportsLink.href = `/report.html${CONFIG.useMock ? '?mock=1' : ''}`;
|
||||
api.getMe().then(
|
||||
(me) => showSignedInUser(me, dom),
|
||||
(err) => console.error(err), // not worth a banner: the jobs request will surface real problems
|
||||
);
|
||||
refresh().then(schedulePoll);
|
||||
@@ -0,0 +1,19 @@
|
||||
// Central runtime config. Kept as a static file so ops can change it without a build.
|
||||
const params = new URLSearchParams(window.location.search);
|
||||
|
||||
export const CONFIG = Object.freeze({
|
||||
apiBaseUrl: '/api/v1',
|
||||
refreshIntervalMs: 60_000,
|
||||
requestTimeoutMs: 15_000,
|
||||
// A job whose next scheduled run is this far in the past is flagged "Overdue".
|
||||
// Catches the silent failure mode: a job stops reporting and stays green forever.
|
||||
overdueGraceMinutes: 30,
|
||||
// Run history shown in the job modal (the API caps this at 30)
|
||||
historyDays: 30,
|
||||
// ?mock=1 → use in-browser sample data, no backend needed
|
||||
useMock: params.has('mock'),
|
||||
// oauth2-proxy clears the dashboard session, then hands over to Microsoft's sign-out page.
|
||||
// Signing out of the dashboard alone would be undone at once: the browser's Microsoft session
|
||||
// would sign it straight back in.
|
||||
signOutUrl: '/oauth2/sign_out?rd=' + encodeURIComponent('https://login.microsoftonline.com/common/oauth2/v2.0/logout'),
|
||||
});
|
||||
@@ -0,0 +1,12 @@
|
||||
// Tiny element builder. All text goes through append() → text nodes, so API data can never inject HTML.
|
||||
export function el(tag, props = {}, ...children) {
|
||||
const node = document.createElement(tag);
|
||||
for (const [k, v] of Object.entries(props)) {
|
||||
if (v == null || v === false) continue;
|
||||
if (k === 'class') node.className = v;
|
||||
else if (k === 'dataset') Object.assign(node.dataset, v);
|
||||
else node.setAttribute(k, v === true ? '' : v);
|
||||
}
|
||||
node.append(...children.flat().filter((c) => c != null && c !== false));
|
||||
return node;
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
const dateTimeFmt = new Intl.DateTimeFormat(undefined, { dateStyle: 'medium', timeStyle: 'short' });
|
||||
const timeFmt = new Intl.DateTimeFormat(undefined, { timeStyle: 'medium' });
|
||||
const relFmt = new Intl.RelativeTimeFormat(undefined, { numeric: 'auto' });
|
||||
|
||||
const REL_UNITS = [
|
||||
['year', 31_536_000], ['month', 2_592_000], ['week', 604_800],
|
||||
['day', 86_400], ['hour', 3_600], ['minute', 60],
|
||||
];
|
||||
|
||||
export function parseDate(value) {
|
||||
if (!value) return null;
|
||||
const d = new Date(value);
|
||||
return Number.isNaN(d.getTime()) ? null : d;
|
||||
}
|
||||
|
||||
export const formatDateTime = (d, emptyLabel = '—') => (d ? dateTimeFmt.format(d) : emptyLabel);
|
||||
export const formatTime = (d) => timeFmt.format(d);
|
||||
|
||||
export function formatRelative(d, now = Date.now()) {
|
||||
if (!d) return '';
|
||||
const diffSec = Math.round((d.getTime() - now) / 1000);
|
||||
for (const [unit, secs] of REL_UNITS) {
|
||||
if (Math.abs(diffSec) >= secs) return relFmt.format(Math.round(diffSec / secs), unit);
|
||||
}
|
||||
return relFmt.format(diffSec, 'second');
|
||||
}
|
||||
|
||||
export function formatDuration(totalSec) {
|
||||
if (totalSec == null || Number.isNaN(Number(totalSec))) return null;
|
||||
const s = Math.max(0, Math.round(totalSec));
|
||||
const h = Math.floor(s / 3600);
|
||||
const m = Math.floor((s % 3600) / 60);
|
||||
const sec = s % 60;
|
||||
return [h && `${h}h`, (h || m) && `${m}m`, `${sec}s`].filter(Boolean).join(' ');
|
||||
}
|
||||
|
||||
export function formatBytes(bytes) {
|
||||
if (bytes == null || Number.isNaN(Number(bytes))) return null;
|
||||
const units = ['B', 'KB', 'MB', 'GB', 'TB', 'PB'];
|
||||
let n = Number(bytes);
|
||||
let i = 0;
|
||||
while (n >= 1024 && i < units.length - 1) { n /= 1024; i++; }
|
||||
return `${n.toFixed(i === 0 ? 0 : 1)} ${units[i]}`;
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
// Run history: runs grouped into days (local time), as shown in the dashboard's job detail and in reports.
|
||||
import { parseDate } from './format.js';
|
||||
import { STATUS, normalizeStatus } from './status.js';
|
||||
|
||||
// Colour of a day cell: the worst thing that happened that day
|
||||
export const DAY_SEVERITY = ['failed', 'warning', 'unknown', 'running', 'success'];
|
||||
|
||||
const pad2 = (n) => String(n).padStart(2, '0');
|
||||
export const localDayKey = (d) => `${d.getFullYear()}-${pad2(d.getMonth() + 1)}-${pad2(d.getDate())}`;
|
||||
export const dayLabelFmt = new Intl.DateTimeFormat(undefined, { weekday: 'short', month: 'short', day: 'numeric' });
|
||||
export const dayTickFmt = new Intl.DateTimeFormat(undefined, { month: 'short', day: 'numeric' });
|
||||
|
||||
export function normalizeRun(r) {
|
||||
return {
|
||||
id: String(r.id),
|
||||
status: normalizeStatus(r.status),
|
||||
startedAt: parseDate(r.started_at),
|
||||
finishedAt: parseDate(r.finished_at),
|
||||
duration: r.duration_seconds,
|
||||
size: r.size_bytes,
|
||||
};
|
||||
}
|
||||
|
||||
export function buildDays(runs, count) {
|
||||
const today = new Date();
|
||||
today.setHours(0, 0, 0, 0);
|
||||
const days = [];
|
||||
for (let i = count - 1; i >= 0; i--) {
|
||||
const date = new Date(today);
|
||||
date.setDate(date.getDate() - i);
|
||||
days.push({ key: localDayKey(date), date, counts: {}, total: 0 });
|
||||
}
|
||||
const byKey = new Map(days.map((d) => [d.key, d]));
|
||||
for (const run of runs) {
|
||||
const day = byKey.get(localDayKey(run.startedAt));
|
||||
if (!day) continue;
|
||||
day.counts[run.status] = (day.counts[run.status] ?? 0) + 1;
|
||||
day.total++;
|
||||
}
|
||||
for (const day of days) day.status = day.total ? DAY_SEVERITY.find((s) => day.counts[s]) : 'none';
|
||||
return days;
|
||||
}
|
||||
|
||||
export function describeCounts(counts) {
|
||||
return DAY_SEVERITY.filter((s) => counts[s])
|
||||
.map((s) => `${counts[s]} ${STATUS[s].label.toLowerCase()}`)
|
||||
.join(', ');
|
||||
}
|
||||
@@ -0,0 +1,209 @@
|
||||
// Sample data matching the API contract. Timestamps are relative to page load so it always looks "live".
|
||||
const now = Date.now();
|
||||
const MIN = 60_000;
|
||||
const HR = 60 * MIN;
|
||||
const DAY = 24 * HR;
|
||||
const iso = (offsetMs) => new Date(now + offsetMs).toISOString();
|
||||
|
||||
// every_h drives the generated run history; it isn't part of the API.
|
||||
const JOBS = [
|
||||
{
|
||||
id: 1, company_name: 'Acme Corp', job_name: 'File Server – Nightly', status: 'success', every_h: 24,
|
||||
last_run_at: iso(-6 * HR), next_run_at: iso(18 * HR),
|
||||
message: 'Job completed successfully. 1,284 files processed.',
|
||||
source_system: 'Veeam B&R', duration_seconds: 2840, size_bytes: 128_849_018_880,
|
||||
},
|
||||
{
|
||||
id: 2, company_name: 'Acme Corp', job_name: 'SQL01 – Transaction Logs', status: 'warning', every_h: 1,
|
||||
last_run_at: iso(-25 * MIN), next_run_at: iso(35 * MIN),
|
||||
message: 'Processing SQL01 Warning:\nFailed to truncate transaction logs for database "ERP_PROD".\nLog backup completed but truncation was skipped.',
|
||||
source_system: 'Veeam B&R', duration_seconds: 312, size_bytes: 2_147_483_648,
|
||||
details: { retry_count: 0, repository: 'REPO-ONSITE-01' },
|
||||
},
|
||||
{
|
||||
// Running now; "last run" is still the previous completion
|
||||
id: 7, company_name: 'Acme Corp', job_name: 'Hyper-V Hosts – Daily', status: 'running', every_h: 24,
|
||||
running_since: iso(-12 * MIN), last_status: 'success',
|
||||
last_run_at: iso(-DAY + 40 * MIN), next_run_at: iso(DAY - 12 * MIN),
|
||||
message: 'Completed successfully. 6 VMs processed.',
|
||||
source_system: 'Veeam B&R', duration_seconds: 3120, size_bytes: 412_316_860_416,
|
||||
},
|
||||
{
|
||||
id: 3, company_name: 'Bluewater Dental', job_name: 'Workstations – Daily', status: 'failed', every_h: 24,
|
||||
last_run_at: iso(-3 * HR), next_run_at: iso(21 * HR),
|
||||
message: 'Error: Unable to connect to backup repository \\\\NAS02\\Backups.\nThe network path was not found. (0x80070035)\nJob terminated after 3 retries.',
|
||||
source_system: 'Datto', duration_seconds: 95,
|
||||
details: { retry_count: 3, target: '\\\\NAS02\\Backups' },
|
||||
},
|
||||
{
|
||||
id: 4, company_name: 'Bluewater Dental', job_name: 'Imaging Server – Hourly', status: 'success', every_h: 1,
|
||||
last_run_at: iso(-50 * MIN), next_run_at: iso(10 * MIN),
|
||||
message: 'Completed.', source_system: 'Datto', duration_seconds: 420, size_bytes: 8_589_934_592,
|
||||
},
|
||||
{
|
||||
// Green status but next run is in the past → should show as Overdue
|
||||
id: 5, company_name: 'Northfield Logistics', job_name: 'Exchange – Nightly', status: 'success', every_h: 24,
|
||||
last_run_at: iso(-30 * HR), next_run_at: iso(-6 * HR),
|
||||
message: 'Completed successfully.', source_system: 'Acronis', duration_seconds: 5400,
|
||||
},
|
||||
{
|
||||
// Never run / unrecognised status → rendered as Unknown
|
||||
id: 6, company_name: 'Northfield Logistics', job_name: 'New NAS – Weekly', status: null,
|
||||
last_run_at: null, next_run_at: iso(2 * DAY),
|
||||
message: '', source_system: 'Synology Hyper Backup',
|
||||
},
|
||||
];
|
||||
|
||||
const MESSAGES = {
|
||||
success: 'Job completed successfully.',
|
||||
warning: 'Completed with warnings: 1 file skipped (file in use).',
|
||||
failed: 'Error: Target repository unreachable. Job terminated.',
|
||||
};
|
||||
|
||||
// Deterministic pseudo-random so history doesn't reshuffle on every modal open
|
||||
function rng(seed) {
|
||||
let x = seed;
|
||||
return () => ((x = (x * 1_103_515_245 + 12_345) % 2_147_483_648) / 2_147_483_648);
|
||||
}
|
||||
|
||||
const RUNS = new Map(); // job id → runs, newest first
|
||||
for (const job of JOBS) {
|
||||
const runs = [];
|
||||
if (job.running_since) {
|
||||
runs.push({ id: `${job.id}-r`, status: 'running', started_at: job.running_since, finished_at: null, message: 'In progress…' });
|
||||
}
|
||||
if (job.last_run_at && job.every_h) {
|
||||
const rand = rng(job.id * 7919);
|
||||
const dur = (job.duration_seconds ?? 600) * 1000;
|
||||
let end = Date.parse(job.last_run_at);
|
||||
for (let i = 0; end - dur > now - 30 * DAY; i++, end -= job.every_h * HR) {
|
||||
const r = rand();
|
||||
const status = i === 0 ? (job.last_status ?? job.status) : r < 0.03 ? 'failed' : r < 0.1 ? 'warning' : 'success';
|
||||
const thisDur = dur * (0.8 + rand() * 0.4);
|
||||
runs.push({
|
||||
id: `${job.id}-${i}`,
|
||||
status,
|
||||
started_at: new Date(end - thisDur).toISOString(),
|
||||
finished_at: new Date(end).toISOString(),
|
||||
duration_seconds: Math.round(thisDur / 1000),
|
||||
size_bytes: status === 'failed' ? null : job.size_bytes,
|
||||
message: i === 0 ? job.message : MESSAGES[status],
|
||||
details: i === 0 ? job.details ?? {} : {},
|
||||
});
|
||||
}
|
||||
}
|
||||
RUNS.set(String(job.id), runs);
|
||||
}
|
||||
|
||||
const delay = (ms) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
// Sample API keys for the admin page: one healthy, one gone quiet, one never used, one revoked
|
||||
const KEYS = [
|
||||
{ id: 1, name: 'veeam-acme', created_at: iso(-40 * DAY), created_by: '[email protected]',
|
||||
last_used_at: iso(-12 * MIN), last_used_tenant: 'Acme Corp', revoked_at: null, revoked_by: null },
|
||||
{ id: 2, name: 'mars-bluewater-fs01', created_at: iso(-20 * DAY), created_by: 'manage.py (backupdash)',
|
||||
last_used_at: iso(-3 * DAY), last_used_tenant: 'Bluewater Dental', revoked_at: null, revoked_by: null },
|
||||
{ id: 3, name: 'wsb-northfield-dc01', created_at: iso(-2 * HR), created_by: '[email protected]',
|
||||
last_used_at: null, last_used_tenant: null, revoked_at: null, revoked_by: null },
|
||||
{ id: 4, name: 'veeam-old-server', created_at: iso(-90 * DAY), created_by: 'manage.py (backupdash)',
|
||||
last_used_at: iso(-31 * DAY), last_used_tenant: 'Acme Corp', revoked_at: iso(-30 * DAY), revoked_by: '[email protected]' },
|
||||
];
|
||||
const summary = ({ details, every_h, last_status, duration_seconds, size_bytes, ...rest }) => rest;
|
||||
|
||||
export const mockApi = {
|
||||
async listJobs() {
|
||||
await delay(250);
|
||||
return { generated_at: new Date().toISOString(), jobs: JOBS.map(summary) };
|
||||
},
|
||||
async getJob(id) {
|
||||
await delay(200);
|
||||
const job = JOBS.find((j) => String(j.id) === String(id));
|
||||
if (!job) throw new Error('Job not found');
|
||||
const { every_h, ...rest } = job;
|
||||
return { last_status: job.status === 'running' ? null : job.status, ...rest, details: job.details ?? {} };
|
||||
},
|
||||
async listRuns(id, days) {
|
||||
await delay(300);
|
||||
const since = now - days * DAY;
|
||||
const runs = (RUNS.get(String(id)) ?? [])
|
||||
.filter((r) => Date.parse(r.started_at) >= since)
|
||||
.map(({ message, details, ...r }) => r);
|
||||
return { job_id: id, since: new Date(since).toISOString(), runs };
|
||||
},
|
||||
async getRun(id) {
|
||||
await delay(150);
|
||||
for (const runs of RUNS.values()) {
|
||||
const run = runs.find((r) => r.id === id);
|
||||
if (run) return { details: {}, ...run };
|
||||
}
|
||||
throw new Error('Run not found');
|
||||
},
|
||||
async getMe() {
|
||||
// No sign-in in mock mode, but show the admin pages so they can be tried out
|
||||
return { email: null, name: null, is_admin: true };
|
||||
},
|
||||
|
||||
async listTenants() {
|
||||
await delay(150);
|
||||
const counts = new Map();
|
||||
for (const j of JOBS) counts.set(j.company_name, (counts.get(j.company_name) ?? 0) + 1);
|
||||
return { tenants: [...counts].map(([name, jobs]) => ({ name, jobs })).sort((a, b) => a.name.localeCompare(b.name)) };
|
||||
},
|
||||
async tenantReport(tenant, days) {
|
||||
await delay(250);
|
||||
const jobs = JOBS.filter((j) => j.company_name.toLowerCase() === tenant.toLowerCase());
|
||||
if (!jobs.length) throw new Error(`No company named '${tenant}'`);
|
||||
const since = now - days * DAY;
|
||||
return {
|
||||
tenant: jobs[0].company_name, days,
|
||||
since: new Date(since).toISOString(), until: new Date(now).toISOString(), generated_at: new Date(now).toISOString(),
|
||||
jobs: jobs.map(({ details, every_h, ...job }) => ({
|
||||
...job,
|
||||
// Earliest run on record, as the API reports it
|
||||
tracked_since: (RUNS.get(String(job.id)) ?? []).map((r) => r.started_at).sort()[0] ?? new Date(now).toISOString(),
|
||||
runs: (RUNS.get(String(job.id)) ?? [])
|
||||
.filter((r) => Date.parse(r.started_at) >= since)
|
||||
.map(({ id, details: _d, message, ...r }) => ({ ...r, message: ['warning', 'failed'].includes(r.status) ? message : null }))
|
||||
.reverse(), // oldest first, like the API
|
||||
})),
|
||||
};
|
||||
},
|
||||
// No server in mock mode: build the CSV here and link to it as a blob
|
||||
async tenantReportCsvUrl(tenant, days, tz) {
|
||||
const report = await this.tenantReport(tenant, days);
|
||||
const quoteCell = (v) => `"${String(v ?? '').replaceAll('"', '""')}"`;
|
||||
// Times stay in UTC here (the real API converts them to `tz`)
|
||||
const rows = [['Company', 'Job', 'Source', 'Started (UTC)', 'Finished (UTC)', 'Status', 'Message']];
|
||||
for (const job of report.jobs) {
|
||||
for (const run of job.runs) {
|
||||
rows.push([report.tenant, job.job_name, job.source_system, run.started_at, run.finished_at ?? '', run.status, run.message ?? '']);
|
||||
}
|
||||
}
|
||||
const csv = rows.map((r) => r.map(quoteCell).join(',')).join('\r\n');
|
||||
return URL.createObjectURL(new Blob(['\ufeff' + csv], { type: 'text/csv' }));
|
||||
},
|
||||
|
||||
async listKeys() {
|
||||
await delay(200);
|
||||
return { keys: [...KEYS].sort((a, b) => (!!a.revoked_at - !!b.revoked_at) || a.name.localeCompare(b.name)) };
|
||||
},
|
||||
async createKey(name) {
|
||||
await delay(300);
|
||||
if (KEYS.some((k) => k.name === name)) {
|
||||
throw new Error(`A key named '${name}' already exists (revoked keys keep their names)`);
|
||||
}
|
||||
const key = { id: KEYS.length + 1, name, created_at: new Date().toISOString(), created_by: '[email protected]',
|
||||
last_used_at: null, last_used_tenant: null, revoked_at: null, revoked_by: null };
|
||||
KEYS.push(key);
|
||||
const random = crypto.getRandomValues(new Uint8Array(32));
|
||||
return { key, secret: `bdk_mock_${btoa(String.fromCharCode(...random)).replace(/[+/=]/g, '')}` };
|
||||
},
|
||||
async revokeKey(id) {
|
||||
await delay(200);
|
||||
const key = KEYS.find((k) => k.id === id);
|
||||
if (!key) throw new Error('Key not found');
|
||||
key.revoked_at ??= new Date().toISOString();
|
||||
key.revoked_by ??= '[email protected]';
|
||||
return { key };
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,248 @@
|
||||
// Per-company backup report: summary, per-job results, day-by-day strips and every warning and
|
||||
// failure for a period. Printable (Print / Save as PDF) and downloadable as CSV.
|
||||
import { CONFIG } from './config.js';
|
||||
import { getApi } from './api.js';
|
||||
import { el } from './dom.js';
|
||||
import { parseDate, formatDateTime } from './format.js';
|
||||
import { buildDays, describeCounts, dayLabelFmt, normalizeRun } from './history.js';
|
||||
import { showSignedInUser } from './signed-in.js';
|
||||
import { normalizeStatus, statusBadge } from './status.js';
|
||||
|
||||
const api = await getApi();
|
||||
|
||||
// Report messages are trimmed to keep a printed report readable; the CSV has them in full
|
||||
const MAX_MESSAGE_CHARS = 400;
|
||||
const TIME_ZONE = Intl.DateTimeFormat().resolvedOptions().timeZone;
|
||||
const dateFmt = new Intl.DateTimeFormat(undefined, { day: 'numeric', month: 'short', year: 'numeric' });
|
||||
const percentFmt = new Intl.NumberFormat(undefined, { style: 'percent', maximumFractionDigits: 1 });
|
||||
|
||||
const $ = (sel) => document.querySelector(sel);
|
||||
const dom = {
|
||||
backLink: $('#back-link'),
|
||||
signedIn: $('#signed-in'),
|
||||
signedInEmail: $('#signed-in-email'),
|
||||
signOut: $('#sign-out'),
|
||||
tenant: $('#tenant'),
|
||||
days: $('#days'),
|
||||
printBtn: $('#print-btn'),
|
||||
csvLink: $('#csv-link'),
|
||||
errorBanner: $('#error-banner'),
|
||||
chooseHint: $('#choose-hint'),
|
||||
loading: $('#loading'),
|
||||
report: $('#report'),
|
||||
title: $('#report-title'),
|
||||
period: $('#report-period'),
|
||||
summary: $('#report-summary'),
|
||||
jobsBody: $('#jobs-body'),
|
||||
daily: $('#daily'),
|
||||
issuesBody: $('#issues-body'),
|
||||
issuesWrap: $('#issues-wrap'),
|
||||
issuesNone: $('#issues-none'),
|
||||
footer: $('#report-footer'),
|
||||
};
|
||||
|
||||
let requestId = 0; // a slow response for one company must not overwrite a later one
|
||||
|
||||
// ---------- Numbers ----------
|
||||
|
||||
function summarize(runs) {
|
||||
const counts = { success: 0, warning: 0, failed: 0, running: 0, unknown: 0 };
|
||||
for (const run of runs) counts[run.status]++;
|
||||
const finished = runs.length - counts.running;
|
||||
const lastFinished = runs.filter((r) => r.finishedAt).at(-1) ?? null;
|
||||
const lastSuccess = runs.filter((r) => r.status === 'success').at(-1) ?? null;
|
||||
return {
|
||||
counts,
|
||||
total: runs.length,
|
||||
finished,
|
||||
// Of finished runs only: one still in progress hasn't succeeded or failed yet
|
||||
rate: finished ? counts.success / finished : null,
|
||||
lastFinished,
|
||||
lastSuccess,
|
||||
};
|
||||
}
|
||||
|
||||
const isOverdue = (job) => {
|
||||
const next = parseDate(job.next_run_at);
|
||||
return normalizeStatus(job.status) !== 'running' && !!next
|
||||
&& next.getTime() + CONFIG.overdueGraceMinutes * 60_000 < Date.now();
|
||||
};
|
||||
|
||||
// ---------- Rendering ----------
|
||||
|
||||
function renderSummary(jobs) {
|
||||
const all = summarize(jobs.flatMap((j) => j.runs));
|
||||
const overdue = jobs.filter((j) => isOverdue(j.job)).length;
|
||||
const item = (label, value, modifier = '') => el('div', { class: `report-summary__item ${modifier}` },
|
||||
el('dt', {}, label), el('dd', {}, value));
|
||||
dom.summary.replaceChildren(
|
||||
item('Jobs', String(jobs.length)),
|
||||
item('Backup runs', String(all.total)),
|
||||
item('Success rate', all.rate === null ? '—' : percentFmt.format(all.rate)),
|
||||
item('Warnings', String(all.counts.warning), all.counts.warning ? 'report-summary__item--warning' : ''),
|
||||
item('Failures', String(all.counts.failed), all.counts.failed ? 'report-summary__item--failed' : ''),
|
||||
item('Overdue now', String(overdue), overdue ? 'report-summary__item--failed' : ''),
|
||||
);
|
||||
}
|
||||
|
||||
function renderJobs(jobs, days) {
|
||||
dom.jobsBody.replaceChildren(...jobs.map(({ job, runs, stats }) => {
|
||||
const last = stats.lastFinished;
|
||||
return el('tr', {},
|
||||
el('th', { scope: 'row' }, job.job_name),
|
||||
el('td', {}, job.source_system),
|
||||
el('td', { class: 'num' }, String(stats.total), stats.counts.running ? el('span', { class: 'time__rel' }, `${stats.counts.running} in progress`) : null),
|
||||
el('td', { class: 'num' }, String(stats.counts.success)),
|
||||
el('td', { class: 'num' }, String(stats.counts.warning)),
|
||||
el('td', { class: 'num' }, String(stats.counts.failed)),
|
||||
el('td', { class: 'num' }, stats.rate === null ? '—' : percentFmt.format(stats.rate)),
|
||||
el('td', {}, last
|
||||
? [statusBadge(last.status), el('span', { class: 'time__rel' }, formatDateTime(last.finishedAt))]
|
||||
: el('span', { class: 'muted' }, 'No finished runs')),
|
||||
el('td', {}, stats.lastSuccess
|
||||
? formatDateTime(stats.lastSuccess.finishedAt)
|
||||
: el('span', { class: 'report-alert' }, `None in the last ${days} days`)));
|
||||
}));
|
||||
}
|
||||
|
||||
function renderDaily(jobs, days) {
|
||||
dom.daily.replaceChildren(...jobs.map(({ job, runs }) => {
|
||||
const cells = buildDays(runs, days);
|
||||
// Days before the dashboard knew of the job aren't missed backups, just unmonitored ones
|
||||
const tracked = parseDate(job.tracked_since);
|
||||
if (tracked) {
|
||||
const firstDay = new Date(tracked);
|
||||
firstDay.setHours(0, 0, 0, 0);
|
||||
for (const d of cells) if (d.date < firstDay && !d.total) d.status = 'untracked';
|
||||
}
|
||||
const tally = {};
|
||||
for (const d of cells) tally[d.status] = (tally[d.status] ?? 0) + 1;
|
||||
const description = `${job.job_name}, ${days} days: ${tally.success ?? 0} successful, ${tally.warning ?? 0} with warnings, `
|
||||
+ `${tally.failed ?? 0} failed, ${tally.none ?? 0} without a backup`
|
||||
+ (tally.untracked ? `, ${tally.untracked} before monitoring began` : '');
|
||||
return el('div', { class: 'report-daily__job' },
|
||||
el('div', { class: 'report-daily__name' }, job.job_name),
|
||||
// Coloured cells are visual only; the label gives screen readers the same picture
|
||||
el('div', { class: 'day-strip report-strip', role: 'img', 'aria-label': description, style: `--days: ${days}` },
|
||||
cells.map((d) => el('span', {
|
||||
class: `day day--${d.status}`,
|
||||
title: `${dayLabelFmt.format(d.date)}: ${d.total ? describeCounts(d.counts) : d.status === 'untracked' ? 'not monitored yet' : 'no backup'}`,
|
||||
}))));
|
||||
}));
|
||||
}
|
||||
|
||||
function renderIssues(jobs) {
|
||||
const issues = jobs
|
||||
.flatMap(({ job, runs }) => runs
|
||||
.filter((r) => r.status === 'warning' || r.status === 'failed')
|
||||
.map((r) => ({ job, run: r })))
|
||||
.sort((a, b) => b.run.startedAt - a.run.startedAt); // newest first
|
||||
dom.issuesBody.replaceChildren(...issues.map(({ job, run }) => {
|
||||
const message = (run.message ?? '').trim() || 'No details reported.';
|
||||
const shown = message.length > MAX_MESSAGE_CHARS ? `${message.slice(0, MAX_MESSAGE_CHARS)}…` : message;
|
||||
return el('tr', {},
|
||||
el('td', { class: 'nowrap' }, formatDateTime(run.startedAt)),
|
||||
el('th', { scope: 'row' }, job.job_name),
|
||||
el('td', {}, statusBadge(run.status)),
|
||||
el('td', { class: 'report-message' }, shown));
|
||||
}));
|
||||
dom.issuesNone.hidden = issues.length > 0;
|
||||
dom.issuesWrap.hidden = issues.length === 0;
|
||||
}
|
||||
|
||||
function render(report) {
|
||||
const days = report.days;
|
||||
const jobs = report.jobs.map((job) => {
|
||||
const runs = job.runs.map((r) => ({ ...normalizeRun({ id: '', ...r }), message: r.message }))
|
||||
.filter((r) => r.startedAt);
|
||||
return { job, runs, stats: summarize(runs) };
|
||||
});
|
||||
const since = parseDate(report.since);
|
||||
const until = parseDate(report.until);
|
||||
|
||||
dom.title.textContent = `Backup report: ${report.tenant}`;
|
||||
dom.period.textContent = `${dateFmt.format(since)} – ${dateFmt.format(until)} (last ${days} days)`;
|
||||
document.title = `Backup report: ${report.tenant} · Backup Monitor`;
|
||||
renderSummary(jobs);
|
||||
renderJobs(jobs, days);
|
||||
renderDaily(jobs, days);
|
||||
renderIssues(jobs);
|
||||
dom.footer.textContent = `Generated ${formatDateTime(parseDate(report.generated_at))}. Times are in ${TIME_ZONE}. `
|
||||
+ 'Success rate counts finished runs only. The CSV export lists every run with its full details.';
|
||||
dom.report.hidden = false;
|
||||
}
|
||||
|
||||
// ---------- Loading ----------
|
||||
|
||||
function setActionsEnabled(enabled) {
|
||||
dom.printBtn.disabled = !enabled;
|
||||
if (!enabled) {
|
||||
dom.csvLink.removeAttribute('href');
|
||||
dom.csvLink.setAttribute('aria-disabled', 'true');
|
||||
}
|
||||
}
|
||||
|
||||
async function load() {
|
||||
const tenant = dom.tenant.value;
|
||||
const days = Number(dom.days.value);
|
||||
// Shareable, bookmarkable URL (keeps ?mock=1)
|
||||
const params = new URLSearchParams(location.search);
|
||||
if (tenant) params.set('tenant', tenant); else params.delete('tenant');
|
||||
params.set('days', String(days));
|
||||
history.replaceState(null, '', `?${params}`);
|
||||
|
||||
dom.errorBanner.hidden = true;
|
||||
setActionsEnabled(false);
|
||||
if (!tenant) {
|
||||
dom.report.hidden = true;
|
||||
dom.chooseHint.hidden = false;
|
||||
return;
|
||||
}
|
||||
dom.chooseHint.hidden = true;
|
||||
dom.loading.hidden = false;
|
||||
const id = ++requestId;
|
||||
try {
|
||||
const report = await api.tenantReport(tenant, days);
|
||||
if (id !== requestId) return;
|
||||
render(report);
|
||||
dom.csvLink.href = await api.tenantReportCsvUrl(report.tenant, days, TIME_ZONE);
|
||||
dom.csvLink.download = `backup-report-${report.tenant.toLowerCase().replace(/[^a-z0-9]+/g, '-')}.csv`;
|
||||
dom.csvLink.removeAttribute('aria-disabled');
|
||||
dom.printBtn.disabled = false;
|
||||
} catch (err) {
|
||||
if (id !== requestId) return;
|
||||
dom.report.hidden = true;
|
||||
dom.errorBanner.textContent = `Couldn't load the report: ${err.message}`;
|
||||
dom.errorBanner.hidden = false;
|
||||
} finally {
|
||||
if (id === requestId) dom.loading.hidden = true;
|
||||
}
|
||||
}
|
||||
|
||||
dom.tenant.addEventListener('change', load);
|
||||
dom.days.addEventListener('change', load);
|
||||
dom.printBtn.addEventListener('click', () => window.print());
|
||||
|
||||
async function main() {
|
||||
dom.backLink.href = `/${location.search.includes('mock') ? '?mock=1' : ''}`;
|
||||
api.getMe().then((me) => showSignedInUser(me, dom), () => {});
|
||||
|
||||
const params = new URLSearchParams(location.search);
|
||||
const requestedDays = params.get('days');
|
||||
if ([...dom.days.options].some((o) => o.value === requestedDays)) dom.days.value = requestedDays;
|
||||
try {
|
||||
const { tenants } = await api.listTenants();
|
||||
dom.tenant.append(...tenants.map((t) => el('option', { value: t.name }, `${t.name} (${t.jobs} job${t.jobs === 1 ? '' : 's'})`)));
|
||||
} catch (err) {
|
||||
dom.errorBanner.textContent = `Couldn't load the companies: ${err.message}`;
|
||||
dom.errorBanner.hidden = false;
|
||||
return;
|
||||
}
|
||||
// ?tenant= from a link: match case-insensitively, as the API does
|
||||
const requested = (params.get('tenant') ?? '').toLowerCase();
|
||||
const match = [...dom.tenant.options].find((o) => o.value && o.value.toLowerCase() === requested);
|
||||
if (match) dom.tenant.value = match.value;
|
||||
await load();
|
||||
}
|
||||
|
||||
main();
|
||||
@@ -0,0 +1,18 @@
|
||||
import { CONFIG } from './config.js';
|
||||
|
||||
/**
|
||||
* Fills in the top bar's "signed in as … / Sign out" from /api/v1/me, and shows the Admin link
|
||||
* to admins. Pass the elements that exist on the page; missing ones are skipped.
|
||||
*/
|
||||
export function showSignedInUser(me, { signedIn, signedInEmail, signOut, adminLink } = {}) {
|
||||
const who = me?.email || me?.name;
|
||||
if (who && signedIn) { // nothing to show without Microsoft 365 sign-in (dev, mock)
|
||||
signedInEmail.textContent = who;
|
||||
signOut.href = CONFIG.signOutUrl;
|
||||
signedIn.hidden = false;
|
||||
}
|
||||
if (me?.is_admin && adminLink) {
|
||||
adminLink.href = `/admin.html${location.search}`; // keeps ?mock=1
|
||||
adminLink.hidden = false;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
// Run/job statuses: labels, icons, sort order, and the badge that shows one.
|
||||
import { el } from './dom.js';
|
||||
|
||||
// rank drives default sort: worst first
|
||||
export const STATUS = {
|
||||
failed: { label: 'Failed', icon: '✕', rank: 0 },
|
||||
warning: { label: 'Warning', icon: '!', rank: 1 },
|
||||
unknown: { label: 'Unknown', icon: '?', rank: 2 },
|
||||
running: { label: 'Running', icon: '↻', rank: 3 },
|
||||
success: { label: 'Success', icon: '✓', rank: 4 },
|
||||
};
|
||||
|
||||
// Anything the backend sends that we don't recognise becomes "unknown" rather than breaking the UI
|
||||
export function normalizeStatus(raw) {
|
||||
const s = String(raw ?? '').toLowerCase();
|
||||
return Object.hasOwn(STATUS, s) ? s : 'unknown';
|
||||
}
|
||||
|
||||
export function statusBadge(status, { asButton = false, opensDialog = false, dataset = null, title = null } = {}) {
|
||||
const { label, icon } = STATUS[status];
|
||||
const content = [el('span', { class: 'badge__icon', 'aria-hidden': 'true' }, icon), label];
|
||||
return asButton
|
||||
? el('button', {
|
||||
type: 'button',
|
||||
class: `badge badge--${status}`,
|
||||
'aria-haspopup': opensDialog ? 'dialog' : null,
|
||||
'aria-label': `${label} – view details`,
|
||||
title,
|
||||
dataset,
|
||||
}, content)
|
||||
: el('span', { class: `badge badge--${status}`, title }, content);
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Backup report · Backup Monitor</title>
|
||||
<link rel="stylesheet" href="/css/styles.css">
|
||||
<script type="module" src="/js/report.js"></script>
|
||||
</head>
|
||||
<body>
|
||||
<header class="topbar no-print">
|
||||
<div class="topbar__nav">
|
||||
<a id="back-link" class="btn" href="/">← Dashboard</a>
|
||||
<h1 class="topbar__title">Reports</h1>
|
||||
</div>
|
||||
<div class="topbar__meta">
|
||||
<span id="signed-in" class="topbar__user" hidden>
|
||||
<span id="signed-in-email" class="muted"></span>
|
||||
<a id="sign-out" class="btn" href="#">Sign out</a>
|
||||
</span>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main class="container container--narrow">
|
||||
<form id="report-form" class="report-controls no-print">
|
||||
<div class="report-controls__field">
|
||||
<label for="tenant">Company</label>
|
||||
<select id="tenant" name="tenant" class="input" required>
|
||||
<option value="">Choose a company…</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="report-controls__field">
|
||||
<label for="days">Period</label>
|
||||
<select id="days" name="days" class="input">
|
||||
<option value="7">Last 7 days</option>
|
||||
<option value="30" selected>Last 30 days</option>
|
||||
<option value="90">Last 90 days</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="report-controls__actions">
|
||||
<button id="print-btn" class="btn btn--primary" type="button" disabled>Print / Save as PDF</button>
|
||||
<a id="csv-link" class="btn" aria-disabled="true">Download CSV</a>
|
||||
</div>
|
||||
</form>
|
||||
|
||||
<div id="error-banner" class="banner banner--error" role="alert" hidden></div>
|
||||
<p id="choose-hint" class="empty no-print">Choose a company to see its backup report.</p>
|
||||
<p id="loading" class="muted no-print" hidden>Loading report…</p>
|
||||
|
||||
<article id="report" class="report" aria-labelledby="report-title" hidden>
|
||||
<header class="report__header">
|
||||
<h2 id="report-title" class="report__title"></h2>
|
||||
<p id="report-period" class="report__period"></p>
|
||||
</header>
|
||||
|
||||
<section aria-label="Summary">
|
||||
<dl id="report-summary" class="report-summary"></dl>
|
||||
</section>
|
||||
|
||||
<section aria-labelledby="jobs-heading">
|
||||
<h3 id="jobs-heading" class="report__section-title">Jobs</h3>
|
||||
<div class="table-wrap">
|
||||
<table class="jobs report-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th scope="col">Job</th>
|
||||
<th scope="col">Source</th>
|
||||
<th scope="col" class="num">Runs</th>
|
||||
<th scope="col" class="num">Successful</th>
|
||||
<th scope="col" class="num">Warnings</th>
|
||||
<th scope="col" class="num">Failed</th>
|
||||
<th scope="col" class="num">Success rate</th>
|
||||
<th scope="col">Last result</th>
|
||||
<th scope="col">Last successful backup</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="jobs-body"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section aria-labelledby="daily-heading">
|
||||
<h3 id="daily-heading" class="report__section-title">Day by day</h3>
|
||||
<ul class="report-legend" aria-label="Legend">
|
||||
<li><span class="day day--success" aria-hidden="true"></span>Successful</li>
|
||||
<li><span class="day day--warning" aria-hidden="true"></span>Warning</li>
|
||||
<li><span class="day day--failed" aria-hidden="true"></span>Failed</li>
|
||||
<li><span class="day day--running" aria-hidden="true"></span>In progress</li>
|
||||
<li><span class="day day--none" aria-hidden="true"></span>No backup</li>
|
||||
<li><span class="day day--untracked" aria-hidden="true"></span>Not monitored yet</li>
|
||||
</ul>
|
||||
<div id="daily" class="report-daily"></div>
|
||||
</section>
|
||||
|
||||
<section aria-labelledby="issues-heading">
|
||||
<h3 id="issues-heading" class="report__section-title">Warnings and failures</h3>
|
||||
<p id="issues-none" class="muted" hidden>None in this period.</p>
|
||||
<div id="issues-wrap" class="table-wrap">
|
||||
<table class="jobs report-table report-issues">
|
||||
<thead>
|
||||
<tr>
|
||||
<th scope="col">When</th>
|
||||
<th scope="col">Job</th>
|
||||
<th scope="col">Result</th>
|
||||
<th scope="col">Details</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="issues-body"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<footer id="report-footer" class="report__footer muted"></footer>
|
||||
</article>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user