Compare commits
64
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dd3e2fe885 | ||
|
|
51b97c4556 | ||
|
|
555e928e01 | ||
|
|
b625a8af65 | ||
|
|
a14b1b0dc9 | ||
|
|
5982fc9434 | ||
|
|
10181e014b | ||
|
|
b4b703ceee | ||
|
|
10bfc7a3aa | ||
|
|
960839ed76 | ||
|
|
f0b4d84c0b | ||
|
|
067b714678 | ||
|
|
3355a7d4bc | ||
|
|
e916afaae1 | ||
|
|
1f96d11a27 | ||
|
|
41c4fa2700 | ||
|
|
a84a35463b | ||
|
|
96b729bec8 | ||
|
|
ae574fa2b6 | ||
|
|
e50711b93b | ||
|
|
8fd17e3817 | ||
|
|
ba798284df | ||
|
|
9ae93cfbc2 | ||
|
|
986cf3be88 | ||
|
|
17ee303670 | ||
|
|
effafc5fa0 | ||
|
|
2318579baa | ||
|
|
1f02c1c84b | ||
|
|
3b1aee7016 | ||
|
|
d3ab05416d | ||
|
|
728b5771cc | ||
|
|
f55a44f2e6 | ||
|
|
b8452e1fbb | ||
|
|
d12e00f69e | ||
|
|
33c1613ef9 | ||
|
|
8e29b9cbc4 | ||
|
|
49bf9ef094 | ||
|
|
4d183da7bd | ||
|
|
25f7717ace | ||
|
|
be23e03c5f | ||
|
|
65c587da0d | ||
|
|
29f961f80e | ||
|
|
02789b1910 | ||
|
|
54a6c0b3d6 | ||
|
|
f11345bb0f | ||
|
|
7217b3e9a8 | ||
|
|
47ec7dc9a7 | ||
|
|
15ea9e0fec | ||
|
|
fe4bd6827c | ||
|
|
4e57eb1a46 | ||
|
|
670994b937 | ||
|
|
a7c3169606 | ||
|
|
938a0a2af0 | ||
|
|
f7e029db8a | ||
|
|
2df1a75fd5 | ||
|
|
4d8ee209a5 | ||
|
|
6c486fbaf3 | ||
|
|
fdc7deff16 | ||
|
|
0391807027 | ||
|
|
b0c560377b | ||
|
|
17346cc058 | ||
|
|
65eb3c88f3 | ||
|
|
996fe33a40 | ||
|
|
bf752a904f |
@@ -0,0 +1,8 @@
|
|||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
.venv/
|
||||||
|
venv/
|
||||||
|
.pytest_cache/
|
||||||
|
*.db
|
||||||
|
*.db-wal
|
||||||
|
*.db-shm
|
||||||
@@ -1,3 +1,108 @@
|
|||||||
# Backup-Dashboard
|
# Backup-Dashboard
|
||||||
|
|
||||||
Proof of concept backup dashboard, intended to display backup jobs from multiple sources
|
Proof of concept backup dashboard, intended to display backup jobs from multiple sources.
|
||||||
|
|
||||||
|
```
|
||||||
|
collectors ──POST /api/v1/ingest (API key)──▶ ┌──────────────┐
|
||||||
|
│ FastAPI + │
|
||||||
|
browser ──nginx (basic auth, TLS)──▶ /api/* ─▶│ SQLite │
|
||||||
|
└─ static files: public/ └──────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
- `public/`: static frontend (no build step). Add `?mock=1` to the URL to use built-in sample data.
|
||||||
|
- `backend/`: FastAPI API backed by a single SQLite file.
|
||||||
|
- `deploy/`: nginx site and systemd unit.
|
||||||
|
- `collectors/`: scripts that report results from backup products ([Veeam B&R](collectors/veeam/README.md), [Azure Backup / MARS](collectors/mars/README.md), [Windows Server Backup](collectors/wsb/README.md)).
|
||||||
|
|
||||||
|
## Data model
|
||||||
|
|
||||||
|
- **Tenant** (company) → **Job** → **Run**.
|
||||||
|
- A job is keyed by `(tenant, source_system, job.external_id)`, so renaming a job at the source updates it instead of creating a duplicate.
|
||||||
|
- A run has `status` (`running` | `success` | `warning` | `failed`), `started_at` and `finished_at`.
|
||||||
|
The dashboard's **Last Run** is the most recent `finished_at`. While a job is running it shows
|
||||||
|
**Running**, and Last Run still shows when the previous run finished.
|
||||||
|
- Runs older than `BACKUP_DASH_RETENTION_DAYS` (default 90, minimum 30) are purged every 6 hours.
|
||||||
|
|
||||||
|
## Reporting results (collectors)
|
||||||
|
|
||||||
|
Each collector (a script on a Veeam server, a webhook relay, etc.) gets its own API key:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend && python manage.py create-key veeam-acme
|
||||||
|
```
|
||||||
|
|
||||||
|
Then it POSTs one report per run. Send `running` when a run starts (optional) and the final status
|
||||||
|
when it ends. Reports with the same `started_at` (or `run.external_id`) update the same run, and
|
||||||
|
retries are safe.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST https://backups.example.com/api/v1/ingest \
|
||||||
|
-H "Authorization: Bearer bdk_..." -H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"tenant": "Acme Corp",
|
||||||
|
"source_system": "Veeam B&R",
|
||||||
|
"job": { "external_id": "4f1c…", "name": "File Server – Nightly", "next_run_at": "2026-10-05T22:00:00-04:00" },
|
||||||
|
"run": {
|
||||||
|
"status": "success",
|
||||||
|
"started_at": "2026-10-04T22:00:05-04:00",
|
||||||
|
"finished_at": "2026-10-04T22:47:12-04:00",
|
||||||
|
"message": "Job completed successfully.",
|
||||||
|
"size_bytes": 128849018880,
|
||||||
|
"details": { "repository": "REPO-01" }
|
||||||
|
}
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
- All timestamps must include a timezone offset (`Get-Date -Format o` in PowerShell does this).
|
||||||
|
- `job.next_run_at` controls the **Overdue** flag. If you omit it, the stored value is kept. If you send `null`, it's cleared.
|
||||||
|
- `run` can be omitted to only register or update a job.
|
||||||
|
- Unknown fields are rejected (HTTP 422), so typos in collector scripts fail loudly.
|
||||||
|
|
||||||
|
## API (read)
|
||||||
|
|
||||||
|
| Endpoint | Returns |
|
||||||
|
|---|---|
|
||||||
|
| `GET /api/v1/jobs` | All jobs with current status, `running_since`, `last_run_at`, `next_run_at` |
|
||||||
|
| `GET /api/v1/jobs/{id}` | One job, plus the last finished run's duration, size, message and details |
|
||||||
|
| `GET /api/v1/jobs/{id}/runs?days=30` | Run history (no messages), newest first |
|
||||||
|
| `GET /api/v1/runs/{id}` | One run including message and details |
|
||||||
|
| `GET /api/v1/health` | `{"status": "ok"}` |
|
||||||
|
|
||||||
|
The OpenAPI schema is at `/api/v1/openapi.json`.
|
||||||
|
|
||||||
|
## Authentication
|
||||||
|
|
||||||
|
- **Dashboard users:** nginx basic auth over HTTPS (`deploy/nginx.conf`). Every user can see every tenant.
|
||||||
|
*Planned:* Microsoft 365 sign-in through [oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/)
|
||||||
|
with the Entra ID provider and nginx `auth_request`. That's an nginx-only change. The frontend
|
||||||
|
already tells the user to reload when a session expires (HTTP 401).
|
||||||
|
- **Collectors:** per-collector bearer keys, checked by the app. Only a SHA-256 hash is stored.
|
||||||
|
To revoke one, run `python manage.py revoke-key NAME`.
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
python -m venv .venv && . .venv/bin/activate
|
||||||
|
pip install -r requirements.txt -r requirements-dev.txt
|
||||||
|
pytest
|
||||||
|
python manage.py --create seed-demo # 30 days of fake history in ./backup-dashboard.db
|
||||||
|
BACKUP_DASH_SERVE_STATIC=1 uvicorn --factory app.main:create_app --reload
|
||||||
|
# → http://localhost:8000
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deployment (Linux, nginx)
|
||||||
|
|
||||||
|
1. Copy the repo to `/opt/backup-dashboard`, create a venv at `/opt/backup-dashboard/venv`, and run `pip install -r backend/requirements.txt`.
|
||||||
|
2. Copy `deploy/backup-dashboard.env` to `/etc/backup-dashboard/backup-dashboard.env`. Both the service and `manage.py` read their settings from it.
|
||||||
|
3. Create a `backupdash` system user, then install `deploy/backup-dashboard-api.service` into `/etc/systemd/system/` and enable it. The database is created on first start.
|
||||||
|
4. Copy `public/` to `/var/www/backup-dashboard/public`, install `deploy/nginx.conf`, get a certificate with certbot, and create users with `htpasswd -B`.
|
||||||
|
5. Run `manage.py` as the service user. It finds the database through the env file:
|
||||||
|
```bash
|
||||||
|
cd /opt/backup-dashboard/backend
|
||||||
|
sudo -u backupdash /opt/backup-dashboard/venv/bin/python manage.py create-key veeam-acme
|
||||||
|
```
|
||||||
|
It prints the database path it's using and won't create a new database unless you pass `--create`.
|
||||||
|
|
||||||
|
Back up the SQLite file with `sqlite3 backup-dashboard.db ".backup copy.db"`. Don't use a plain `cp`
|
||||||
|
while the service is running.
|
||||||
|
|||||||
@@ -0,0 +1,8 @@
|
|||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
.venv/
|
||||||
|
venv/
|
||||||
|
.pytest_cache/
|
||||||
|
*.db
|
||||||
|
*.db-wal
|
||||||
|
*.db-shm
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
"""Collector (machine) authentication.
|
||||||
|
|
||||||
|
Dashboard *users* are authenticated by nginx in front of the app (basic auth today,
|
||||||
|
oauth2-proxy + Microsoft Entra ID later), so nothing here deals with people.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import secrets
|
||||||
|
import sqlite3
|
||||||
|
|
||||||
|
from fastapi import Depends, Header, HTTPException
|
||||||
|
|
||||||
|
from .db import get_db
|
||||||
|
from .timeutil import to_db, utcnow
|
||||||
|
|
||||||
|
KEY_PREFIX = "bdk_"
|
||||||
|
|
||||||
|
|
||||||
|
def generate_key() -> str:
|
||||||
|
return KEY_PREFIX + secrets.token_urlsafe(32)
|
||||||
|
|
||||||
|
|
||||||
|
def hash_key(key: str) -> str:
|
||||||
|
# Keys are 256-bit random, so a fast unsalted hash is fine (no dictionary to attack).
|
||||||
|
return hashlib.sha256(key.encode()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def require_ingest_key(
|
||||||
|
authorization: str | None = Header(None),
|
||||||
|
conn: sqlite3.Connection = Depends(get_db),
|
||||||
|
) -> str:
|
||||||
|
scheme, _, token = (authorization or "").partition(" ")
|
||||||
|
token = token.strip()
|
||||||
|
if scheme.lower() != "bearer" or not token:
|
||||||
|
raise HTTPException(401, "Missing bearer token", headers={"WWW-Authenticate": "Bearer"})
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT id, name FROM api_keys WHERE key_hash = ? AND revoked_at IS NULL", (hash_key(token),)
|
||||||
|
).fetchone()
|
||||||
|
if row is None:
|
||||||
|
raise HTTPException(401, "Invalid or revoked API key", headers={"WWW-Authenticate": "Bearer"})
|
||||||
|
conn.execute("UPDATE api_keys SET last_used_at = ? WHERE id = ?", (to_db(utcnow()), row["id"]))
|
||||||
|
return row["name"]
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
import sqlite3
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from contextlib import contextmanager
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from fastapi import Request
|
||||||
|
|
||||||
|
SCHEMA_VERSION = 1
|
||||||
|
|
||||||
|
SCHEMA = """
|
||||||
|
CREATE TABLE IF NOT EXISTS tenants (
|
||||||
|
id INTEGER PRIMARY KEY,
|
||||||
|
name TEXT NOT NULL UNIQUE COLLATE NOCASE
|
||||||
|
);
|
||||||
|
|
||||||
|
-- A job is identified by (tenant, source system, the source's own job id), so two
|
||||||
|
-- products can't collide and renaming a job in Veeam doesn't create a duplicate.
|
||||||
|
CREATE TABLE IF NOT EXISTS jobs (
|
||||||
|
id INTEGER PRIMARY KEY,
|
||||||
|
tenant_id INTEGER NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
|
||||||
|
source_system TEXT NOT NULL,
|
||||||
|
external_id TEXT NOT NULL,
|
||||||
|
name TEXT NOT NULL,
|
||||||
|
next_run_at TEXT,
|
||||||
|
created_at TEXT NOT NULL,
|
||||||
|
updated_at TEXT NOT NULL,
|
||||||
|
UNIQUE (tenant_id, source_system, external_id)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS runs (
|
||||||
|
id INTEGER PRIMARY KEY,
|
||||||
|
job_id INTEGER NOT NULL REFERENCES jobs(id) ON DELETE CASCADE,
|
||||||
|
external_id TEXT NOT NULL,
|
||||||
|
status TEXT NOT NULL CHECK (status IN ('running', 'success', 'warning', 'failed')),
|
||||||
|
started_at TEXT NOT NULL,
|
||||||
|
finished_at TEXT,
|
||||||
|
message TEXT NOT NULL DEFAULT '',
|
||||||
|
size_bytes INTEGER,
|
||||||
|
details TEXT NOT NULL DEFAULT '{}',
|
||||||
|
received_at TEXT NOT NULL,
|
||||||
|
CHECK ((status = 'running') = (finished_at IS NULL)),
|
||||||
|
UNIQUE (job_id, external_id)
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS runs_job_started ON runs (job_id, started_at);
|
||||||
|
CREATE INDEX IF NOT EXISTS runs_job_finished ON runs (job_id, finished_at);
|
||||||
|
|
||||||
|
-- Keys used by collectors to push results. Only the SHA-256 is stored.
|
||||||
|
CREATE TABLE IF NOT EXISTS api_keys (
|
||||||
|
id INTEGER PRIMARY KEY,
|
||||||
|
name TEXT NOT NULL UNIQUE,
|
||||||
|
key_hash TEXT NOT NULL UNIQUE,
|
||||||
|
created_at TEXT NOT NULL,
|
||||||
|
last_used_at TEXT,
|
||||||
|
revoked_at TEXT
|
||||||
|
);
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def connect(path: Path | str) -> sqlite3.Connection:
|
||||||
|
# Autocommit mode; writes that must be atomic go through transaction().
|
||||||
|
# check_same_thread=False: FastAPI may run a dependency and its endpoint on different
|
||||||
|
# threadpool threads, but a connection is only ever used by one request at a time.
|
||||||
|
conn = sqlite3.connect(path, timeout=10, isolation_level=None, check_same_thread=False)
|
||||||
|
conn.row_factory = sqlite3.Row
|
||||||
|
conn.execute("PRAGMA foreign_keys = ON")
|
||||||
|
conn.execute("PRAGMA busy_timeout = 10000")
|
||||||
|
return conn
|
||||||
|
|
||||||
|
|
||||||
|
def init_db(path: Path | str) -> None:
|
||||||
|
conn = connect(path)
|
||||||
|
try:
|
||||||
|
conn.execute("PRAGMA journal_mode = WAL")
|
||||||
|
version = conn.execute("PRAGMA user_version").fetchone()[0]
|
||||||
|
if version > SCHEMA_VERSION:
|
||||||
|
raise RuntimeError(f"Database schema v{version} is newer than this app (v{SCHEMA_VERSION})")
|
||||||
|
conn.executescript(SCHEMA)
|
||||||
|
conn.execute(f"PRAGMA user_version = {SCHEMA_VERSION}")
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def transaction(conn: sqlite3.Connection) -> Iterator[None]:
|
||||||
|
conn.execute("BEGIN IMMEDIATE")
|
||||||
|
try:
|
||||||
|
yield
|
||||||
|
except BaseException:
|
||||||
|
conn.execute("ROLLBACK")
|
||||||
|
raise
|
||||||
|
conn.execute("COMMIT")
|
||||||
|
|
||||||
|
|
||||||
|
def get_db(request: Request) -> Iterator[sqlite3.Connection]:
|
||||||
|
conn = connect(request.app.state.settings.db_path)
|
||||||
|
try:
|
||||||
|
yield conn
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
@@ -0,0 +1,118 @@
|
|||||||
|
import asyncio
|
||||||
|
import logging
|
||||||
|
import sqlite3
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
from datetime import timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, FastAPI, HTTPException, Query, Request
|
||||||
|
|
||||||
|
from . import db, repo
|
||||||
|
from .auth import require_ingest_key
|
||||||
|
from .models import IngestIn
|
||||||
|
from .settings import HISTORY_DAYS, Settings, load_settings
|
||||||
|
from .timeutil import to_db, utcnow
|
||||||
|
|
||||||
|
log = logging.getLogger("backup_dashboard")
|
||||||
|
PURGE_INTERVAL_SECONDS = 6 * 3600
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/api/v1")
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/health")
|
||||||
|
def health(conn: sqlite3.Connection = Depends(db.get_db)):
|
||||||
|
conn.execute("SELECT 1")
|
||||||
|
return {"status": "ok"}
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/jobs")
|
||||||
|
def list_jobs(conn: sqlite3.Connection = Depends(db.get_db)):
|
||||||
|
return {"generated_at": to_db(utcnow()), "jobs": repo.list_jobs(conn)}
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/jobs/{job_id}")
|
||||||
|
def get_job(job_id: int, conn: sqlite3.Connection = Depends(db.get_db)):
|
||||||
|
job = repo.get_job(conn, job_id)
|
||||||
|
if job is None:
|
||||||
|
raise HTTPException(404, "Job not found")
|
||||||
|
return job
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/jobs/{job_id}/runs")
|
||||||
|
def list_runs(
|
||||||
|
job_id: int,
|
||||||
|
days: int = Query(HISTORY_DAYS, ge=1, le=HISTORY_DAYS),
|
||||||
|
conn: sqlite3.Connection = Depends(db.get_db),
|
||||||
|
):
|
||||||
|
if conn.execute("SELECT 1 FROM jobs WHERE id = ?", (job_id,)).fetchone() is None:
|
||||||
|
raise HTTPException(404, "Job not found")
|
||||||
|
since = utcnow() - timedelta(days=days)
|
||||||
|
return {"job_id": job_id, "since": to_db(since), "runs": repo.list_runs(conn, job_id, since)}
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/runs/{run_id}")
|
||||||
|
def get_run(run_id: int, conn: sqlite3.Connection = Depends(db.get_db)):
|
||||||
|
run = repo.get_run(conn, run_id)
|
||||||
|
if run is None:
|
||||||
|
raise HTTPException(404, "Run not found")
|
||||||
|
return run
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/ingest")
|
||||||
|
def ingest(
|
||||||
|
item: IngestIn,
|
||||||
|
key_name: str = Depends(require_ingest_key),
|
||||||
|
conn: sqlite3.Connection = Depends(db.get_db),
|
||||||
|
):
|
||||||
|
job_id, run_id = repo.ingest(conn, item)
|
||||||
|
log.info("ingest key=%s tenant=%r job=%r run=%s", key_name, item.tenant, item.job.name, run_id)
|
||||||
|
return {"job_id": job_id, "run_id": run_id}
|
||||||
|
|
||||||
|
|
||||||
|
async def _purge_loop(settings: Settings) -> None:
|
||||||
|
def purge() -> int:
|
||||||
|
conn = db.connect(settings.db_path)
|
||||||
|
try:
|
||||||
|
return repo.purge_old_runs(conn, settings.retention_days)
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
if deleted := await asyncio.to_thread(purge):
|
||||||
|
log.info("purged %d runs older than %d days", deleted, settings.retention_days)
|
||||||
|
except Exception:
|
||||||
|
log.exception("run purge failed")
|
||||||
|
await asyncio.sleep(PURGE_INTERVAL_SECONDS)
|
||||||
|
|
||||||
|
|
||||||
|
def create_app(settings: Settings | None = None) -> FastAPI:
|
||||||
|
settings = settings or load_settings()
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def lifespan(app: FastAPI):
|
||||||
|
db.init_db(settings.db_path)
|
||||||
|
task = asyncio.create_task(_purge_loop(settings))
|
||||||
|
try:
|
||||||
|
yield
|
||||||
|
finally:
|
||||||
|
task.cancel()
|
||||||
|
|
||||||
|
# Interactive docs are off: nginx's strict CSP would block their CDN assets anyway.
|
||||||
|
app = FastAPI(
|
||||||
|
title="Backup Dashboard API",
|
||||||
|
lifespan=lifespan,
|
||||||
|
docs_url=None,
|
||||||
|
redoc_url=None,
|
||||||
|
openapi_url="/api/v1/openapi.json",
|
||||||
|
)
|
||||||
|
app.state.settings = settings
|
||||||
|
app.include_router(router)
|
||||||
|
|
||||||
|
if settings.serve_static:
|
||||||
|
from fastapi.staticfiles import StaticFiles
|
||||||
|
|
||||||
|
public = Path(__file__).resolve().parents[2] / "public"
|
||||||
|
app.mount("/", StaticFiles(directory=public, html=True), name="static")
|
||||||
|
|
||||||
|
return app
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
import json
|
||||||
|
from typing import Annotated, Any, Literal
|
||||||
|
|
||||||
|
from pydantic import AwareDatetime, BaseModel, ConfigDict, Field, StringConstraints, field_validator, model_validator
|
||||||
|
|
||||||
|
RunStatus = Literal["running", "success", "warning", "failed"]
|
||||||
|
Name = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=200)]
|
||||||
|
|
||||||
|
MAX_MESSAGE_CHARS = 32_000
|
||||||
|
MAX_DETAILS_BYTES = 16_000
|
||||||
|
|
||||||
|
|
||||||
|
class Strict(BaseModel):
|
||||||
|
# Reject unknown fields so a typo in a collector script fails loudly instead of being dropped.
|
||||||
|
model_config = ConfigDict(extra="forbid")
|
||||||
|
|
||||||
|
|
||||||
|
class JobIn(Strict):
|
||||||
|
external_id: Name = Field(description="The source system's own stable ID for the job")
|
||||||
|
name: Name
|
||||||
|
# Omit to leave the stored value untouched; send null to clear it.
|
||||||
|
next_run_at: AwareDatetime | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class RunIn(Strict):
|
||||||
|
external_id: Name | None = Field(
|
||||||
|
None, description="The source's ID for this run. Defaults to started_at, so a 'running' "
|
||||||
|
"report and its later completion report merge into one run.")
|
||||||
|
status: RunStatus
|
||||||
|
started_at: AwareDatetime
|
||||||
|
# Omit for a finished run to use the time the report was received.
|
||||||
|
finished_at: AwareDatetime | None = None
|
||||||
|
message: str = ""
|
||||||
|
size_bytes: int | None = Field(None, ge=0)
|
||||||
|
details: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
@field_validator("message")
|
||||||
|
@classmethod
|
||||||
|
def truncate_message(cls, v: str) -> str:
|
||||||
|
# Backup logs can be huge; keep the head, which is where the summary usually is.
|
||||||
|
return v if len(v) <= MAX_MESSAGE_CHARS else v[:MAX_MESSAGE_CHARS] + "\n… [truncated]"
|
||||||
|
|
||||||
|
@field_validator("details")
|
||||||
|
@classmethod
|
||||||
|
def limit_details(cls, v: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
if len(json.dumps(v)) > MAX_DETAILS_BYTES:
|
||||||
|
raise ValueError(f"details must serialize to at most {MAX_DETAILS_BYTES} bytes")
|
||||||
|
return v
|
||||||
|
|
||||||
|
@model_validator(mode="after")
|
||||||
|
def check_times(self) -> "RunIn":
|
||||||
|
if self.status == "running" and self.finished_at is not None:
|
||||||
|
raise ValueError("a running run cannot have finished_at")
|
||||||
|
if self.finished_at is not None and self.finished_at < self.started_at:
|
||||||
|
raise ValueError("finished_at is before started_at")
|
||||||
|
return self
|
||||||
|
|
||||||
|
|
||||||
|
class IngestIn(Strict):
|
||||||
|
tenant: Name = Field(description="Company name as shown on the dashboard")
|
||||||
|
source_system: Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=100)]
|
||||||
|
job: JobIn
|
||||||
|
run: RunIn | None = Field(None, description="Omit to only register/update the job (e.g. next_run_at)")
|
||||||
@@ -0,0 +1,162 @@
|
|||||||
|
import json
|
||||||
|
import sqlite3
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .db import transaction
|
||||||
|
from .models import IngestIn
|
||||||
|
from .timeutil import to_db, utcnow
|
||||||
|
|
||||||
|
# One row per job: `cur` is the most recent run (may be in progress), `fin` the most recent
|
||||||
|
# finished one. "Last run" on the dashboard is fin.finished_at, so a job that's currently
|
||||||
|
# running still shows when its previous run completed.
|
||||||
|
_JOB_SQL = """
|
||||||
|
SELECT j.id, t.name AS company_name, j.name AS job_name, j.source_system, j.next_run_at,
|
||||||
|
cur.status,
|
||||||
|
CASE WHEN cur.status = 'running' THEN cur.started_at END AS running_since,
|
||||||
|
fin.status AS last_status,
|
||||||
|
fin.started_at AS last_started_at,
|
||||||
|
fin.finished_at AS last_run_at,
|
||||||
|
unixepoch(fin.finished_at) - unixepoch(fin.started_at) AS duration_seconds,
|
||||||
|
fin.size_bytes,
|
||||||
|
fin.details,
|
||||||
|
COALESCE(fin.message, cur.message, '') AS message
|
||||||
|
FROM jobs j
|
||||||
|
JOIN tenants t ON t.id = j.tenant_id
|
||||||
|
LEFT JOIN runs cur ON cur.id = (
|
||||||
|
SELECT r.id FROM runs r WHERE r.job_id = j.id
|
||||||
|
ORDER BY r.started_at DESC, r.id DESC LIMIT 1)
|
||||||
|
LEFT JOIN runs fin ON fin.id = (
|
||||||
|
SELECT r.id FROM runs r WHERE r.job_id = j.id AND r.finished_at IS NOT NULL
|
||||||
|
ORDER BY r.finished_at DESC, r.id DESC LIMIT 1)
|
||||||
|
"""
|
||||||
|
|
||||||
|
_SUMMARY_FIELDS = (
|
||||||
|
"id", "company_name", "job_name", "source_system", "status",
|
||||||
|
"running_since", "last_run_at", "next_run_at", "message",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def list_jobs(conn: sqlite3.Connection) -> list[dict[str, Any]]:
|
||||||
|
rows = conn.execute(_JOB_SQL + " ORDER BY t.name, j.name").fetchall()
|
||||||
|
return [{k: row[k] for k in _SUMMARY_FIELDS} for row in rows]
|
||||||
|
|
||||||
|
|
||||||
|
def get_job(conn: sqlite3.Connection, job_id: int) -> dict[str, Any] | None:
|
||||||
|
row = conn.execute(_JOB_SQL + " WHERE j.id = ?", (job_id,)).fetchone()
|
||||||
|
if row is None:
|
||||||
|
return None
|
||||||
|
job = dict(row)
|
||||||
|
job["details"] = json.loads(job["details"]) if job["details"] else {}
|
||||||
|
return job
|
||||||
|
|
||||||
|
|
||||||
|
def list_runs(conn: sqlite3.Connection, job_id: int, since: datetime) -> list[dict[str, Any]]:
|
||||||
|
# Messages/details are left out: an hourly job has ~720 runs a month and logs can be large.
|
||||||
|
# The UI fetches a single run's full record on demand.
|
||||||
|
rows = conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT id, status, started_at, finished_at,
|
||||||
|
unixepoch(finished_at) - unixepoch(started_at) AS duration_seconds, size_bytes
|
||||||
|
FROM runs WHERE job_id = ? AND started_at >= ?
|
||||||
|
ORDER BY started_at DESC, id DESC
|
||||||
|
""",
|
||||||
|
(job_id, to_db(since)),
|
||||||
|
).fetchall()
|
||||||
|
return [dict(r) for r in rows]
|
||||||
|
|
||||||
|
|
||||||
|
def get_run(conn: sqlite3.Connection, run_id: int) -> dict[str, Any] | None:
|
||||||
|
row = conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT id, job_id, status, started_at, finished_at,
|
||||||
|
unixepoch(finished_at) - unixepoch(started_at) AS duration_seconds,
|
||||||
|
size_bytes, message, details
|
||||||
|
FROM runs WHERE id = ?
|
||||||
|
""",
|
||||||
|
(run_id,),
|
||||||
|
).fetchone()
|
||||||
|
if row is None:
|
||||||
|
return None
|
||||||
|
run = dict(row)
|
||||||
|
run["details"] = json.loads(run["details"])
|
||||||
|
return run
|
||||||
|
|
||||||
|
|
||||||
|
def ingest(conn: sqlite3.Connection, item: IngestIn) -> tuple[int, int | None]:
|
||||||
|
"""Upsert tenant → job → run. Safe to call repeatedly with the same payload."""
|
||||||
|
now = utcnow()
|
||||||
|
now_s = to_db(now)
|
||||||
|
with transaction(conn):
|
||||||
|
conn.execute("INSERT INTO tenants (name) VALUES (?) ON CONFLICT (name) DO NOTHING", (item.tenant,))
|
||||||
|
tenant_id = conn.execute("SELECT id FROM tenants WHERE name = ?", (item.tenant,)).fetchone()[0]
|
||||||
|
|
||||||
|
job_id = conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO jobs (tenant_id, source_system, external_id, name, next_run_at, created_at, updated_at)
|
||||||
|
VALUES (:tenant_id, :source, :ext, :name, :next_run, :now, :now)
|
||||||
|
ON CONFLICT (tenant_id, source_system, external_id) DO UPDATE SET
|
||||||
|
name = excluded.name,
|
||||||
|
next_run_at = CASE WHEN :set_next_run THEN excluded.next_run_at ELSE jobs.next_run_at END,
|
||||||
|
updated_at = excluded.updated_at
|
||||||
|
RETURNING id
|
||||||
|
""",
|
||||||
|
{
|
||||||
|
"tenant_id": tenant_id,
|
||||||
|
"source": item.source_system,
|
||||||
|
"ext": item.job.external_id,
|
||||||
|
"name": item.job.name,
|
||||||
|
"next_run": to_db(item.job.next_run_at) if item.job.next_run_at else None,
|
||||||
|
"set_next_run": "next_run_at" in item.job.model_fields_set,
|
||||||
|
"now": now_s,
|
||||||
|
},
|
||||||
|
).fetchone()[0]
|
||||||
|
|
||||||
|
if item.run is None:
|
||||||
|
return job_id, None
|
||||||
|
|
||||||
|
run = item.run
|
||||||
|
started = to_db(run.started_at)
|
||||||
|
finished = None
|
||||||
|
if run.status != "running":
|
||||||
|
finished = to_db(run.finished_at or max(now, run.started_at))
|
||||||
|
params = {
|
||||||
|
"job_id": job_id,
|
||||||
|
"ext": run.external_id or started,
|
||||||
|
"status": run.status,
|
||||||
|
"started": started,
|
||||||
|
"finished": finished,
|
||||||
|
"message": run.message,
|
||||||
|
"size": run.size_bytes,
|
||||||
|
"details": json.dumps(run.details),
|
||||||
|
"now": now_s,
|
||||||
|
}
|
||||||
|
# The WHERE stops a late/retried "running" report from reopening a run that already finished.
|
||||||
|
row = conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO runs (job_id, external_id, status, started_at, finished_at, message,
|
||||||
|
size_bytes, details, received_at)
|
||||||
|
VALUES (:job_id, :ext, :status, :started, :finished, :message, :size, :details, :now)
|
||||||
|
ON CONFLICT (job_id, external_id) DO UPDATE SET
|
||||||
|
status = excluded.status,
|
||||||
|
started_at = excluded.started_at,
|
||||||
|
finished_at = excluded.finished_at,
|
||||||
|
message = excluded.message,
|
||||||
|
size_bytes = excluded.size_bytes,
|
||||||
|
details = excluded.details,
|
||||||
|
received_at = excluded.received_at
|
||||||
|
WHERE NOT (runs.status != 'running' AND excluded.status = 'running')
|
||||||
|
RETURNING id
|
||||||
|
""",
|
||||||
|
params,
|
||||||
|
).fetchone()
|
||||||
|
if row is None:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT id FROM runs WHERE job_id = :job_id AND external_id = :ext", params
|
||||||
|
).fetchone()
|
||||||
|
return job_id, row[0]
|
||||||
|
|
||||||
|
|
||||||
|
def purge_old_runs(conn: sqlite3.Connection, retention_days: int) -> int:
|
||||||
|
cutoff = to_db(utcnow() - timedelta(days=retention_days))
|
||||||
|
return conn.execute("DELETE FROM runs WHERE started_at < ?", (cutoff,)).rowcount
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
import os
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# The modal shows this many days of run history; retention can never be shorter.
|
||||||
|
HISTORY_DAYS = 30
|
||||||
|
|
||||||
|
# Shared by the systemd service (EnvironmentFile=) and manage.py, so both always use the same DB.
|
||||||
|
DEFAULT_ENV_FILE = "/etc/backup-dashboard/backup-dashboard.env"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Settings:
|
||||||
|
db_path: Path
|
||||||
|
retention_days: int = 90
|
||||||
|
# Dev only: serve ../public from the API so you can run without nginx.
|
||||||
|
serve_static: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
def read_env_file(path: Path) -> dict[str, str]:
|
||||||
|
"""Parse simple KEY=VALUE lines (the subset systemd's EnvironmentFile accepts)."""
|
||||||
|
values = {}
|
||||||
|
try:
|
||||||
|
lines = path.read_text().splitlines()
|
||||||
|
except FileNotFoundError:
|
||||||
|
return values
|
||||||
|
for line in lines:
|
||||||
|
line = line.strip()
|
||||||
|
if not line or line.startswith("#") or "=" not in line:
|
||||||
|
continue
|
||||||
|
key, _, value = line.partition("=")
|
||||||
|
values[key.strip()] = value.strip().strip("\"'")
|
||||||
|
return values
|
||||||
|
|
||||||
|
|
||||||
|
def load_settings() -> Settings:
|
||||||
|
# Real environment variables win over the env file; the env file wins over dev defaults.
|
||||||
|
env = read_env_file(Path(os.environ.get("BACKUP_DASH_ENV_FILE", DEFAULT_ENV_FILE))) | os.environ
|
||||||
|
return Settings(
|
||||||
|
db_path=Path(env.get("BACKUP_DASH_DB", "backup-dashboard.db")),
|
||||||
|
retention_days=max(HISTORY_DAYS, int(env.get("BACKUP_DASH_RETENTION_DAYS", "90"))),
|
||||||
|
serve_static=env.get("BACKUP_DASH_SERVE_STATIC") == "1",
|
||||||
|
)
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
|
# Every timestamp in the DB uses this exact format, so string comparison == chronological comparison.
|
||||||
|
DB_FORMAT = "%Y-%m-%dT%H:%M:%SZ"
|
||||||
|
|
||||||
|
|
||||||
|
def utcnow() -> datetime:
|
||||||
|
return datetime.now(UTC).replace(microsecond=0)
|
||||||
|
|
||||||
|
|
||||||
|
def to_db(dt: datetime) -> str:
|
||||||
|
return dt.astimezone(UTC).strftime(DB_FORMAT)
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Admin CLI. Finds the database the same way the API does: BACKUP_DASH_DB from the environment,
|
||||||
|
else from /etc/backup-dashboard/backup-dashboard.env, else ./backup-dashboard.db (dev).
|
||||||
|
Run it as the service user so files it creates stay writable by the API:
|
||||||
|
|
||||||
|
sudo -u backupdash /opt/backup-dashboard/venv/bin/python manage.py create-key veeam-acme
|
||||||
|
|
||||||
|
python manage.py create-key veeam-acme # prints the key once
|
||||||
|
python manage.py list-keys
|
||||||
|
python manage.py revoke-key veeam-acme
|
||||||
|
python manage.py list-jobs
|
||||||
|
python manage.py delete-job 12 # e.g. a decommissioned job stuck as Overdue
|
||||||
|
python manage.py seed-demo # 30 days of fake history for trying the UI
|
||||||
|
python manage.py --db ./dev.db --create seed-demo
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import random
|
||||||
|
import sys
|
||||||
|
from datetime import timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from app import db, repo
|
||||||
|
from app.auth import generate_key, hash_key
|
||||||
|
from app.models import IngestIn
|
||||||
|
from app.settings import DEFAULT_ENV_FILE, load_settings
|
||||||
|
from app.timeutil import to_db, utcnow
|
||||||
|
|
||||||
|
|
||||||
|
def create_key(conn, args):
|
||||||
|
key = generate_key()
|
||||||
|
try:
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO api_keys (name, key_hash, created_at) VALUES (?, ?, ?)",
|
||||||
|
(args.name, hash_key(key), to_db(utcnow())),
|
||||||
|
)
|
||||||
|
except db.sqlite3.IntegrityError:
|
||||||
|
sys.exit(f"A key named {args.name!r} already exists")
|
||||||
|
print(f"API key for {args.name!r} (shown once, store it in the collector's config):\n{key}")
|
||||||
|
|
||||||
|
|
||||||
|
def list_keys(conn, args):
|
||||||
|
for r in conn.execute("SELECT name, created_at, last_used_at, revoked_at FROM api_keys ORDER BY name"):
|
||||||
|
state = f"revoked {r['revoked_at']}" if r["revoked_at"] else "active"
|
||||||
|
print(f"{r['name']:<30} {state:<30} created {r['created_at']} last used {r['last_used_at'] or 'never'}")
|
||||||
|
|
||||||
|
|
||||||
|
def revoke_key(conn, args):
|
||||||
|
cur = conn.execute(
|
||||||
|
"UPDATE api_keys SET revoked_at = ? WHERE name = ? AND revoked_at IS NULL", (to_db(utcnow()), args.name)
|
||||||
|
)
|
||||||
|
sys.exit(0 if cur.rowcount else f"No active key named {args.name!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def list_jobs(conn, args):
|
||||||
|
for j in repo.list_jobs(conn):
|
||||||
|
print(f"{j['id']:>5} {j['company_name']:<28} {j['job_name']:<40} {j['status'] or '-':<8} last {j['last_run_at'] or 'never'}")
|
||||||
|
|
||||||
|
|
||||||
|
def delete_job(conn, args):
|
||||||
|
cur = conn.execute("DELETE FROM jobs WHERE id = ?", (args.job_id,))
|
||||||
|
sys.exit(0 if cur.rowcount else f"No job with id {args.job_id}")
|
||||||
|
|
||||||
|
|
||||||
|
DEMO = [
|
||||||
|
("Acme Corp", "Veeam B&R", "File Server - Nightly", 24),
|
||||||
|
("Acme Corp", "Veeam B&R", "SQL01 - Transaction Logs", 4),
|
||||||
|
("Bluewater Dental", "Datto", "Workstations - Daily", 24),
|
||||||
|
("Bluewater Dental", "Datto", "Imaging Server - Hourly", 1),
|
||||||
|
("Northfield Logistics", "Acronis", "Exchange - Nightly", 24),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def seed_demo(conn, args):
|
||||||
|
rng = random.Random(42)
|
||||||
|
now = utcnow()
|
||||||
|
for tenant, source, name, every_h in DEMO:
|
||||||
|
ext = name.lower().replace(" ", "-")
|
||||||
|
# Offset so runs don't all line up exactly on the current time
|
||||||
|
start = now - timedelta(days=30) + timedelta(minutes=rng.randint(5, every_h * 60 - 5))
|
||||||
|
while start < now:
|
||||||
|
dur = timedelta(minutes=rng.randint(2, 50 if every_h > 1 else 15))
|
||||||
|
end = start + dur
|
||||||
|
status = "running" if end > now else rng.choices(["success", "warning", "failed"], [90, 7, 3])[0]
|
||||||
|
run = {"status": status, "started_at": start.isoformat()}
|
||||||
|
if status != "running":
|
||||||
|
run |= {"finished_at": end.isoformat(), "size_bytes": rng.randint(10**8, 10**11),
|
||||||
|
"message": {"success": "Job completed successfully.",
|
||||||
|
"warning": "Completed with warnings: 1 file skipped (locked).",
|
||||||
|
"failed": "Error: target repository unreachable."}[status]}
|
||||||
|
repo.ingest(conn, IngestIn.model_validate({
|
||||||
|
"tenant": tenant, "source_system": source,
|
||||||
|
"job": {"external_id": ext, "name": name,
|
||||||
|
"next_run_at": (start + timedelta(hours=every_h)).isoformat()},
|
||||||
|
"run": run,
|
||||||
|
}))
|
||||||
|
start += timedelta(hours=every_h)
|
||||||
|
print(f"Seeded {len(DEMO)} demo jobs")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
|
p.add_argument("--db", type=Path, help="database file (overrides BACKUP_DASH_DB and the env file)")
|
||||||
|
p.add_argument("--create", action="store_true", help="create the database if it doesn't exist")
|
||||||
|
sub = p.add_subparsers(required=True)
|
||||||
|
s = sub.add_parser("create-key"); s.add_argument("name"); s.set_defaults(fn=create_key)
|
||||||
|
s = sub.add_parser("list-keys"); s.set_defaults(fn=list_keys)
|
||||||
|
s = sub.add_parser("revoke-key"); s.add_argument("name"); s.set_defaults(fn=revoke_key)
|
||||||
|
s = sub.add_parser("list-jobs"); s.set_defaults(fn=list_jobs)
|
||||||
|
s = sub.add_parser("delete-job"); s.add_argument("job_id", type=int); s.set_defaults(fn=delete_job)
|
||||||
|
s = sub.add_parser("seed-demo"); s.set_defaults(fn=seed_demo)
|
||||||
|
args = p.parse_args()
|
||||||
|
|
||||||
|
db_path = args.db or load_settings().db_path
|
||||||
|
# Refuse to silently create a fresh DB: that almost always means we're looking in the wrong
|
||||||
|
# place, and anything written there (like a new API key) would be invisible to the service.
|
||||||
|
if not db_path.exists() and not args.create:
|
||||||
|
sys.exit(
|
||||||
|
f"Database not found: {db_path.resolve()}\n"
|
||||||
|
f"Set BACKUP_DASH_DB in {DEFAULT_ENV_FILE} (or pass --db), "
|
||||||
|
"or add --create to start a new database there."
|
||||||
|
)
|
||||||
|
print(f"Using database {db_path.resolve()}", file=sys.stderr)
|
||||||
|
db.init_db(db_path)
|
||||||
|
conn = db.connect(db_path)
|
||||||
|
try:
|
||||||
|
args.fn(conn, args)
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
[pytest]
|
||||||
|
pythonpath = .
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
httpx
|
||||||
|
pytest
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
fastapi>=0.115
|
||||||
|
uvicorn[standard]>=0.30
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
from datetime import timedelta
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from app import db
|
||||||
|
from app.auth import generate_key, hash_key
|
||||||
|
from app.main import create_app
|
||||||
|
from app.settings import Settings
|
||||||
|
from app.timeutil import to_db, utcnow
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client(tmp_path):
|
||||||
|
settings = Settings(db_path=tmp_path / "test.db")
|
||||||
|
with TestClient(create_app(settings)) as c:
|
||||||
|
key = generate_key()
|
||||||
|
conn = db.connect(settings.db_path)
|
||||||
|
conn.execute("INSERT INTO api_keys (name, key_hash, created_at) VALUES ('test', ?, ?)",
|
||||||
|
(hash_key(key), to_db(utcnow())))
|
||||||
|
conn.close()
|
||||||
|
c.headers["Authorization"] = f"Bearer {key}"
|
||||||
|
yield c
|
||||||
|
|
||||||
|
|
||||||
|
def iso(delta: timedelta) -> str:
|
||||||
|
return (utcnow() + delta).isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
def payload(run=None, **job):
|
||||||
|
body = {"tenant": "Acme Corp", "source_system": "Veeam B&R",
|
||||||
|
"job": {"external_id": "job-1", "name": "File Server", **job}}
|
||||||
|
if run is not None:
|
||||||
|
body["run"] = run
|
||||||
|
return body
|
||||||
|
|
||||||
|
|
||||||
|
def test_ingest_requires_valid_key(client):
|
||||||
|
assert client.post("/api/v1/ingest", json=payload(), headers={"Authorization": ""}).status_code == 401
|
||||||
|
assert client.post("/api/v1/ingest", json=payload(), headers={"Authorization": "Bearer nope"}).status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
def test_running_then_finished_merges_into_one_run(client):
|
||||||
|
started = iso(-timedelta(hours=1))
|
||||||
|
r = client.post("/api/v1/ingest", json=payload({"status": "running", "started_at": started}))
|
||||||
|
assert r.status_code == 200, r.text
|
||||||
|
job_id = r.json()["job_id"]
|
||||||
|
|
||||||
|
job = client.get("/api/v1/jobs").json()["jobs"][0]
|
||||||
|
assert job["status"] == "running"
|
||||||
|
assert job["running_since"] is not None
|
||||||
|
assert job["last_run_at"] is None # nothing has finished yet
|
||||||
|
|
||||||
|
finished = iso(-timedelta(minutes=10))
|
||||||
|
client.post("/api/v1/ingest", json=payload(
|
||||||
|
{"status": "warning", "started_at": started, "finished_at": finished, "message": "1 file skipped"}))
|
||||||
|
|
||||||
|
detail = client.get(f"/api/v1/jobs/{job_id}").json()
|
||||||
|
assert detail["status"] == "warning"
|
||||||
|
assert detail["running_since"] is None
|
||||||
|
assert detail["last_run_at"] == to_db(utcnow().fromisoformat(finished))
|
||||||
|
assert detail["duration_seconds"] == 50 * 60
|
||||||
|
assert detail["message"] == "1 file skipped"
|
||||||
|
assert len(client.get(f"/api/v1/jobs/{job_id}/runs").json()["runs"]) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_late_running_report_does_not_reopen_finished_run(client):
|
||||||
|
started = iso(-timedelta(hours=1))
|
||||||
|
client.post("/api/v1/ingest", json=payload({"status": "success", "started_at": started, "finished_at": iso(-timedelta(minutes=5))}))
|
||||||
|
client.post("/api/v1/ingest", json=payload({"status": "running", "started_at": started}))
|
||||||
|
assert client.get("/api/v1/jobs").json()["jobs"][0]["status"] == "success"
|
||||||
|
|
||||||
|
|
||||||
|
def test_last_run_is_previous_finish_while_new_run_in_progress(client):
|
||||||
|
prev_end = iso(-timedelta(hours=23))
|
||||||
|
client.post("/api/v1/ingest", json=payload({"status": "failed", "started_at": iso(-timedelta(hours=24)), "finished_at": prev_end, "message": "boom"}))
|
||||||
|
client.post("/api/v1/ingest", json=payload({"status": "running", "started_at": iso(-timedelta(minutes=5))}))
|
||||||
|
job = client.get("/api/v1/jobs").json()["jobs"][0]
|
||||||
|
assert job["status"] == "running"
|
||||||
|
assert job["last_run_at"] == to_db(utcnow().fromisoformat(prev_end))
|
||||||
|
detail = client.get(f"/api/v1/jobs/{job['id']}").json()
|
||||||
|
assert detail["last_status"] == "failed"
|
||||||
|
assert detail["message"] == "boom"
|
||||||
|
|
||||||
|
|
||||||
|
def test_next_run_at_only_changes_when_sent(client):
|
||||||
|
nxt = iso(timedelta(hours=5))
|
||||||
|
client.post("/api/v1/ingest", json=payload(next_run_at=nxt))
|
||||||
|
client.post("/api/v1/ingest", json=payload()) # omitted → keep
|
||||||
|
assert client.get("/api/v1/jobs").json()["jobs"][0]["next_run_at"] == to_db(utcnow().fromisoformat(nxt))
|
||||||
|
client.post("/api/v1/ingest", json=payload(next_run_at=None)) # explicit null → clear
|
||||||
|
assert client.get("/api/v1/jobs").json()["jobs"][0]["next_run_at"] is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_history_window_and_run_detail(client):
|
||||||
|
for days_ago in (40, 20, 1):
|
||||||
|
start = iso(-timedelta(days=days_ago))
|
||||||
|
client.post("/api/v1/ingest", json=payload({"status": "success", "started_at": start,
|
||||||
|
"finished_at": start, "message": f"{days_ago}d"}))
|
||||||
|
job_id = client.get("/api/v1/jobs").json()["jobs"][0]["id"]
|
||||||
|
runs = client.get(f"/api/v1/jobs/{job_id}/runs").json()["runs"]
|
||||||
|
assert len(runs) == 2 # the 40-day-old run is outside the 30-day window
|
||||||
|
assert "message" not in runs[0]
|
||||||
|
run = client.get(f"/api/v1/runs/{runs[0]['id']}").json()
|
||||||
|
assert run["message"] == "1d"
|
||||||
|
assert client.get(f"/api/v1/jobs/{job_id}/runs?days=31").status_code == 422
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("run", [
|
||||||
|
{"status": "running", "started_at": iso(-timedelta(hours=1)), "finished_at": iso(timedelta(0))},
|
||||||
|
{"status": "success", "started_at": iso(timedelta(0)), "finished_at": iso(-timedelta(hours=1))},
|
||||||
|
{"status": "success", "started_at": "2026-01-01T00:00:00"}, # no timezone
|
||||||
|
{"status": "done", "started_at": iso(timedelta(0))},
|
||||||
|
])
|
||||||
|
def test_rejects_bad_runs(client, run):
|
||||||
|
assert client.post("/api/v1/ingest", json=payload(run)).status_code == 422
|
||||||
|
|
||||||
|
|
||||||
|
def test_tenant_name_is_case_insensitive(client):
|
||||||
|
client.post("/api/v1/ingest", json=payload())
|
||||||
|
body = payload()
|
||||||
|
body["tenant"] = "ACME CORP"
|
||||||
|
client.post("/api/v1/ingest", json=body)
|
||||||
|
assert len(client.get("/api/v1/jobs").json()["jobs"]) == 1
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from app.settings import load_settings
|
||||||
|
|
||||||
|
BACKEND = Path(__file__).resolve().parents[1]
|
||||||
|
|
||||||
|
|
||||||
|
def run_manage(*args, env, cwd):
|
||||||
|
return subprocess.run([sys.executable, str(BACKEND / "manage.py"), *args],
|
||||||
|
capture_output=True, text=True, env=env, cwd=cwd)
|
||||||
|
|
||||||
|
|
||||||
|
def base_env(tmp_path, monkeypatch):
|
||||||
|
env_file = tmp_path / "backup-dashboard.env"
|
||||||
|
monkeypatch.setenv("BACKUP_DASH_ENV_FILE", str(env_file))
|
||||||
|
monkeypatch.delenv("BACKUP_DASH_DB", raising=False)
|
||||||
|
return env_file, {"BACKUP_DASH_ENV_FILE": str(env_file), "PYTHONPATH": str(BACKEND), "PATH": "/usr/bin:/bin"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_file_supplies_db_path_and_real_env_wins(tmp_path, monkeypatch):
|
||||||
|
env_file, _ = base_env(tmp_path, monkeypatch)
|
||||||
|
env_file.write_text('# comment\nBACKUP_DASH_DB="/srv/from-file.db"\nBACKUP_DASH_RETENTION_DAYS=120\n')
|
||||||
|
assert load_settings().db_path == Path("/srv/from-file.db")
|
||||||
|
assert load_settings().retention_days == 120
|
||||||
|
monkeypatch.setenv("BACKUP_DASH_DB", "/srv/from-env.db")
|
||||||
|
assert load_settings().db_path == Path("/srv/from-env.db")
|
||||||
|
|
||||||
|
|
||||||
|
def test_manage_refuses_to_create_db_implicitly(tmp_path, monkeypatch):
|
||||||
|
_, env = base_env(tmp_path, monkeypatch)
|
||||||
|
workdir = tmp_path / "somewhere-else"
|
||||||
|
workdir.mkdir()
|
||||||
|
r = run_manage("create-key", "k", env=env, cwd=workdir)
|
||||||
|
assert r.returncode != 0
|
||||||
|
assert "Database not found" in r.stderr
|
||||||
|
assert not (workdir / "backup-dashboard.db").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_manage_uses_db_from_env_file(tmp_path, monkeypatch):
|
||||||
|
env_file, env = base_env(tmp_path, monkeypatch)
|
||||||
|
service_db = tmp_path / "service.db"
|
||||||
|
env_file.write_text(f"BACKUP_DASH_DB={service_db}\n")
|
||||||
|
assert run_manage("--create", "list-keys", env=env, cwd=tmp_path).returncode == 0 # service's first start
|
||||||
|
workdir = tmp_path / "somewhere-else"
|
||||||
|
workdir.mkdir()
|
||||||
|
r = run_manage("create-key", "veeam-acme", env=env, cwd=workdir)
|
||||||
|
assert r.returncode == 0, r.stderr
|
||||||
|
assert str(service_db) in r.stderr
|
||||||
|
assert "veeam-acme" in run_manage("list-keys", env=env, cwd=tmp_path).stdout
|
||||||
|
assert not (workdir / "backup-dashboard.db").exists()
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Read-only dump of what the MARS agent's cmdlets return, for troubleshooting the collector.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
The collector reads properties of Get-OBJob / Get-OBPolicy / Get-OBSchedule results whose names
|
||||||
|
aren't documented. This prints those objects' real structure (property names, types and values,
|
||||||
|
a few levels deep) so mismatches can be fixed. Nothing is changed. Output is also saved to a
|
||||||
|
text file.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Get-MarsCollectorDiagnostics.ps1
|
||||||
|
#>
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[int]$Jobs = 3,
|
||||||
|
[string]$OutFile = (Join-Path $env:TEMP 'mars-collector-diagnostics.txt')
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
$MaxDepth = 6
|
||||||
|
|
||||||
|
if (-not (Get-Command Get-OBJob -ErrorAction SilentlyContinue)) {
|
||||||
|
if (Get-Module -ListAvailable -Name MSOnlineBackup) {
|
||||||
|
Import-Module MSOnlineBackup
|
||||||
|
} else {
|
||||||
|
Import-Module (Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\Modules\MSOnlineBackup')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function Test-Scalar($Value) {
|
||||||
|
return $null -eq $Value -or $Value -is [string] -or $Value -is [ValueType] -or $Value -is [guid]
|
||||||
|
}
|
||||||
|
|
||||||
|
# Recursive "name (type) = value" listing. Collections show their first 3 items.
|
||||||
|
function Show-Object($Value, [string]$Indent = ' ', [int]$Depth = 0) {
|
||||||
|
if (Test-Scalar $Value) { return }
|
||||||
|
if ($Depth -ge $MaxDepth) { "$Indent..."; return }
|
||||||
|
if ($Value -is [Collections.IEnumerable]) {
|
||||||
|
$items = @($Value)
|
||||||
|
"$Indent[$($items.Count) item(s)]"
|
||||||
|
$i = 0
|
||||||
|
foreach ($item in $items | Select-Object -First 3) {
|
||||||
|
"$Indent[$i] ($($item.GetType().FullName))$(if (Test-Scalar $item) { " = $item" })"
|
||||||
|
Show-Object $item "$Indent " ($Depth + 1)
|
||||||
|
$i++
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
foreach ($p in $Value.PSObject.Properties) {
|
||||||
|
try { $v = $p.Value } catch { "$Indent$($p.Name) = <error: $($_.Exception.Message)>"; continue }
|
||||||
|
$type = if ($null -eq $v) { 'null' } else { $v.GetType().FullName }
|
||||||
|
if (Test-Scalar $v) {
|
||||||
|
"$Indent$($p.Name) ($type) = $v"
|
||||||
|
} else {
|
||||||
|
$text = [string]$v
|
||||||
|
if ($text.Length -gt 100) { $text = $text.Substring(0, 97) + '...' }
|
||||||
|
"$Indent$($p.Name) ($type) = $text"
|
||||||
|
Show-Object $v "$Indent " ($Depth + 1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function Show-Section([string]$Title, [scriptblock]$Get) {
|
||||||
|
"== $Title"
|
||||||
|
try {
|
||||||
|
$result = @(& $Get)
|
||||||
|
if ($result.Count -eq 0 -or ($result.Count -eq 1 -and $null -eq $result[0])) { ' (nothing returned)' }
|
||||||
|
$n = 0
|
||||||
|
foreach ($item in $result) {
|
||||||
|
if ($null -eq $item) { continue }
|
||||||
|
" [$n] ($($item.GetType().FullName))"
|
||||||
|
Show-Object $item ' '
|
||||||
|
$n++
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
" error: $($_.Exception.Message)"
|
||||||
|
}
|
||||||
|
''
|
||||||
|
}
|
||||||
|
|
||||||
|
$report = & {
|
||||||
|
$agent = try {
|
||||||
|
(Get-Item (Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\cbengine.exe')).VersionInfo.ProductVersion
|
||||||
|
} catch { 'unknown' }
|
||||||
|
"Generated $(Get-Date -Format o) on $env:COMPUTERNAME, MARS agent $agent, PowerShell $($PSVersionTable.PSVersion)"
|
||||||
|
''
|
||||||
|
Show-Section 'Get-OBJob (in progress)' { Get-OBJob }
|
||||||
|
Show-Section "Get-OBJob -Previous $Jobs" { Get-OBJob -Previous $Jobs }
|
||||||
|
Show-Section 'Get-OBPolicy' { Get-OBPolicy }
|
||||||
|
Show-Section 'Get-OBPolicy | Get-OBSchedule' { Get-OBSchedule -Policy (Get-OBPolicy) }
|
||||||
|
if (Get-Command Get-OBSystemStatePolicy -ErrorAction SilentlyContinue) {
|
||||||
|
Show-Section 'Get-OBSystemStatePolicy | Get-OBSchedule' {
|
||||||
|
$ssPolicy = Get-OBSystemStatePolicy
|
||||||
|
if ($ssPolicy) { Get-OBSchedule -Policy $ssPolicy } else { '(no System State policy configured)' }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} | Out-String -Width 300
|
||||||
|
|
||||||
|
$report
|
||||||
|
$report | Set-Content -Path $OutFile -Encoding UTF8
|
||||||
|
Write-Host "Saved to $OutFile"
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Installs the MARS (Azure Backup agent) collector as a SYSTEM scheduled task.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
- Copies Send-MarsBackupStatus.ps1 to Program Files (only admins can modify it, which
|
||||||
|
matters because it runs as SYSTEM).
|
||||||
|
- Writes the config to ProgramData\BackupDashboard and restricts that folder to SYSTEM and
|
||||||
|
Administrators, since it contains the API key.
|
||||||
|
- Registers a scheduled task that runs every IntervalMinutes under Windows PowerShell 5.1
|
||||||
|
(powershell.exe), which the agent's MSOnlineBackup module targets.
|
||||||
|
Safe to re-run to change settings or update the script. Can sit alongside the Veeam collector.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Install-MarsCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||||
|
(prompts for the API key)
|
||||||
|
#>
|
||||||
|
#Requires -RunAsAdministrator
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[Parameter(Mandatory)][string]$ApiUrl,
|
||||||
|
[Parameter(Mandatory)][string]$Tenant,
|
||||||
|
# Name shown on the dashboard; defaults to "<server> - Azure Backup"
|
||||||
|
[string]$JobName,
|
||||||
|
[string]$SourceSystem = 'Azure Backup (MARS)',
|
||||||
|
[ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
|
||||||
|
[ValidateRange(1, 720)][int]$LookbackHours = 48,
|
||||||
|
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard')
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
$TaskName = 'Backup Dashboard - MARS collector'
|
||||||
|
$DataDir = Join-Path $env:ProgramData 'BackupDashboard'
|
||||||
|
$ConfigPath = Join-Path $DataDir 'mars-collector.json'
|
||||||
|
$WindowsPowerShell = Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\powershell.exe'
|
||||||
|
|
||||||
|
# Check the agent is here before changing anything
|
||||||
|
$agentModule = Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\Modules\MSOnlineBackup'
|
||||||
|
if (-not (Get-Module -ListAvailable -Name MSOnlineBackup) -and -not (Test-Path $agentModule)) {
|
||||||
|
throw 'The MARS agent (MSOnlineBackup PowerShell module) is not installed on this server.'
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($ApiUrl -notmatch '^https://') {
|
||||||
|
Write-Warning 'ApiUrl is not HTTPS: the API key would be sent in clear text.'
|
||||||
|
}
|
||||||
|
|
||||||
|
# Read the key as a SecureString so it isn't echoed to the console or kept in shell history
|
||||||
|
$secure = Read-Host -AsSecureString 'API key (from: python manage.py create-key <name>)'
|
||||||
|
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
|
||||||
|
try { $apiKey = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) }
|
||||||
|
finally { [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr) }
|
||||||
|
if (-not $apiKey.StartsWith('bdk_')) { throw "That doesn't look like a dashboard API key (expected 'bdk_...')." }
|
||||||
|
|
||||||
|
# Config folder: SYSTEM + Administrators only (SIDs so this works on non-English Windows)
|
||||||
|
New-Item -ItemType Directory -Path $DataDir -Force | Out-Null
|
||||||
|
& icacls.exe $DataDir /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' | Out-Null
|
||||||
|
if ($LASTEXITCODE) { throw "icacls failed on $DataDir" }
|
||||||
|
|
||||||
|
$config = [ordered]@{
|
||||||
|
ApiUrl = $ApiUrl.TrimEnd('/')
|
||||||
|
ApiKey = $apiKey
|
||||||
|
Tenant = $Tenant
|
||||||
|
SourceSystem = $SourceSystem
|
||||||
|
LookbackHours = $LookbackHours
|
||||||
|
}
|
||||||
|
if ($JobName) { $config.JobName = $JobName }
|
||||||
|
$config | ConvertTo-Json | Set-Content -Path $ConfigPath -Encoding UTF8
|
||||||
|
|
||||||
|
New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null
|
||||||
|
$scriptPath = Join-Path $InstallDir 'Send-MarsBackupStatus.ps1'
|
||||||
|
Copy-Item -Path (Join-Path $PSScriptRoot 'Send-MarsBackupStatus.ps1') -Destination $scriptPath -Force
|
||||||
|
Unblock-File -Path $scriptPath
|
||||||
|
|
||||||
|
$action = New-ScheduledTaskAction -Execute $WindowsPowerShell `
|
||||||
|
-Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$scriptPath`""
|
||||||
|
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date).AddMinutes(1) `
|
||||||
|
-RepetitionInterval (New-TimeSpan -Minutes $IntervalMinutes)
|
||||||
|
$principal = New-ScheduledTaskPrincipal -UserId 'S-1-5-18' -LogonType ServiceAccount -RunLevel Highest
|
||||||
|
$settings = New-ScheduledTaskSettingsSet -ExecutionTimeLimit (New-TimeSpan -Minutes 30) `
|
||||||
|
-MultipleInstances IgnoreNew -StartWhenAvailable
|
||||||
|
Register-ScheduledTask -TaskName $TaskName -Action $action -Trigger $trigger -Principal $principal `
|
||||||
|
-Settings $settings -Description 'Sends Azure Backup (MARS) results to the Backup Dashboard' -Force | Out-Null
|
||||||
|
|
||||||
|
Write-Host "Installed. Config: $ConfigPath"
|
||||||
|
Write-Host "Task '$TaskName' runs every $IntervalMinutes minutes as SYSTEM, using Windows PowerShell."
|
||||||
|
Write-Host ''
|
||||||
|
Write-Host 'Next steps:'
|
||||||
|
Write-Host " 1. Preview what will be sent: & '$WindowsPowerShell' -File '$scriptPath' -DryRun"
|
||||||
|
Write-Host " 2. Backfill 30 days of history: & '$WindowsPowerShell' -File '$scriptPath' -LookbackHours 720"
|
||||||
|
Write-Host " 3. Check the log: Get-Content '$DataDir\mars-collector.log' -Tail 20"
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
# Azure Backup (MARS agent) collector
|
||||||
|
|
||||||
|
A PowerShell script that runs on each server with the Microsoft Azure Recovery Services (MARS)
|
||||||
|
agent and reports its backups to the dashboard. Each server appears as one job,
|
||||||
|
named `<SERVER> - Azure Backup` by default.
|
||||||
|
|
||||||
|
> **Checked against:** MARS agent 2.0.9274, a Files & Folders policy with completed backups only.
|
||||||
|
> Not yet seen on a real agent: a failed backup, a backup in progress, or a System State policy.
|
||||||
|
> Those paths are written to the same object model, but if one looks wrong, run
|
||||||
|
> `Get-MarsCollectorDiagnostics.ps1` while it's in that state and send the output.
|
||||||
|
|
||||||
|
## Install (on the server, as an administrator)
|
||||||
|
|
||||||
|
Runs under Windows PowerShell 5.1 (`powershell.exe`), which the agent's `MSOnlineBackup` module
|
||||||
|
targets and which every supported Windows Server has.
|
||||||
|
|
||||||
|
1. On the dashboard host, create a key for this server:
|
||||||
|
```bash
|
||||||
|
python manage.py create-key mars-acme-fs01
|
||||||
|
```
|
||||||
|
2. Copy this folder to the server and run:
|
||||||
|
```powershell
|
||||||
|
.\Install-MarsCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||||
|
```
|
||||||
|
Add `-JobName "File Server"` to choose the name shown on the dashboard. It prompts for the
|
||||||
|
key, then:
|
||||||
|
- writes `C:\ProgramData\BackupDashboard\mars-collector.json` (readable only by SYSTEM and Administrators)
|
||||||
|
- copies the script to `C:\Program Files\BackupDashboard\`
|
||||||
|
- registers the scheduled task **Backup Dashboard - MARS collector** (SYSTEM, every 60 minutes)
|
||||||
|
3. Preview the payloads, then backfill 30 days so the history view is populated straight away:
|
||||||
|
```powershell
|
||||||
|
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-MarsBackupStatus.ps1' -DryRun
|
||||||
|
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-MarsBackupStatus.ps1' -LookbackHours 720
|
||||||
|
```
|
||||||
|
|
||||||
|
It can be installed alongside the Veeam collector: it uses its own config, state, log and task.
|
||||||
|
Re-run the installer to change settings or update the script.
|
||||||
|
|
||||||
|
## What it sends
|
||||||
|
|
||||||
|
| Dashboard | MARS |
|
||||||
|
|---|---|
|
||||||
|
| Company | `Tenant` from the config |
|
||||||
|
| Job | One per server, ID `mars:<computer name>`, so changing `JobName` renames rather than duplicates it |
|
||||||
|
| Status | Backup job state: Failed → **Failed**. Completed with failed bytes or an error → **Warning**. Aborted/cancelled or unrecognised → **Warning**. In progress → **Running**. Restore jobs are ignored |
|
||||||
|
| Started / finished | Job start / end time. MARS reports these in UTC without marking them as UTC; the collector handles that |
|
||||||
|
| Next scheduled | Next slot from the backup policy schedule, or the System State policy's if that's sooner. Empty (so never Overdue) for schedules that run every 2+ weeks, since which week is next can't be told |
|
||||||
|
| Message | Job and per-datasource errors (non-zero error codes), failed file counts, and the failed-file log path |
|
||||||
|
| Size | Bytes transferred, summed across datasources |
|
||||||
|
|
||||||
|
Each run covers the last 48 hours (`LookbackHours`). Jobs already delivered are recorded in
|
||||||
|
`mars-collector.state.json`.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
- Log: `C:\ProgramData\BackupDashboard\mars-collector.log`. Exit codes: `0` OK, `1` some runs
|
||||||
|
failed to send (retried next run), `2` setup failed (config or MARS module).
|
||||||
|
- Wrong status, sizes or messages: run `.\Get-MarsCollectorDiagnostics.ps1` (read-only). It prints
|
||||||
|
the real structure of the objects the agent's cmdlets return.
|
||||||
|
|
||||||
|
## Testing without MARS
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
. ./tests/MockMars.ps1
|
||||||
|
./Send-MarsBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||||
|
```
|
||||||
@@ -0,0 +1,413 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Reports Microsoft Azure Recovery Services (MARS) agent backups to the Backup Dashboard.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
Runs on a server with the MARS agent, normally as a SYSTEM scheduled task every 60 minutes
|
||||||
|
(see Install-MarsCollector.ps1). The server appears on the dashboard as one job. Each run sends:
|
||||||
|
- the next scheduled backup, worked out from the backup (and System State) policy schedules,
|
||||||
|
which drives the dashboard's Overdue flag, and
|
||||||
|
- every backup job from the last LookbackHours that the dashboard hasn't already received in
|
||||||
|
its final state. A backup in progress is sent as Running until it finishes.
|
||||||
|
|
||||||
|
Already-reported jobs are tracked in a state file, and the ingest API is idempotent, so a
|
||||||
|
failed or repeated run is harmless: anything not confirmed is retried next time.
|
||||||
|
|
||||||
|
Written for Windows PowerShell 5.1, which the MARS agent's MSOnlineBackup module targets.
|
||||||
|
|
||||||
|
.PARAMETER ConfigPath
|
||||||
|
JSON config with ApiUrl, ApiKey and Tenant (optional: JobName, SourceSystem, LookbackHours).
|
||||||
|
|
||||||
|
.PARAMETER LookbackHours
|
||||||
|
Overrides the config value. Use 720 once to backfill 30 days of history.
|
||||||
|
|
||||||
|
.PARAMETER DryRun
|
||||||
|
Print the payloads instead of sending them. Doesn't touch the state or log files.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Send-MarsBackupStatus.ps1 -DryRun
|
||||||
|
.EXAMPLE
|
||||||
|
.\Send-MarsBackupStatus.ps1 -LookbackHours 720
|
||||||
|
#>
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[string]$ConfigPath = (Join-Path $env:ProgramData 'BackupDashboard\mars-collector.json'),
|
||||||
|
[int]$LookbackHours = 0,
|
||||||
|
[switch]$DryRun
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
|
||||||
|
$DataDir = Split-Path -Parent $ConfigPath
|
||||||
|
$LogPath = Join-Path $DataDir 'mars-collector.log'
|
||||||
|
$StatePath = Join-Path $DataDir 'mars-collector.state.json'
|
||||||
|
$MaxLogBytes = 5MB
|
||||||
|
# MARS keeps a limited job history; this comfortably covers 30 days at 3 backups a day
|
||||||
|
$MaxPreviousJobs = 200
|
||||||
|
$Invariant = [Globalization.CultureInfo]::InvariantCulture
|
||||||
|
|
||||||
|
# ---------- Helpers ----------
|
||||||
|
|
||||||
|
function Write-Log {
|
||||||
|
param([string]$Message, [string]$Level = 'INFO')
|
||||||
|
$line = '{0} [{1}] {2}' -f (Get-Date).ToString('yyyy-MM-dd HH:mm:ss'), $Level, $Message
|
||||||
|
Write-Host $line
|
||||||
|
if ($DryRun) { return }
|
||||||
|
if ((Test-Path $LogPath) -and (Get-Item $LogPath).Length -gt $MaxLogBytes) {
|
||||||
|
Move-Item $LogPath "$LogPath.1" -Force
|
||||||
|
}
|
||||||
|
Add-Content -Path $LogPath -Value $line -Encoding UTF8
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-ErrorText($ErrorRecord) {
|
||||||
|
# Windows PowerShell puts the HTTP response body (e.g. the API's 422 explanation) here
|
||||||
|
if ($ErrorRecord.ErrorDetails -and $ErrorRecord.ErrorDetails.Message) {
|
||||||
|
return "$($ErrorRecord.Exception.Message) $($ErrorRecord.ErrorDetails.Message)"
|
||||||
|
}
|
||||||
|
return $ErrorRecord.Exception.Message
|
||||||
|
}
|
||||||
|
|
||||||
|
# The API needs an explicit UTC offset. DateTimeOffset keeps a Local/Unspecified time as local and a
|
||||||
|
# Utc time as +00:00.
|
||||||
|
function ConvertTo-IsoTime($Value) {
|
||||||
|
if ($null -eq $Value) { return $null }
|
||||||
|
$dt = [datetime]$Value
|
||||||
|
if ($dt.Year -le 1900) { return $null } # "not set" (MARS uses 1601-01-01)
|
||||||
|
return ([DateTimeOffset]$dt).ToString("yyyy-MM-dd'T'HH:mm:sszzz", $Invariant)
|
||||||
|
}
|
||||||
|
|
||||||
|
# First property of $Object, out of $Names, that exists and isn't empty. MARS object property names
|
||||||
|
# aren't documented, so reads go through this rather than assuming one spelling.
|
||||||
|
function Get-FirstValue($Object, [string[]]$Names) {
|
||||||
|
if ($null -eq $Object) { return $null }
|
||||||
|
foreach ($name in $Names) {
|
||||||
|
try {
|
||||||
|
$prop = $Object.PSObject.Properties[$name]
|
||||||
|
if (-not $prop) { continue }
|
||||||
|
$value = $prop.Value
|
||||||
|
# Check emptiness by type: casting to [string] would make a one-item list of objects ''
|
||||||
|
if ($null -eq $value) { continue }
|
||||||
|
if ($value -is [string] -and $value -eq '') { continue }
|
||||||
|
if ($value -is [Collections.ICollection] -and $value.Count -eq 0) { continue }
|
||||||
|
return $value
|
||||||
|
} catch { }
|
||||||
|
}
|
||||||
|
return $null
|
||||||
|
}
|
||||||
|
|
||||||
|
function Import-Mars {
|
||||||
|
if (Get-Command Get-OBJob -ErrorAction SilentlyContinue) { return } # already loaded
|
||||||
|
if (Get-Module -ListAvailable -Name MSOnlineBackup) {
|
||||||
|
Import-Module MSOnlineBackup
|
||||||
|
return
|
||||||
|
}
|
||||||
|
# The agent normally adds its module folder to PSModulePath; fall back to the default install path
|
||||||
|
$agentModule = Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\Modules\MSOnlineBackup'
|
||||||
|
Import-Module $agentModule
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------- Schedule ----------
|
||||||
|
|
||||||
|
# Next slot strictly after $After for a MARS schedule (days of week + times of day).
|
||||||
|
# Returns $null when there's no schedule, or when it runs every N>1 weeks: we can't tell which
|
||||||
|
# week is next, and a wrong guess would raise a false Overdue alarm.
|
||||||
|
function Get-NextSlot($Schedule, [datetime]$After) {
|
||||||
|
if ($null -eq $Schedule) { return $null }
|
||||||
|
$weeks = Get-FirstValue $Schedule @('ScheduleWeeklyFrequency', 'WeeksInterval')
|
||||||
|
if ($weeks -and [int]$weeks -gt 1) { return $null }
|
||||||
|
|
||||||
|
$days = @(Get-FirstValue $Schedule @('ScheduleRunDays', 'DaysOfWeek') | ForEach-Object { [string]$_ })
|
||||||
|
# TimeSpans (time of day, local) on real agents; accept DateTimes too
|
||||||
|
$times = @(Get-FirstValue $Schedule @('ScheduleRunTimes', 'TimesOfDay') | ForEach-Object {
|
||||||
|
if ($_ -is [timespan]) { $_ } else { ([datetime]$_).TimeOfDay }
|
||||||
|
} | Sort-Object)
|
||||||
|
if ($days.Count -eq 0 -or $times.Count -eq 0) { return $null }
|
||||||
|
|
||||||
|
for ($offset = 0; $offset -le 7; $offset++) {
|
||||||
|
$date = $After.Date.AddDays($offset)
|
||||||
|
if ($days -notcontains [string]$date.DayOfWeek) { continue }
|
||||||
|
foreach ($time in $times) {
|
||||||
|
$candidate = $date + $time
|
||||||
|
if ($candidate -gt $After) { return $candidate }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return $null
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-NextRun {
|
||||||
|
$next = $null
|
||||||
|
$policies = @()
|
||||||
|
try { $policies += @(Get-OBPolicy) } catch { Write-Log "Couldn't read the backup policy: $(Get-ErrorText $_)" 'WARN' }
|
||||||
|
# Windows Server can have a separate System State policy with its own schedule
|
||||||
|
if (Get-Command Get-OBSystemStatePolicy -ErrorAction SilentlyContinue) {
|
||||||
|
try { $policies += @(Get-OBSystemStatePolicy) } catch { }
|
||||||
|
}
|
||||||
|
foreach ($policy in $policies) {
|
||||||
|
if ($null -eq $policy) { continue }
|
||||||
|
try {
|
||||||
|
$slot = Get-NextSlot (Get-OBSchedule -Policy $policy) (Get-Date)
|
||||||
|
if ($slot -and (-not $next -or $slot -lt $next)) { $next = $slot }
|
||||||
|
} catch {
|
||||||
|
Write-Log "Couldn't read a policy schedule: $(Get-ErrorText $_)" 'WARN'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ConvertTo-IsoTime $next
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------- Jobs ----------
|
||||||
|
|
||||||
|
# A job's StartTime/EndTime are UTC values, though not necessarily marked as UTC: on a real agent
|
||||||
|
# StartTime matched the "StartTimeUTC" in the status text, while DisplayTime held local time.
|
||||||
|
# Returns $null for "not set" (1601-01-01), e.g. EndTime while a backup is running.
|
||||||
|
function Get-JobTimeUtc($Job, [string]$Field) {
|
||||||
|
$value = Get-FirstValue $Job.JobStatus @($Field)
|
||||||
|
if ($null -eq $value) { return $null }
|
||||||
|
$dt = [datetime]$value
|
||||||
|
if ($dt.Year -le 1900) { return $null }
|
||||||
|
return [datetime]::SpecifyKind($dt, [DateTimeKind]::Utc)
|
||||||
|
}
|
||||||
|
|
||||||
|
# MARS always includes an ErrorInfo; ErrorCode 0 means no error
|
||||||
|
function Test-HasError($ErrorInfo) {
|
||||||
|
if (-not $ErrorInfo) { return $false }
|
||||||
|
$code = Get-FirstValue $ErrorInfo @('ErrorCode')
|
||||||
|
$detailed = Get-FirstValue $ErrorInfo @('DetailedErrorCode')
|
||||||
|
return ($code -and [long]$code -ne 0) -or ($detailed -and [long]$detailed -ne 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
# DatasourceStatus.Datasource is an object whose DataSourceName is e.g. "E:\" or "System State"
|
||||||
|
function Get-DatasourceName($DatasourceStatus) {
|
||||||
|
$ds = Get-FirstValue $DatasourceStatus @('Datasource', 'DataSource')
|
||||||
|
$name = Get-FirstValue $ds @('DataSourceName', 'FriendlyNameDSPS', 'Name')
|
||||||
|
if ($name) { return [string]$name }
|
||||||
|
if ($ds -is [string]) { return $ds }
|
||||||
|
return 'Datasource'
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-FailedFiles($Job) {
|
||||||
|
$sum = [long]0
|
||||||
|
foreach ($ds in Get-DatasourceStatuses $Job) {
|
||||||
|
$value = Get-FirstValue (Get-FirstValue $ds @('FileProgress')) @('Failed')
|
||||||
|
if ($value) { $sum += [long]$value }
|
||||||
|
}
|
||||||
|
return $sum
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-JobStatus($Job) {
|
||||||
|
$status = $Job.JobStatus
|
||||||
|
$state = [string](Get-FirstValue $status @('JobState', 'State'))
|
||||||
|
if (-not (Get-JobTimeUtc $Job 'EndTime') -and $state -notmatch 'Fail|Abort|Cancel|Complete|Success|Succeed|Warn') {
|
||||||
|
return 'running'
|
||||||
|
}
|
||||||
|
if ($state -match 'Fail') { return 'failed' }
|
||||||
|
if ($state -match 'Warn|Abort|Cancel|Stop') { return 'warning' }
|
||||||
|
if ($state -match 'Complete|Success|Succeed') {
|
||||||
|
# MARS can report Completed while individual files failed or an error was recorded
|
||||||
|
if ((Get-FailedBytes $Job) -gt 0 -or (Get-FailedFiles $Job) -gt 0 -or
|
||||||
|
(Test-HasError (Get-FirstValue $status @('ErrorInfo')))) { return 'warning' }
|
||||||
|
return 'success'
|
||||||
|
}
|
||||||
|
return 'warning' # finished in a state we don't recognise: surface it rather than hide it
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-DatasourceStatuses($Job) {
|
||||||
|
return @(Get-FirstValue $Job.JobStatus @('DatasourceStatus', 'DataSourceStatus') | Where-Object { $_ })
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-ByteTotal($Job, [string]$Field) {
|
||||||
|
$sum = [long]0
|
||||||
|
foreach ($ds in Get-DatasourceStatuses $Job) {
|
||||||
|
$progress = Get-FirstValue $ds @('ByteProgress')
|
||||||
|
$value = Get-FirstValue $progress @($Field)
|
||||||
|
if ($value) { $sum += [long]$value }
|
||||||
|
}
|
||||||
|
return $sum
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-FailedBytes($Job) { return Get-ByteTotal $Job 'Failed' }
|
||||||
|
|
||||||
|
function Format-ErrorInfo($ErrorInfo) {
|
||||||
|
if (-not (Test-HasError $ErrorInfo)) { return $null }
|
||||||
|
$text = Get-FirstValue $ErrorInfo @('ErrorMessage', 'Message', 'Description')
|
||||||
|
if ($text) { return ([string]$text).Trim() }
|
||||||
|
$code = Get-FirstValue $ErrorInfo @('ErrorCode', 'DetailedErrorCode')
|
||||||
|
$params = @(Get-FirstValue $ErrorInfo @('ErrorParamList') | Where-Object { $_ } | ForEach-Object {
|
||||||
|
$name = Get-FirstValue $_ @('Name')
|
||||||
|
$value = Get-FirstValue $_ @('Value')
|
||||||
|
if ($name) { "$name=$value" } else { [string]$_ }
|
||||||
|
} | Where-Object { $_ })
|
||||||
|
$line = if ($code) { "Error code $code" } else { [string]$ErrorInfo }
|
||||||
|
if ($params.Count) { $line += " ($($params -join ', '))" }
|
||||||
|
return $line
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-RunMessage($Job, [string]$Status) {
|
||||||
|
$lines = New-Object 'System.Collections.Generic.List[string]'
|
||||||
|
$state = [string](Get-FirstValue $Job.JobStatus @('JobState', 'State'))
|
||||||
|
$jobError = Format-ErrorInfo (Get-FirstValue $Job.JobStatus @('ErrorInfo'))
|
||||||
|
if ($jobError) { $lines.Add($jobError) }
|
||||||
|
|
||||||
|
foreach ($ds in Get-DatasourceStatuses $Job) {
|
||||||
|
$name = Get-DatasourceName $ds
|
||||||
|
$dsError = Format-ErrorInfo (Get-FirstValue $ds @('ErrorInfo'))
|
||||||
|
$failedFiles = Get-FirstValue (Get-FirstValue $ds @('FileProgress')) @('Failed')
|
||||||
|
$failedBytes = Get-FirstValue (Get-FirstValue $ds @('ByteProgress')) @('Failed')
|
||||||
|
if ($dsError) { $lines.Add("${name}: $dsError") }
|
||||||
|
if ($failedFiles -and [long]$failedFiles -gt 0) { $lines.Add("${name}: $failedFiles file(s) failed to back up") }
|
||||||
|
elseif ($failedBytes -and [long]$failedBytes -gt 0) { $lines.Add("${name}: $failedBytes bytes failed to back up") }
|
||||||
|
}
|
||||||
|
$failedLog = Get-FirstValue $Job.JobStatus @('FailedFileLog')
|
||||||
|
if ($failedLog -and $lines.Count) { $lines.Add("Failed file log: $failedLog") }
|
||||||
|
|
||||||
|
if ($lines.Count -eq 0) {
|
||||||
|
switch ($Status) {
|
||||||
|
'success' { return 'Backup completed successfully.' }
|
||||||
|
'running' { return '' }
|
||||||
|
default { return "Backup finished with state '$state'." }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if ($state -and $Status -ne 'success') { $lines.Insert(0, "Job state: $state") }
|
||||||
|
return ($lines -join "`n")
|
||||||
|
}
|
||||||
|
|
||||||
|
function ConvertTo-Run($Job) {
|
||||||
|
$status = Get-JobStatus $Job
|
||||||
|
$datasources = @(Get-DatasourceStatuses $Job | ForEach-Object { Get-DatasourceName $_ })
|
||||||
|
|
||||||
|
$details = [ordered]@{ server = $env:COMPUTERNAME }
|
||||||
|
if ($datasources.Count) { $details.datasources = $datasources }
|
||||||
|
$agentState = Get-FirstValue $Job.JobStatus @('JobState', 'State')
|
||||||
|
if ($agentState) { $details.mars_state = [string]$agentState }
|
||||||
|
|
||||||
|
$run = [ordered]@{
|
||||||
|
external_id = [string]$Job.JobId
|
||||||
|
status = $status
|
||||||
|
started_at = ConvertTo-IsoTime (Get-JobTimeUtc $Job 'StartTime')
|
||||||
|
message = Get-RunMessage $Job $status
|
||||||
|
details = $details
|
||||||
|
}
|
||||||
|
if ($status -ne 'running') {
|
||||||
|
$finished = ConvertTo-IsoTime (Get-JobTimeUtc $Job 'EndTime')
|
||||||
|
if ($finished) { $run.finished_at = $finished } # otherwise the API uses receipt time
|
||||||
|
$transferred = Get-ByteTotal $Job 'Progress'
|
||||||
|
if ($transferred -gt 0) { $run.size_bytes = $transferred }
|
||||||
|
$failedBytes = Get-FailedBytes $Job
|
||||||
|
if ($failedBytes -gt 0) { $details.failed_bytes = $failedBytes }
|
||||||
|
$failedFiles = Get-FailedFiles $Job
|
||||||
|
if ($failedFiles -gt 0) { $details.failed_files = $failedFiles }
|
||||||
|
}
|
||||||
|
return $run
|
||||||
|
}
|
||||||
|
|
||||||
|
function Test-IsBackupJob($Job) {
|
||||||
|
# Get-OBJob also lists restores (JobType Recovery); only backups belong on the dashboard
|
||||||
|
$type = [string](Get-FirstValue $Job @('JobType', 'Type'))
|
||||||
|
return (-not $type) -or $type -match 'Backup'
|
||||||
|
}
|
||||||
|
|
||||||
|
function Send-Report($Body) {
|
||||||
|
$json = $Body | ConvertTo-Json -Depth 6 -Compress
|
||||||
|
if ($DryRun) {
|
||||||
|
Write-Host $json
|
||||||
|
return
|
||||||
|
}
|
||||||
|
# Send bytes: Windows PowerShell would otherwise encode a string body as Latin-1
|
||||||
|
$params = @{
|
||||||
|
Uri = "$($Config.ApiUrl.TrimEnd('/'))/api/v1/ingest"
|
||||||
|
Method = 'Post'
|
||||||
|
Headers = @{ Authorization = "Bearer $($Config.ApiKey)" }
|
||||||
|
ContentType = 'application/json; charset=utf-8'
|
||||||
|
Body = [Text.Encoding]::UTF8.GetBytes($json)
|
||||||
|
TimeoutSec = 30
|
||||||
|
}
|
||||||
|
Invoke-RestMethod @params | Out-Null
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------- Main ----------
|
||||||
|
|
||||||
|
try {
|
||||||
|
$Config = Get-Content -Path $ConfigPath -Raw | ConvertFrom-Json
|
||||||
|
foreach ($key in 'ApiUrl', 'ApiKey', 'Tenant') {
|
||||||
|
if (-not $Config.$key) { throw "Config $ConfigPath is missing '$key'" }
|
||||||
|
}
|
||||||
|
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Azure Backup (MARS)' }
|
||||||
|
$JobName = if ($Config.JobName) { [string]$Config.JobName } else { "$env:COMPUTERNAME - Azure Backup" }
|
||||||
|
if ($LookbackHours -le 0) {
|
||||||
|
$LookbackHours = if ($Config.LookbackHours) { [int]$Config.LookbackHours } else { 48 }
|
||||||
|
}
|
||||||
|
|
||||||
|
# Windows PowerShell 5.1 may default to TLS 1.0
|
||||||
|
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
|
||||||
|
|
||||||
|
Import-Mars
|
||||||
|
|
||||||
|
# Job ID -> start time (Unix seconds), for jobs already delivered in their final state
|
||||||
|
$sent = @{}
|
||||||
|
if (Test-Path $StatePath) {
|
||||||
|
$saved = Get-Content -Path $StatePath -Raw | ConvertFrom-Json
|
||||||
|
if ($saved) { foreach ($p in $saved.PSObject.Properties) { $sent[$p.Name] = [long]$p.Value } }
|
||||||
|
}
|
||||||
|
|
||||||
|
$sinceUtc = (Get-Date).ToUniversalTime().AddHours(-$LookbackHours)
|
||||||
|
# Get-OBJob with no arguments lists jobs in progress; -Previous lists finished ones
|
||||||
|
$jobs = @(@(Get-OBJob) + @(Get-OBJob -Previous $MaxPreviousJobs) | Where-Object {
|
||||||
|
$_ -and (Test-IsBackupJob $_)
|
||||||
|
})
|
||||||
|
$byId = [ordered]@{}
|
||||||
|
foreach ($job in $jobs) {
|
||||||
|
$start = Get-JobTimeUtc $job 'StartTime'
|
||||||
|
if (-not $start -or $start -lt $sinceUtc) { continue }
|
||||||
|
$byId[[string]$job.JobId] = $job # a job in both lists: the finished entry (later) wins
|
||||||
|
}
|
||||||
|
$pending = @($byId.Values |
|
||||||
|
Where-Object { -not $sent.ContainsKey([string]$_.JobId) } |
|
||||||
|
Sort-Object { Get-JobTimeUtc $_ 'StartTime' })
|
||||||
|
|
||||||
|
$jobInfo = [ordered]@{
|
||||||
|
# Stable per server, so renaming the dashboard job via JobName doesn't create a new one
|
||||||
|
external_id = "mars:$($env:COMPUTERNAME.ToLowerInvariant())"
|
||||||
|
name = $JobName
|
||||||
|
# Always sent (null when unscheduled) so the dashboard's value tracks the agent's
|
||||||
|
next_run_at = Get-NextRun
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
Write-Log "Collector setup failed: $(Get-ErrorText $_)" 'ERROR'
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
$failures = 0
|
||||||
|
$runsSent = 0
|
||||||
|
try {
|
||||||
|
if ($pending.Count -eq 0) {
|
||||||
|
# Keep next_run_at fresh even when nothing new ran
|
||||||
|
Send-Report ([ordered]@{ tenant = $Config.Tenant; source_system = $SourceSystem; job = $jobInfo })
|
||||||
|
}
|
||||||
|
foreach ($job in $pending) {
|
||||||
|
$run = ConvertTo-Run $job
|
||||||
|
Send-Report ([ordered]@{ tenant = $Config.Tenant; source_system = $SourceSystem; job = $jobInfo; run = $run })
|
||||||
|
$runsSent++
|
||||||
|
if ($run.status -ne 'running') {
|
||||||
|
$sent[$run.external_id] = ([DateTimeOffset](Get-JobTimeUtc $job 'StartTime')).ToUnixTimeSeconds()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
$failures++
|
||||||
|
Write-Log "Failed to report: $(Get-ErrorText $_)" 'ERROR'
|
||||||
|
}
|
||||||
|
|
||||||
|
if (-not $DryRun) {
|
||||||
|
# Forget jobs that have aged out of the lookback window (plus a day of slack)
|
||||||
|
$cutoff = [DateTimeOffset]::Now.AddHours(-$LookbackHours - 24).ToUnixTimeSeconds()
|
||||||
|
$keep = [ordered]@{}
|
||||||
|
foreach ($id in $sent.Keys) {
|
||||||
|
if ($sent[$id] -ge $cutoff) { $keep[$id] = $sent[$id] }
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
$keep | ConvertTo-Json -Compress | Set-Content -Path $StatePath -Encoding UTF8
|
||||||
|
} catch {
|
||||||
|
Write-Log "Couldn't save state file: $(Get-ErrorText $_)" 'WARN'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Write-Log ("Reported {0} runs, {1} failures" -f $runsSent, $failures)
|
||||||
|
if ($failures) { exit 1 }
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
# Stand-ins for the MARS agent cmdlets. Object shapes follow Get-MarsCollectorDiagnostics.ps1
|
||||||
|
# output from a real agent (2.0.9274): job times are UTC values with no UTC marking, "not set" is
|
||||||
|
# 1601-01-01, ErrorInfo is always present (ErrorCode 0 = no error), Datasource is an object, and
|
||||||
|
# schedule times are TimeSpans.
|
||||||
|
#
|
||||||
|
# . ./tests/MockMars.ps1
|
||||||
|
# ./Send-MarsBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||||
|
|
||||||
|
# Fixed to the top of the hour, and IDs derived from times, so repeated runs see the same jobs
|
||||||
|
$now = Get-Date -Minute 0 -Second 0 -Millisecond 0
|
||||||
|
$notSet = [datetime]::new(1601, 1, 1)
|
||||||
|
function New-StableGuid([string]$Seed) {
|
||||||
|
[guid]::new([Security.Cryptography.MD5]::Create().ComputeHash([Text.Encoding]::UTF8.GetBytes($Seed)))
|
||||||
|
}
|
||||||
|
|
||||||
|
# Local time -> the UTC value MARS reports, with Kind Unspecified like the real objects
|
||||||
|
function ConvertTo-MarsTime([datetime]$Local) {
|
||||||
|
[datetime]::SpecifyKind($Local.ToUniversalTime(), [DateTimeKind]::Unspecified)
|
||||||
|
}
|
||||||
|
|
||||||
|
function New-MockErrorInfo([int]$Code = 0, $Params = @()) {
|
||||||
|
[pscustomobject]@{ DetailedErrorCode = 0; DetailedErrorSource = $null; ErrorCode = $Code; ErrorParamList = $Params }
|
||||||
|
}
|
||||||
|
|
||||||
|
function New-MockDatasource($Name, [uint64]$Done, [uint64]$FailedBytes = 0, [uint64]$FailedFiles = 0, $ErrorInfo = (New-MockErrorInfo)) {
|
||||||
|
[pscustomobject]@{
|
||||||
|
ByteProgress = [pscustomobject]@{ Changed = $Done; Failed = $FailedBytes; Progress = $Done; Total = 2TB }
|
||||||
|
Datasource = [pscustomobject]@{ DataSourceName = $Name; DataSourceId = 6310158806455554328 }
|
||||||
|
ErrorInfo = $ErrorInfo
|
||||||
|
FileProgress = [pscustomobject]@{ Changed = 100; CurrentFile = ''; Failed = $FailedFiles; Progress = 100; Total = 100 }
|
||||||
|
JobState = 'Completed'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function New-MockJob($Type, [datetime]$Start, $End, $State, $Datasources = @(), $ErrorInfo = (New-MockErrorInfo), $FailedFileLog = '') {
|
||||||
|
$endValue = if ($End) { ConvertTo-MarsTime $End } else { $notSet }
|
||||||
|
[pscustomobject]@{
|
||||||
|
IsRetried = $false
|
||||||
|
JobId = New-StableGuid "$Type|$($Start.Ticks)"
|
||||||
|
JobType = $Type
|
||||||
|
DisplayTime = $Start # the only local time on the object
|
||||||
|
JobStatus = [pscustomobject]@{
|
||||||
|
DatasourceStatus = $Datasources
|
||||||
|
EndTime = $endValue
|
||||||
|
ErrorInfo = $ErrorInfo
|
||||||
|
FailedFileLog = $FailedFileLog
|
||||||
|
JobState = $State
|
||||||
|
StartTime = ConvertTo-MarsTime $Start
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
$global:MockMarsRunning = @(
|
||||||
|
(New-MockJob 'Backup' $now.AddMinutes(-10) $null 'InProgress' @((New-MockDatasource 'E:\' 2GB)))
|
||||||
|
)
|
||||||
|
$global:MockMarsPrevious = @(
|
||||||
|
(New-MockJob 'Backup' $now.AddHours(-12) $now.AddHours(-11.5) 'Completed' @((New-MockDatasource 'E:\' 523407872))),
|
||||||
|
# Completed, but some files failed
|
||||||
|
(New-MockJob 'Backup' $now.AddHours(-36) $now.AddHours(-35) 'Completed' @(
|
||||||
|
(New-MockDatasource 'E:\' 3GB 40MB 12)) -FailedFileLog 'C:\Program Files\Microsoft Azure Recovery Services Agent\Temp\Errors.log'),
|
||||||
|
(New-MockJob 'Backup' $now.AddHours(-30) $now.AddHours(-29.5) 'Failed' @() `
|
||||||
|
(New-MockErrorInfo 100050 @([pscustomobject]@{ Name = 'Volume'; Value = 'E:' }))),
|
||||||
|
(New-MockJob 'Recovery' $now.AddHours(-5) $now.AddHours(-4) 'Completed'), # a restore: must be ignored
|
||||||
|
(New-MockJob 'Backup' $now.AddDays(-5) $now.AddDays(-5).AddHours(1) 'Completed') # outside 48h lookback
|
||||||
|
)
|
||||||
|
|
||||||
|
# Every day at 21:00 local, as on the real server
|
||||||
|
$global:MockMarsSchedule = [pscustomobject]@{
|
||||||
|
ScheduleRunDays = [DayOfWeek[]](0..6)
|
||||||
|
ScheduleRunTimes = [timespan[]]@([timespan]::FromHours(21))
|
||||||
|
ScheduleWeeklyFrequency = 1
|
||||||
|
TypeOfBackupSchedule = 'Daily'
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-OBJob { param([int]$Previous) if ($PSBoundParameters.ContainsKey('Previous')) { $global:MockMarsPrevious } else { $global:MockMarsRunning } }
|
||||||
|
function Get-OBPolicy { [pscustomobject]@{ PolicyName = 'policy'; BackupSchedule = $global:MockMarsSchedule } }
|
||||||
|
function Get-OBSchedule { param($Policy) $global:MockMarsSchedule }
|
||||||
|
function Get-OBSystemStatePolicy { $null } # none configured, as on the real server
|
||||||
@@ -0,0 +1,188 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Read-only report of how Veeam jobs and sessions relate, for troubleshooting the collector.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
The collector matches sessions to jobs by Session.JobId = Job.Id. When a job shows as
|
||||||
|
"Unknown" on the dashboard, no session matched. This lists:
|
||||||
|
1. every job from Get-VBRJob and how many sessions matched it
|
||||||
|
2. sessions whose JobId matches no job (the runs that are going missing)
|
||||||
|
3. for jobs with no matching sessions, their type/link-related properties and methods
|
||||||
|
4. what other job cmdlets (e.g. Get-VBRBackupCopyJob) return
|
||||||
|
5. each job's schedule properties, which the collector reads for the next run
|
||||||
|
Nothing is changed in Veeam or the dashboard. Output is also saved to a text file.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Get-VeeamCollectorDiagnostics.ps1
|
||||||
|
.EXAMPLE
|
||||||
|
.\Get-VeeamCollectorDiagnostics.ps1 -LookbackHours 168
|
||||||
|
#>
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[int]$LookbackHours = 48,
|
||||||
|
[string]$OutFile = (Join-Path $env:TEMP 'veeam-collector-diagnostics.txt')
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
$LinkPattern = 'Parent|Linked|Worker|Child|Policy|Source|Target|Type|Cloud|Mode|Session'
|
||||||
|
|
||||||
|
if (-not (Get-Command Get-VBRJob -ErrorAction SilentlyContinue)) {
|
||||||
|
if (Get-Module -ListAvailable -Name Veeam.Backup.PowerShell) {
|
||||||
|
Import-Module Veeam.Backup.PowerShell -WarningAction SilentlyContinue
|
||||||
|
} else {
|
||||||
|
Add-PSSnapin VeeamPSSnapin
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function Format-Value($Value) {
|
||||||
|
if ($null -eq $Value) { return '<null>' }
|
||||||
|
$text = ($Value | ForEach-Object { [string]$_ }) -join ', '
|
||||||
|
if ($text.Length -gt 120) { $text = $text.Substring(0, 117) + '...' }
|
||||||
|
return $text
|
||||||
|
}
|
||||||
|
|
||||||
|
# Property and method names matching $LinkPattern, with property values. Every read is guarded:
|
||||||
|
# some Veeam properties throw when accessed.
|
||||||
|
function Show-LinkMembers($Object, [string]$Pattern = $LinkPattern, [string]$Indent = ' ') {
|
||||||
|
foreach ($p in $Object.PSObject.Properties | Where-Object { $_.Name -match $Pattern }) {
|
||||||
|
$value = try { Format-Value $p.Value } catch { "<error: $($_.Exception.Message)>" }
|
||||||
|
'{0}{1} = {2}' -f $Indent, $p.Name, $value
|
||||||
|
}
|
||||||
|
$methods = @($Object | Get-Member -MemberType Methods -ErrorAction SilentlyContinue |
|
||||||
|
Where-Object { $_.Name -match $Pattern } | ForEach-Object Name)
|
||||||
|
if ($methods) { $Indent + 'methods: ' + ($methods -join ', ') }
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-Prop($Object, [string]$Name) {
|
||||||
|
try { Format-Value $Object.$Name } catch { '<error>' }
|
||||||
|
}
|
||||||
|
|
||||||
|
$report = & {
|
||||||
|
$vbrVersion = try {
|
||||||
|
(Get-Item (Join-Path ${env:ProgramFiles} 'Veeam\Backup and Replication\Backup\Veeam.Backup.Core.dll')).VersionInfo.ProductVersion
|
||||||
|
} catch { 'unknown' }
|
||||||
|
"Generated $(Get-Date -Format o) on $env:COMPUTERNAME, VBR $vbrVersion, PowerShell $($PSVersionTable.PSVersion)"
|
||||||
|
"Lookback: $LookbackHours hours"
|
||||||
|
''
|
||||||
|
|
||||||
|
$since = (Get-Date).AddHours(-$LookbackHours)
|
||||||
|
$jobs = @(Get-VBRJob -WarningAction SilentlyContinue)
|
||||||
|
$sessions = @(Get-VBRBackupSession -WarningAction SilentlyContinue | Where-Object { $_.CreationTime -ge $since })
|
||||||
|
$jobIds = @{}
|
||||||
|
foreach ($j in $jobs) { $jobIds[$j.Id.ToString()] = $j }
|
||||||
|
|
||||||
|
'== 1. Jobs from Get-VBRJob'
|
||||||
|
$jobs | ForEach-Object {
|
||||||
|
$id = $_.Id.ToString()
|
||||||
|
[pscustomobject]@{
|
||||||
|
Name = $_.Name
|
||||||
|
JobType = Get-Prop $_ 'JobType'
|
||||||
|
TypeToString = Get-Prop $_ 'TypeToString'
|
||||||
|
Id = $id
|
||||||
|
MatchingSessions = @($sessions | Where-Object { $_.JobId.ToString() -eq $id }).Count
|
||||||
|
}
|
||||||
|
} | Format-Table -AutoSize | Out-String -Width 300
|
||||||
|
|
||||||
|
'== 2. Sessions whose JobId matches no job from Get-VBRJob'
|
||||||
|
$orphans = @($sessions | Where-Object { -not $jobIds.ContainsKey($_.JobId.ToString()) })
|
||||||
|
if ($orphans.Count -eq 0) {
|
||||||
|
' (none)'
|
||||||
|
} else {
|
||||||
|
$orphans | Group-Object { $_.JobId.ToString() } | ForEach-Object {
|
||||||
|
$latest = $_.Group | Sort-Object CreationTime | Select-Object -Last 1
|
||||||
|
[pscustomobject]@{
|
||||||
|
JobId = $_.Name
|
||||||
|
JobName = Get-Prop $latest 'JobName'
|
||||||
|
JobType = Get-Prop $latest 'JobType'
|
||||||
|
Sessions = $_.Count
|
||||||
|
LatestStart = $latest.CreationTime
|
||||||
|
State = Get-Prop $latest 'State'
|
||||||
|
Result = Get-Prop $latest 'Result'
|
||||||
|
}
|
||||||
|
} | Format-Table -AutoSize | Out-String -Width 300
|
||||||
|
|
||||||
|
' Link-related members of one orphan session:'
|
||||||
|
Show-LinkMembers $orphans[0]
|
||||||
|
''
|
||||||
|
' Job object for that session (via GetJob(), if available):'
|
||||||
|
try {
|
||||||
|
$owner = $orphans[0].GetJob()
|
||||||
|
' Name = {0}, Id = {1}' -f (Get-Prop $owner 'Name'), (Get-Prop $owner 'Id')
|
||||||
|
Show-LinkMembers $owner
|
||||||
|
} catch {
|
||||||
|
" not available: $($_.Exception.Message)"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
''
|
||||||
|
|
||||||
|
'== 3. Jobs with no matching sessions in the window'
|
||||||
|
$empty = @($jobs | Where-Object { $id = $_.Id.ToString(); -not ($sessions | Where-Object { $_.JobId.ToString() -eq $id }) })
|
||||||
|
if ($empty.Count -eq 0) { ' (none)' }
|
||||||
|
foreach ($j in $empty) {
|
||||||
|
" $($j.Name) [$($j.Id)]"
|
||||||
|
Show-LinkMembers $j
|
||||||
|
try {
|
||||||
|
$last = $j.FindLastSession()
|
||||||
|
if ($last) {
|
||||||
|
' FindLastSession(): JobId = {0}, JobName = {1}, Start = {2}, Result = {3}' -f `
|
||||||
|
(Get-Prop $last 'JobId'), (Get-Prop $last 'JobName'), $last.CreationTime, (Get-Prop $last 'Result')
|
||||||
|
} else {
|
||||||
|
' FindLastSession(): <null>'
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
" FindLastSession(): not available: $($_.Exception.Message)"
|
||||||
|
}
|
||||||
|
''
|
||||||
|
}
|
||||||
|
|
||||||
|
'== 4. Other job cmdlets'
|
||||||
|
foreach ($cmd in 'Get-VBRBackupCopyJob', 'Get-VBRComputerBackupCopyJob', 'Get-VBRComputerBackupJob') {
|
||||||
|
if (-not (Get-Command $cmd -ErrorAction SilentlyContinue)) { " ${cmd}: not available"; continue }
|
||||||
|
try {
|
||||||
|
$found = @(& $cmd -WarningAction SilentlyContinue)
|
||||||
|
" ${cmd}: $($found.Count) job(s)"
|
||||||
|
foreach ($f in $found) {
|
||||||
|
' {0} [{1}] Type = {2}, Mode = {3}' -f (Get-Prop $f 'Name'), (Get-Prop $f 'Id'), (Get-Prop $f 'Type'), (Get-Prop $f 'Mode')
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
" ${cmd}: error: $($_.Exception.Message)"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
''
|
||||||
|
|
||||||
|
# The collector uses IsScheduleEnabled and ScheduleOptions.NextRun; show what else is there in
|
||||||
|
# case this version keeps the next run somewhere different
|
||||||
|
"== 5. Schedules (culture: $([Globalization.CultureInfo]::CurrentCulture.Name))"
|
||||||
|
$SchedulePattern = 'Schedul|Next|Latest|Run|Enabled|Daily|Periodic|Monthly|Continuous|After|Option'
|
||||||
|
foreach ($j in $jobs) {
|
||||||
|
" $($j.Name) [$(Get-Prop $j 'JobType')]"
|
||||||
|
Show-LinkMembers $j $SchedulePattern ' '
|
||||||
|
foreach ($source in 'ScheduleOptions', 'GetScheduleOptions()', 'Get-VBRJobScheduleOptions') {
|
||||||
|
try {
|
||||||
|
$opts = switch ($source) {
|
||||||
|
'ScheduleOptions' { $j.ScheduleOptions }
|
||||||
|
'GetScheduleOptions()' { $j.GetScheduleOptions() }
|
||||||
|
'Get-VBRJobScheduleOptions' { Get-VBRJobScheduleOptions -Job $j -WarningAction SilentlyContinue }
|
||||||
|
}
|
||||||
|
if ($null -eq $opts) { " ${source}: <null>"; continue }
|
||||||
|
" ${source} ($($opts.GetType().FullName)):"
|
||||||
|
foreach ($p in $opts.PSObject.Properties) {
|
||||||
|
$value = try { $p.Value } catch { "<error: $($_.Exception.Message)>" }
|
||||||
|
$type = if ($null -eq $value) { 'null' } else { $value.GetType().Name }
|
||||||
|
if ($value -is [string] -or $value -is [ValueType] -or $null -eq $value) {
|
||||||
|
' {0} ({1}) = {2}' -f $p.Name, $type, (Format-Value $value)
|
||||||
|
} else {
|
||||||
|
' {0} ({1})' -f $p.Name, $type
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
" ${source}: not available: $($_.Exception.Message)"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
''
|
||||||
|
}
|
||||||
|
} | Out-String -Width 300
|
||||||
|
|
||||||
|
$report
|
||||||
|
$report | Set-Content -Path $OutFile -Encoding UTF8
|
||||||
|
Write-Host "Saved to $OutFile"
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Installs the Veeam collector on a VBR server as a SYSTEM scheduled task.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
- Copies Send-VeeamBackupStatus.ps1 to Program Files (only admins can modify it, which
|
||||||
|
matters because it runs as SYSTEM).
|
||||||
|
- Writes the config to ProgramData\BackupDashboard and restricts that folder to SYSTEM and
|
||||||
|
Administrators, since it contains the API key.
|
||||||
|
- Registers a scheduled task that runs every IntervalMinutes under PowerShell 7 (pwsh.exe),
|
||||||
|
which the Veeam module on VBR 13 expects. Install PowerShell 7 machine-wide (MSI or winget)
|
||||||
|
first: Microsoft Store installs can't be run by SYSTEM.
|
||||||
|
Safe to re-run to change settings or update the script.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Install-VeeamCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||||
|
(prompts for the API key)
|
||||||
|
#>
|
||||||
|
#Requires -RunAsAdministrator
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[Parameter(Mandatory)][string]$ApiUrl,
|
||||||
|
[Parameter(Mandatory)][string]$Tenant,
|
||||||
|
[string]$SourceSystem = 'Veeam B&R',
|
||||||
|
[ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
|
||||||
|
[ValidateRange(1, 720)][int]$LookbackHours = 48,
|
||||||
|
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard'),
|
||||||
|
# Defaults to the machine-wide PowerShell 7 install
|
||||||
|
[string]$PwshPath
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
$TaskName = 'Backup Dashboard - Veeam collector'
|
||||||
|
$DataDir = Join-Path $env:ProgramData 'BackupDashboard'
|
||||||
|
$ConfigPath = Join-Path $DataDir 'veeam-collector.json'
|
||||||
|
|
||||||
|
# Find PowerShell 7 before changing anything, so a missing install fails cleanly
|
||||||
|
if (-not $PwshPath) {
|
||||||
|
$PwshPath = Join-Path $env:ProgramFiles 'PowerShell\7\pwsh.exe'
|
||||||
|
if (-not (Test-Path $PwshPath)) {
|
||||||
|
$onPath = Get-Command pwsh.exe -ErrorAction SilentlyContinue
|
||||||
|
if ($onPath) { $PwshPath = $onPath.Source }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (-not (Test-Path $PwshPath)) {
|
||||||
|
throw "PowerShell 7 not found at '$PwshPath'. Install it machine-wide (e.g. winget install Microsoft.PowerShell) or pass -PwshPath."
|
||||||
|
}
|
||||||
|
if ($PwshPath -match '\\WindowsApps\\') {
|
||||||
|
throw "'$PwshPath' is a Microsoft Store install, which the SYSTEM account can't run. Install the MSI or winget version instead."
|
||||||
|
}
|
||||||
|
$pwshVersion = & $PwshPath -NoProfile -NonInteractive -Command '$PSVersionTable.PSVersion.ToString()'
|
||||||
|
$pwshMajor = & $PwshPath -NoProfile -NonInteractive -Command '$PSVersionTable.PSVersion.Major'
|
||||||
|
if ($LASTEXITCODE -or ($pwshMajor -as [int]) -lt 7) {
|
||||||
|
throw "'$PwshPath' isn't a working PowerShell 7 (reported '$pwshVersion')."
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($ApiUrl -notmatch '^https://') {
|
||||||
|
Write-Warning 'ApiUrl is not HTTPS: the API key would be sent in clear text.'
|
||||||
|
}
|
||||||
|
|
||||||
|
# Read the key as a SecureString so it isn't echoed to the console or kept in shell history
|
||||||
|
$secure = Read-Host -AsSecureString 'API key (from: python manage.py create-key <name>)'
|
||||||
|
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
|
||||||
|
try { $apiKey = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) }
|
||||||
|
finally { [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr) }
|
||||||
|
if (-not $apiKey.StartsWith('bdk_')) { throw "That doesn't look like a dashboard API key (expected 'bdk_...')." }
|
||||||
|
|
||||||
|
# Config folder: SYSTEM + Administrators only (SIDs so this works on non-English Windows)
|
||||||
|
New-Item -ItemType Directory -Path $DataDir -Force | Out-Null
|
||||||
|
& icacls.exe $DataDir /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' | Out-Null
|
||||||
|
if ($LASTEXITCODE) { throw "icacls failed on $DataDir" }
|
||||||
|
|
||||||
|
[ordered]@{
|
||||||
|
ApiUrl = $ApiUrl.TrimEnd('/')
|
||||||
|
ApiKey = $apiKey
|
||||||
|
Tenant = $Tenant
|
||||||
|
SourceSystem = $SourceSystem
|
||||||
|
LookbackHours = $LookbackHours
|
||||||
|
} | ConvertTo-Json | Set-Content -Path $ConfigPath -Encoding UTF8
|
||||||
|
|
||||||
|
New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null
|
||||||
|
$scriptPath = Join-Path $InstallDir 'Send-VeeamBackupStatus.ps1'
|
||||||
|
Copy-Item -Path (Join-Path $PSScriptRoot 'Send-VeeamBackupStatus.ps1') -Destination $scriptPath -Force
|
||||||
|
Unblock-File -Path $scriptPath
|
||||||
|
|
||||||
|
$action = New-ScheduledTaskAction -Execute $PwshPath `
|
||||||
|
-Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$scriptPath`""
|
||||||
|
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date).AddMinutes(1) `
|
||||||
|
-RepetitionInterval (New-TimeSpan -Minutes $IntervalMinutes)
|
||||||
|
$principal = New-ScheduledTaskPrincipal -UserId 'S-1-5-18' -LogonType ServiceAccount -RunLevel Highest
|
||||||
|
$settings = New-ScheduledTaskSettingsSet -ExecutionTimeLimit (New-TimeSpan -Minutes 30) `
|
||||||
|
-MultipleInstances IgnoreNew -StartWhenAvailable
|
||||||
|
Register-ScheduledTask -TaskName $TaskName -Action $action -Trigger $trigger -Principal $principal `
|
||||||
|
-Settings $settings -Description 'Sends Veeam job results to the Backup Dashboard' -Force | Out-Null
|
||||||
|
|
||||||
|
Write-Host "Installed. Config: $ConfigPath"
|
||||||
|
Write-Host "Task '$TaskName' runs every $IntervalMinutes minutes as SYSTEM, using PowerShell $pwshVersion ($PwshPath)."
|
||||||
|
Write-Host ''
|
||||||
|
Write-Host 'Next steps:'
|
||||||
|
Write-Host " 1. Preview what will be sent: & '$PwshPath' -File '$scriptPath' -DryRun"
|
||||||
|
Write-Host " 2. Backfill 30 days of history: & '$PwshPath' -File '$scriptPath' -LookbackHours 720"
|
||||||
|
Write-Host " 3. Check the log: Get-Content '$DataDir\veeam-collector.log' -Tail 20"
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
# Veeam Backup & Replication collector
|
||||||
|
|
||||||
|
A PowerShell script that runs on each client's VBR server and reports job results to the dashboard.
|
||||||
|
It covers jobs returned by `Get-VBRJob`: VM backup, replication and backup copy jobs.
|
||||||
|
|
||||||
|
## Install (on the VBR server, as an administrator)
|
||||||
|
|
||||||
|
Requires PowerShell 7 installed machine-wide (`winget install Microsoft.PowerShell` or the MSI),
|
||||||
|
which VBR 13's Veeam module expects. The Microsoft Store version won't work, because the SYSTEM
|
||||||
|
account can't run it. The installer checks for this before changing anything.
|
||||||
|
|
||||||
|
1. On the dashboard host, create a key for this server:
|
||||||
|
```bash
|
||||||
|
python manage.py create-key veeam-acme
|
||||||
|
```
|
||||||
|
2. Copy this folder to the VBR server and run, from PowerShell 7 (`pwsh`):
|
||||||
|
```powershell
|
||||||
|
.\Install-VeeamCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||||
|
```
|
||||||
|
It prompts for the key, then:
|
||||||
|
- writes `C:\ProgramData\BackupDashboard\veeam-collector.json` (readable only by SYSTEM and Administrators)
|
||||||
|
- copies the script to `C:\Program Files\BackupDashboard\`
|
||||||
|
- registers the scheduled task **Backup Dashboard - Veeam collector** (SYSTEM, every 60 minutes,
|
||||||
|
run by `C:\Program Files\PowerShell\7\pwsh.exe`; use `-PwshPath` if it's installed elsewhere)
|
||||||
|
3. Preview the payloads, then backfill 30 days so the history view is populated straight away
|
||||||
|
(from `pwsh`):
|
||||||
|
```powershell
|
||||||
|
& 'C:\Program Files\BackupDashboard\Send-VeeamBackupStatus.ps1' -DryRun
|
||||||
|
& 'C:\Program Files\BackupDashboard\Send-VeeamBackupStatus.ps1' -LookbackHours 720
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-run the installer to change settings or update the script. To remove the collector,
|
||||||
|
delete the scheduled task and both `BackupDashboard` folders.
|
||||||
|
|
||||||
|
## What it sends
|
||||||
|
|
||||||
|
| Dashboard | Veeam |
|
||||||
|
|---|---|
|
||||||
|
| Company | `Tenant` from the config |
|
||||||
|
| Job | Job name. The job's GUID is the ID, so renaming a job in Veeam doesn't duplicate it |
|
||||||
|
| Status | Session result (Success / Warning / Failed). Any state other than Stopped is **Running**. A session that stopped with no result (e.g. cancelled) is reported as **Warning** |
|
||||||
|
| Started / finished | Session `CreationTime` / `EndTime` |
|
||||||
|
| Next scheduled | Job's next run, from the live schedule (`GetScheduleOptions()`). Sent as empty, so never flagged Overdue, for disabled jobs, chained ("run after") jobs, and immediate-mode backup copy jobs, which run whenever their source does |
|
||||||
|
| Message | Session reason, each VM or object that didn't succeed, and warning/error lines from the session log |
|
||||||
|
| Size | `BackupStats.BackupSize` |
|
||||||
|
|
||||||
|
**Backup copy jobs (VBR 12+):** Veeam records their sessions under hidden per-source "worker" jobs.
|
||||||
|
The collector files each worker session under its copy job (via the worker's `PolicyJobId`), so a
|
||||||
|
copy job gets one run per source job it copies, with the source in `copied_from`.
|
||||||
|
|
||||||
|
Each run covers the last 48 hours (`LookbackHours`), so a few missed runs (server rebooting, API
|
||||||
|
down) fill in automatically. Sessions already delivered are recorded in
|
||||||
|
`veeam-collector.state.json`. Running sessions are re-sent until they finish.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
- Log: `C:\ProgramData\BackupDashboard\veeam-collector.log` (rotates at 5 MB).
|
||||||
|
- Exit codes: `0` OK, `1` some jobs failed to send (they're retried next run), `2` setup failed
|
||||||
|
(config, Veeam module, or Veeam connection).
|
||||||
|
- `API key rejected`: the key was revoked or mistyped. Re-run the installer with a new key.
|
||||||
|
- A job shows **Unknown** on the dashboard: the collector sent the job but matched no sessions to it.
|
||||||
|
Run `.\Get-VeeamCollectorDiagnostics.ps1` (read-only). It lists sessions whose job ID matches no
|
||||||
|
job, plus the type and link properties of the jobs left without sessions.
|
||||||
|
- `Get-VBRBackupSession` loads every session in the VBR database. On a server with years of history,
|
||||||
|
this can make each run take a minute or more. The task's 30-minute limit and "don't start if
|
||||||
|
already running" setting keep that safe.
|
||||||
|
|
||||||
|
## Testing without Veeam
|
||||||
|
|
||||||
|
`tests/MockVeeam.ps1` defines stand-in `Get-VBRJob` / `Get-VBRBackupSession` cmdlets with a mix of
|
||||||
|
success, warning, failed, running and disabled jobs:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
. ./tests/MockVeeam.ps1
|
||||||
|
./Send-VeeamBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Not covered yet
|
||||||
|
|
||||||
|
- Veeam Agent jobs managed by VBR (`Get-VBRComputerBackupJob`), tape jobs, and SureBackup.
|
||||||
|
- Veeam Backup for Microsoft 365, which has its own PowerShell module.
|
||||||
@@ -0,0 +1,346 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Reports Veeam Backup & Replication job results to the Backup Dashboard.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
Runs on the VBR server, normally as a SYSTEM scheduled task every 60 minutes
|
||||||
|
(see Install-VeeamCollector.ps1). For every backup, replica and backup copy job it sends:
|
||||||
|
- the job's next scheduled run (drives the dashboard's Overdue flag), and
|
||||||
|
- every session from the last LookbackHours that the dashboard hasn't already received
|
||||||
|
in its final state. Running sessions are re-sent each time until they finish.
|
||||||
|
|
||||||
|
Already-reported sessions are tracked in a state file, and the ingest API is idempotent,
|
||||||
|
so a failed or repeated run is harmless: anything not confirmed is retried next time.
|
||||||
|
|
||||||
|
Compatible with Windows PowerShell 5.1 and PowerShell 7.
|
||||||
|
|
||||||
|
.PARAMETER ConfigPath
|
||||||
|
JSON config with ApiUrl, ApiKey and Tenant (optional: SourceSystem, LookbackHours).
|
||||||
|
|
||||||
|
.PARAMETER LookbackHours
|
||||||
|
Overrides the config value. Use 720 once to backfill 30 days of history.
|
||||||
|
|
||||||
|
.PARAMETER DryRun
|
||||||
|
Print the payloads instead of sending them. Doesn't touch the state or log files.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Send-VeeamBackupStatus.ps1 -DryRun
|
||||||
|
.EXAMPLE
|
||||||
|
.\Send-VeeamBackupStatus.ps1 -LookbackHours 720
|
||||||
|
#>
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[string]$ConfigPath = (Join-Path $env:ProgramData 'BackupDashboard\veeam-collector.json'),
|
||||||
|
[int]$LookbackHours = 0,
|
||||||
|
[switch]$DryRun
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
|
||||||
|
$DataDir = Split-Path -Parent $ConfigPath
|
||||||
|
$LogPath = Join-Path $DataDir 'veeam-collector.log'
|
||||||
|
$StatePath = Join-Path $DataDir 'veeam-collector.state.json'
|
||||||
|
$MaxLogBytes = 5MB
|
||||||
|
$Invariant = [Globalization.CultureInfo]::InvariantCulture
|
||||||
|
|
||||||
|
# ---------- Helpers ----------
|
||||||
|
|
||||||
|
function Write-Log {
|
||||||
|
param([string]$Message, [string]$Level = 'INFO')
|
||||||
|
$line = '{0} [{1}] {2}' -f (Get-Date).ToString('yyyy-MM-dd HH:mm:ss'), $Level, $Message
|
||||||
|
Write-Host $line
|
||||||
|
if ($DryRun) { return }
|
||||||
|
if ((Test-Path $LogPath) -and (Get-Item $LogPath).Length -gt $MaxLogBytes) {
|
||||||
|
Move-Item $LogPath "$LogPath.1" -Force
|
||||||
|
}
|
||||||
|
Add-Content -Path $LogPath -Value $line -Encoding UTF8
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-ErrorText($ErrorRecord) {
|
||||||
|
# Windows PowerShell puts the HTTP response body (e.g. the API's 422 explanation) here
|
||||||
|
if ($ErrorRecord.ErrorDetails -and $ErrorRecord.ErrorDetails.Message) {
|
||||||
|
return "$($ErrorRecord.Exception.Message) $($ErrorRecord.ErrorDetails.Message)"
|
||||||
|
}
|
||||||
|
return $ErrorRecord.Exception.Message
|
||||||
|
}
|
||||||
|
|
||||||
|
# The API needs an explicit UTC offset. Veeam returns local times, often with Kind=Unspecified,
|
||||||
|
# which DateTimeOffset treats as local, exactly what we want.
|
||||||
|
function ConvertTo-IsoTime($Value) {
|
||||||
|
if ($null -eq $Value) { return $null }
|
||||||
|
$dt = [datetime]$Value
|
||||||
|
if ($dt.Year -le 1900) { return $null } # Veeam's "not set" value (e.g. EndTime while running)
|
||||||
|
return ([DateTimeOffset]$dt).ToString("yyyy-MM-dd'T'HH:mm:sszzz", $Invariant)
|
||||||
|
}
|
||||||
|
|
||||||
|
function Import-Veeam {
|
||||||
|
if (Get-Command Get-VBRJob -ErrorAction SilentlyContinue) { return } # already loaded
|
||||||
|
if (Get-Module -ListAvailable -Name Veeam.Backup.PowerShell) {
|
||||||
|
Import-Module Veeam.Backup.PowerShell -WarningAction SilentlyContinue # VBR 11+
|
||||||
|
} else {
|
||||||
|
Add-PSSnapin VeeamPSSnapin # VBR 10 and earlier
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-NextRun($Job) {
|
||||||
|
try {
|
||||||
|
if (-not $Job.IsScheduleEnabled) { return $null }
|
||||||
|
# Immediate-mode backup copy jobs run whenever their source job does. Their schedule still
|
||||||
|
# carries a NextRun, but it's a stale date (months old on VBR 13) that would show as Overdue.
|
||||||
|
if ($Job.IsContinuous) { return $null }
|
||||||
|
|
||||||
|
# The ScheduleOptions property is a stale snapshot on VBR 13 (NextRun empty, last run years
|
||||||
|
# old); GetScheduleOptions() reads the live schedule. Fall back for versions without it.
|
||||||
|
$options = $null
|
||||||
|
try { $options = $Job.GetScheduleOptions() } catch { }
|
||||||
|
if (-not $options) { $options = $Job.ScheduleOptions }
|
||||||
|
$raw = $options.NextRun
|
||||||
|
if (-not $raw) { return $null } # e.g. chained jobs that run "after" another job
|
||||||
|
if ($raw -is [datetime]) { return ConvertTo-IsoTime $raw }
|
||||||
|
# A string formatted by VBR ("2026-10-09 12:15:00 AM" on VBR 13, en-CA). Try the server's
|
||||||
|
# culture first, then the invariant one
|
||||||
|
foreach ($culture in [Globalization.CultureInfo]::CurrentCulture, [Globalization.CultureInfo]::InvariantCulture) {
|
||||||
|
$parsed = [datetime]::MinValue
|
||||||
|
if ([datetime]::TryParse([string]$raw, $culture, [Globalization.DateTimeStyles]::None, [ref]$parsed)) {
|
||||||
|
return ConvertTo-IsoTime $parsed
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Write-Log "Couldn't parse next run '$raw' for job '$($Job.Name)'" 'WARN'
|
||||||
|
} catch {
|
||||||
|
Write-Log "Couldn't read schedule for job '$($Job.Name)': $(Get-ErrorText $_)" 'WARN'
|
||||||
|
}
|
||||||
|
return $null
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-RunStatus($Session) {
|
||||||
|
if ([string]$Session.State -ne 'Stopped') { return 'running' }
|
||||||
|
switch ([string]$Session.Result) {
|
||||||
|
'Success' { return 'success' }
|
||||||
|
'Warning' { return 'warning' }
|
||||||
|
'Failed' { return 'failed' }
|
||||||
|
# Stopped without a result (e.g. cancelled): surface it rather than hide it
|
||||||
|
default { return 'warning' }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Builds a readable summary: the session's reason, any task (VM/object) that didn't succeed,
|
||||||
|
# and warning/error lines from the session log.
|
||||||
|
function Get-RunMessage($Session, [string]$Status, $Tasks) {
|
||||||
|
$lines = New-Object 'System.Collections.Generic.List[string]'
|
||||||
|
try {
|
||||||
|
$reason = [string]$Session.Info.Reason
|
||||||
|
if ($reason.Trim()) { $lines.Add($reason.Trim()) }
|
||||||
|
} catch { }
|
||||||
|
|
||||||
|
foreach ($task in $Tasks) {
|
||||||
|
$taskStatus = [string]$task.Status
|
||||||
|
if ($taskStatus -and $taskStatus -ne 'Success' -and $taskStatus -ne 'InProgress' -and $taskStatus -ne 'Pending') {
|
||||||
|
$taskReason = ''
|
||||||
|
try { $taskReason = ([string]$task.Info.Reason).Trim() } catch { }
|
||||||
|
$line = "$($task.Name): $taskStatus"
|
||||||
|
if ($taskReason) { $line += " - $taskReason" }
|
||||||
|
$lines.Add($line)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($Status -eq 'warning' -or $Status -eq 'failed') {
|
||||||
|
try {
|
||||||
|
foreach ($record in $Session.Logger.GetLog().UpdatedRecords) {
|
||||||
|
$recordStatus = [string]$record.Status
|
||||||
|
if ($recordStatus -eq 'EWarning' -or $recordStatus -eq 'EFailed') {
|
||||||
|
$title = ([string]$record.Title).Trim()
|
||||||
|
if ($title -and -not $lines.Contains($title)) { $lines.Add($title) }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch { }
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($lines.Count -eq 0) {
|
||||||
|
switch ($Status) {
|
||||||
|
'success' { return 'Job completed successfully.' }
|
||||||
|
'running' { return '' }
|
||||||
|
default { return "Job finished with result '$($Session.Result)'." }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ($lines -join "`n")
|
||||||
|
}
|
||||||
|
|
||||||
|
# VBR 12+ backup copy jobs: Get-VBRJob returns the copy (policy) job, but Get-VBRBackupSession
|
||||||
|
# returns only the sessions of its hidden worker jobs, one per source job, never the policy's own.
|
||||||
|
# Each worker job records its policy in PolicyJobId, so those sessions can be filed under it.
|
||||||
|
function Get-PolicyJobId($Session, [hashtable]$KnownJobIds) {
|
||||||
|
try {
|
||||||
|
$policyId = [string]$Session.GetJob().PolicyJobId
|
||||||
|
if ($policyId -and $KnownJobIds.ContainsKey($policyId)) { return $policyId }
|
||||||
|
} catch { }
|
||||||
|
return ''
|
||||||
|
}
|
||||||
|
|
||||||
|
function ConvertTo-Run($Session, $Job) {
|
||||||
|
$status = Get-RunStatus $Session
|
||||||
|
$tasks = @()
|
||||||
|
try { $tasks = @($Session.GetTaskSessions()) } catch { }
|
||||||
|
|
||||||
|
$details = [ordered]@{
|
||||||
|
job_type = [string]$Job.JobType
|
||||||
|
vbr_server = $env:COMPUTERNAME
|
||||||
|
objects = $tasks.Count
|
||||||
|
}
|
||||||
|
$failedTasks = @($tasks | Where-Object { [string]$_.Status -eq 'Failed' }).Count
|
||||||
|
$warningTasks = @($tasks | Where-Object { [string]$_.Status -eq 'Warning' }).Count
|
||||||
|
if ($failedTasks) { $details.objects_failed = $failedTasks }
|
||||||
|
if ($warningTasks) { $details.objects_warning = $warningTasks }
|
||||||
|
try { if ($Session.IsRetryMode) { $details.retry = $true } } catch { }
|
||||||
|
# Worker session of a backup copy job: say which source job it copied
|
||||||
|
# (worker sessions are named "<copy job>\<source job>")
|
||||||
|
if ($Session.JobId.ToString() -ne $Job.Id.ToString()) {
|
||||||
|
$details.copied_from = ([string]$Session.JobName -split '\\')[-1]
|
||||||
|
}
|
||||||
|
|
||||||
|
$run = [ordered]@{
|
||||||
|
external_id = $Session.Id.ToString()
|
||||||
|
status = $status
|
||||||
|
started_at = ConvertTo-IsoTime $Session.CreationTime
|
||||||
|
message = Get-RunMessage $Session $status $tasks
|
||||||
|
details = $details
|
||||||
|
}
|
||||||
|
if ($status -ne 'running') {
|
||||||
|
$finished = ConvertTo-IsoTime $Session.EndTime
|
||||||
|
if ($finished) { $run.finished_at = $finished } # otherwise the API uses receipt time
|
||||||
|
try {
|
||||||
|
$size = [long]$Session.BackupStats.BackupSize
|
||||||
|
if ($size -gt 0) { $run.size_bytes = $size }
|
||||||
|
$dataSize = [long]$Session.BackupStats.DataSize
|
||||||
|
if ($dataSize -gt 0) { $details.data_size_bytes = $dataSize }
|
||||||
|
} catch { }
|
||||||
|
}
|
||||||
|
return $run
|
||||||
|
}
|
||||||
|
|
||||||
|
function Send-Report($Body) {
|
||||||
|
$json = $Body | ConvertTo-Json -Depth 6 -Compress
|
||||||
|
if ($DryRun) {
|
||||||
|
Write-Host $json
|
||||||
|
return
|
||||||
|
}
|
||||||
|
# Send bytes: Windows PowerShell would otherwise encode a string body as Latin-1,
|
||||||
|
# mangling non-ASCII job names (e.g. en dashes).
|
||||||
|
$params = @{
|
||||||
|
Uri = "$($Config.ApiUrl.TrimEnd('/'))/api/v1/ingest"
|
||||||
|
Method = 'Post'
|
||||||
|
Headers = @{ Authorization = "Bearer $($Config.ApiKey)" }
|
||||||
|
ContentType = 'application/json; charset=utf-8'
|
||||||
|
Body = [Text.Encoding]::UTF8.GetBytes($json)
|
||||||
|
TimeoutSec = 30
|
||||||
|
}
|
||||||
|
Invoke-RestMethod @params | Out-Null
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------- Main ----------
|
||||||
|
|
||||||
|
try {
|
||||||
|
$Config = Get-Content -Path $ConfigPath -Raw | ConvertFrom-Json
|
||||||
|
foreach ($key in 'ApiUrl', 'ApiKey', 'Tenant') {
|
||||||
|
if (-not $Config.$key) { throw "Config $ConfigPath is missing '$key'" }
|
||||||
|
}
|
||||||
|
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Veeam B&R' }
|
||||||
|
if ($LookbackHours -le 0) {
|
||||||
|
$LookbackHours = if ($Config.LookbackHours) { [int]$Config.LookbackHours } else { 48 }
|
||||||
|
}
|
||||||
|
|
||||||
|
# Windows PowerShell 5.1 may default to TLS 1.0
|
||||||
|
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
|
||||||
|
|
||||||
|
Import-Veeam
|
||||||
|
|
||||||
|
# Session ID -> start time (Unix seconds), for sessions already delivered in their final state.
|
||||||
|
# Numbers rather than ISO strings: PowerShell 7's ConvertFrom-Json turns date strings into
|
||||||
|
# DateTimes and loses the offset.
|
||||||
|
$sent = @{}
|
||||||
|
if (Test-Path $StatePath) {
|
||||||
|
$saved = Get-Content -Path $StatePath -Raw | ConvertFrom-Json
|
||||||
|
if ($saved) { foreach ($p in $saved.PSObject.Properties) { $sent[$p.Name] = [long]$p.Value } }
|
||||||
|
}
|
||||||
|
|
||||||
|
$since = (Get-Date).AddHours(-$LookbackHours)
|
||||||
|
$jobs = @(Get-VBRJob -WarningAction SilentlyContinue)
|
||||||
|
|
||||||
|
$jobIds = @{}
|
||||||
|
foreach ($j in $jobs) { $jobIds[$j.Id.ToString()] = $true }
|
||||||
|
|
||||||
|
# Get-VBRBackupSession returns every session in the database, so fetch once and bucket by job
|
||||||
|
$sessionsByJob = @{}
|
||||||
|
$policyByWorker = @{} # worker job ID -> policy job ID ('' if it isn't a worker of a listed job)
|
||||||
|
foreach ($s in Get-VBRBackupSession -WarningAction SilentlyContinue) {
|
||||||
|
if ($s.CreationTime -lt $since) { continue }
|
||||||
|
$key = $s.JobId.ToString()
|
||||||
|
if (-not $jobIds.ContainsKey($key)) {
|
||||||
|
if (-not $policyByWorker.ContainsKey($key)) { $policyByWorker[$key] = Get-PolicyJobId $s $jobIds }
|
||||||
|
if (-not $policyByWorker[$key]) { continue }
|
||||||
|
$key = $policyByWorker[$key]
|
||||||
|
}
|
||||||
|
if (-not $sessionsByJob.ContainsKey($key)) { $sessionsByJob[$key] = New-Object 'System.Collections.Generic.List[object]' }
|
||||||
|
$sessionsByJob[$key].Add($s)
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
Write-Log "Collector setup failed: $(Get-ErrorText $_)" 'ERROR'
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
$failures = 0
|
||||||
|
$runsSent = 0
|
||||||
|
foreach ($job in $jobs) {
|
||||||
|
$jobInfo = [ordered]@{
|
||||||
|
external_id = $job.Id.ToString()
|
||||||
|
name = $job.Name
|
||||||
|
# Always sent (null when unscheduled/disabled) so the dashboard's value tracks Veeam's
|
||||||
|
next_run_at = Get-NextRun $job
|
||||||
|
}
|
||||||
|
$pending = @()
|
||||||
|
if ($sessionsByJob.ContainsKey($jobInfo.external_id)) {
|
||||||
|
$pending = @($sessionsByJob[$jobInfo.external_id] |
|
||||||
|
Where-Object { -not $sent.ContainsKey($_.Id.ToString()) } |
|
||||||
|
Sort-Object CreationTime)
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
if ($pending.Count -eq 0) {
|
||||||
|
# Keep next_run_at fresh even when nothing new ran
|
||||||
|
Send-Report ([ordered]@{ tenant = $Config.Tenant; source_system = $SourceSystem; job = $jobInfo })
|
||||||
|
}
|
||||||
|
foreach ($session in $pending) {
|
||||||
|
$run = ConvertTo-Run $session $job
|
||||||
|
Send-Report ([ordered]@{ tenant = $Config.Tenant; source_system = $SourceSystem; job = $jobInfo; run = $run })
|
||||||
|
$runsSent++
|
||||||
|
if ($run.status -ne 'running') {
|
||||||
|
$sent[$run.external_id] = ([DateTimeOffset][datetime]$session.CreationTime).ToUnixTimeSeconds()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
$failures++
|
||||||
|
Write-Log "Failed to report job '$($job.Name)': $(Get-ErrorText $_)" 'ERROR'
|
||||||
|
$httpStatus = 0
|
||||||
|
try { $httpStatus = [int]$_.Exception.Response.StatusCode } catch { }
|
||||||
|
if ($httpStatus -eq 401) {
|
||||||
|
Write-Log 'API key rejected; check ApiKey in the config. Stopping.' 'ERROR'
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (-not $DryRun) {
|
||||||
|
# Forget sessions that have aged out of the lookback window (plus a day of slack)
|
||||||
|
$cutoff = [DateTimeOffset]::Now.AddHours(-$LookbackHours - 24).ToUnixTimeSeconds()
|
||||||
|
$keep = [ordered]@{}
|
||||||
|
foreach ($id in $sent.Keys) {
|
||||||
|
if ($sent[$id] -ge $cutoff) { $keep[$id] = $sent[$id] }
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
$keep | ConvertTo-Json -Compress | Set-Content -Path $StatePath -Encoding UTF8
|
||||||
|
} catch {
|
||||||
|
Write-Log "Couldn't save state file: $(Get-ErrorText $_)" 'WARN'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Write-Log ("Reported {0} jobs, {1} runs, {2} failures" -f $jobs.Count, $runsSent, $failures)
|
||||||
|
if ($failures) { exit 1 }
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
# Stand-ins for the Veeam cmdlets, shaped like the real objects' properties the collector reads.
|
||||||
|
# Dot-source this before running the collector to try it without a VBR server:
|
||||||
|
#
|
||||||
|
# . ./tests/MockVeeam.ps1
|
||||||
|
# ./Send-VeeamBackupStatus.ps1 -ConfigPath ./tests/config.json -DryRun
|
||||||
|
|
||||||
|
# Fixed to the top of the hour, and IDs derived from names/times, so repeated runs see the same
|
||||||
|
# jobs and sessions, like a real server
|
||||||
|
$now = Get-Date -Minute 0 -Second 0 -Millisecond 0
|
||||||
|
function New-StableGuid([string]$Seed) {
|
||||||
|
[guid]::new([Security.Cryptography.MD5]::Create().ComputeHash([Text.Encoding]::UTF8.GetBytes($Seed)))
|
||||||
|
}
|
||||||
|
$notSet = Get-Date '1900-01-01'
|
||||||
|
|
||||||
|
# As on VBR 13: the ScheduleOptions property is a stale snapshot with an empty NextRun, and the live
|
||||||
|
# schedule (NextRun as a "2026-10-09 12:15:00 AM" string) comes from GetScheduleOptions().
|
||||||
|
# -Continuous: an immediate-mode copy job, whose live NextRun is a stale date in the past.
|
||||||
|
function New-MockJob($Name, $Type, $NextRun, [bool]$Enabled = $true, [switch]$Continuous) {
|
||||||
|
$job = [pscustomobject]@{
|
||||||
|
Id = New-StableGuid $Name; Name = $Name; JobType = $Type
|
||||||
|
IsScheduleEnabled = $Enabled
|
||||||
|
IsContinuous = [bool]$Continuous
|
||||||
|
ScheduleOptions = [pscustomobject]@{ NextRun = '' }
|
||||||
|
LiveNextRun = $NextRun
|
||||||
|
}
|
||||||
|
$job | Add-Member ScriptMethod GetScheduleOptions { [pscustomobject]@{ NextRun = $this.LiveNextRun } }
|
||||||
|
$job
|
||||||
|
}
|
||||||
|
function Format-VbrTime([datetime]$Value) {
|
||||||
|
$Value.ToString('yyyy-MM-dd h:mm:ss tt', [Globalization.CultureInfo]::InvariantCulture)
|
||||||
|
}
|
||||||
|
|
||||||
|
function New-MockSession($Job, $Start, $End, $State, $Result, $Reason = '', $Tasks = @(), $LogRecords = @(), $Size = 0) {
|
||||||
|
$s = [pscustomobject]@{
|
||||||
|
Id = New-StableGuid "$($Job.Name)|$($Start.Ticks)"; JobId = $Job.Id; JobName = $Job.Name
|
||||||
|
CreationTime = $Start; EndTime = $End; State = $State; Result = $Result
|
||||||
|
IsRetryMode = $false
|
||||||
|
Info = [pscustomobject]@{ Reason = $Reason }
|
||||||
|
BackupStats = [pscustomobject]@{ BackupSize = $Size; DataSize = $Size * 3 }
|
||||||
|
Logger = [pscustomobject]@{ Records = $LogRecords }
|
||||||
|
Tasks = $Tasks
|
||||||
|
}
|
||||||
|
$s.Logger | Add-Member ScriptMethod GetLog { [pscustomobject]@{ UpdatedRecords = $this.Records } }
|
||||||
|
$s | Add-Member ScriptMethod GetTaskSessions { $this.Tasks }
|
||||||
|
$s
|
||||||
|
}
|
||||||
|
|
||||||
|
function New-MockTask($Name, $Status, $Reason = '') {
|
||||||
|
[pscustomobject]@{ Name = $Name; Status = $Status; Info = [pscustomobject]@{ Reason = $Reason } }
|
||||||
|
}
|
||||||
|
|
||||||
|
$global:MockJobs = @(
|
||||||
|
(New-MockJob 'File Server – Nightly' 'Backup' (Format-VbrTime $now.Date.AddDays(1).AddMinutes(15))),
|
||||||
|
# Some versions return NextRun as a DateTime
|
||||||
|
(New-MockJob 'SQL01 – Logs' 'Backup' $now.AddMinutes(40)),
|
||||||
|
(New-MockJob 'Offsite Copy' 'BackupSync' $null),
|
||||||
|
(New-MockJob 'Old Exchange' 'Backup' $now.AddDays(1) $false),
|
||||||
|
# VBR 12+ backup copy job (immediate mode, no schedule): the "policy" job Get-VBRJob returns
|
||||||
|
(New-MockJob 'Cloud Copy' 'SimpleBackupCopyPolicy' (Format-VbrTime $now.AddMonths(-6)) -Continuous)
|
||||||
|
)
|
||||||
|
$fs, $sql, $copy, $old, $cloud = $global:MockJobs
|
||||||
|
|
||||||
|
# Its hidden worker job, one per source job. Get-VBRJob doesn't return it, and Get-VBRBackupSession
|
||||||
|
# returns only the worker's sessions. Shape taken from a real VBR 13 server.
|
||||||
|
$cloudWorker = [pscustomobject]@{
|
||||||
|
Id = New-StableGuid 'Cloud Copy\File Server – Nightly'; Name = 'Cloud Copy\File Server – Nightly'
|
||||||
|
JobType = 'SimpleBackupCopyWorker'; PolicyJobId = $cloud.Id
|
||||||
|
}
|
||||||
|
function New-MockWorkerSession($Start, $End, $Result) {
|
||||||
|
$s = New-MockSession $cloudWorker $Start $End 'Stopped' $Result -Size 40GB
|
||||||
|
$s | Add-Member ScriptMethod GetJob { $cloudWorker }
|
||||||
|
$s
|
||||||
|
}
|
||||||
|
|
||||||
|
$global:MockSessions = @(
|
||||||
|
(New-MockSession $fs $now.AddDays(-1).AddHours(-2) $now.AddDays(-1).AddHours(-1) 'Stopped' 'Success' -Size 128GB -Tasks @(
|
||||||
|
(New-MockTask 'FS01' 'Success'), (New-MockTask 'FS02' 'Success'))),
|
||||||
|
(New-MockSession $fs $now.AddMinutes(-20) $notSet 'Working' 'None' -Tasks @((New-MockTask 'FS01' 'InProgress'))),
|
||||||
|
(New-MockSession $sql $now.AddHours(-1) $now.AddMinutes(-55) 'Stopped' 'Warning' -Size 2GB `
|
||||||
|
-Tasks @((New-MockTask 'SQL01' 'Warning' 'Failed to truncate transaction logs for database "ERP_PROD"'))),
|
||||||
|
(New-MockSession $copy $now.AddHours(-5) $now.AddHours(-4) 'Stopped' 'Failed' 'Error: Unable to connect to repository NAS02' `
|
||||||
|
-LogRecords @(
|
||||||
|
[pscustomobject]@{ Title = 'Preparing backup chain'; Status = 'ESucceeded' },
|
||||||
|
[pscustomobject]@{ Title = 'Network path not found (0x80070035)'; Status = 'EFailed' })),
|
||||||
|
(New-MockSession $old $now.AddDays(-10) $now.AddDays(-10).AddHours(1) 'Stopped' 'Success'), # outside 48h lookback
|
||||||
|
(New-MockWorkerSession $now.AddDays(-1).AddMinutes(-55) $now.AddDays(-1).AddMinutes(-30) 'Success'),
|
||||||
|
(New-MockWorkerSession $now.AddMinutes(-55) $now.AddMinutes(-35) 'Warning'),
|
||||||
|
# A session from a job the collector doesn't know (e.g. deleted): must be ignored, not crash
|
||||||
|
(New-MockSession ([pscustomobject]@{ Id = New-StableGuid 'gone'; Name = 'Deleted job' }) $now.AddHours(-2) $now.AddHours(-1) 'Stopped' 'Success')
|
||||||
|
)
|
||||||
|
|
||||||
|
function Get-VBRJob { [CmdletBinding()] param() $global:MockJobs }
|
||||||
|
function Get-VBRBackupSession { [CmdletBinding()] param() $global:MockSessions }
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Read-only dump of what the Windows Server Backup cmdlets return, for troubleshooting the collector.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
The collector reads properties of Get-WBJob / Get-WBSummary results whose names and time zones
|
||||||
|
aren't fully documented. This prints those objects' real structure (property names, types and
|
||||||
|
values, a few levels deep), plus recent events from the Microsoft-Windows-Backup log, whose
|
||||||
|
timestamps are reliable local times to compare against. Nothing is changed. Output is also
|
||||||
|
saved to a text file.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Get-WsbCollectorDiagnostics.ps1
|
||||||
|
#>
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[int]$Jobs = 3,
|
||||||
|
[int]$Events = 15,
|
||||||
|
[string]$OutFile = (Join-Path $env:TEMP 'wsb-collector-diagnostics.txt')
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
$MaxDepth = 6
|
||||||
|
|
||||||
|
if (-not (Get-Command Get-WBJob -ErrorAction SilentlyContinue)) {
|
||||||
|
Import-Module WindowsServerBackup
|
||||||
|
}
|
||||||
|
|
||||||
|
function Test-Scalar($Value) {
|
||||||
|
return $null -eq $Value -or $Value -is [string] -or $Value -is [ValueType] -or $Value -is [guid]
|
||||||
|
}
|
||||||
|
|
||||||
|
# Recursive "name (type) = value" listing. Collections show their first 3 items.
|
||||||
|
function Show-Object($Value, [string]$Indent = ' ', [int]$Depth = 0) {
|
||||||
|
if (Test-Scalar $Value) { return }
|
||||||
|
if ($Depth -ge $MaxDepth) { "$Indent..."; return }
|
||||||
|
if ($Value -is [Collections.IEnumerable]) {
|
||||||
|
$items = @($Value)
|
||||||
|
"$Indent[$($items.Count) item(s)]"
|
||||||
|
$i = 0
|
||||||
|
foreach ($item in $items | Select-Object -First 3) {
|
||||||
|
"$Indent[$i] ($($item.GetType().FullName))$(if (Test-Scalar $item) { " = $item" })"
|
||||||
|
Show-Object $item "$Indent " ($Depth + 1)
|
||||||
|
$i++
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
foreach ($p in $Value.PSObject.Properties) {
|
||||||
|
try { $v = $p.Value } catch { "$Indent$($p.Name) = <error: $($_.Exception.Message)>"; continue }
|
||||||
|
$type = if ($null -eq $v) { 'null' } else { $v.GetType().FullName }
|
||||||
|
if (Test-Scalar $v) {
|
||||||
|
"$Indent$($p.Name) ($type) = $v"
|
||||||
|
} else {
|
||||||
|
$text = [string]$v
|
||||||
|
if ($text.Length -gt 100) { $text = $text.Substring(0, 97) + '...' }
|
||||||
|
"$Indent$($p.Name) ($type) = $text"
|
||||||
|
Show-Object $v "$Indent " ($Depth + 1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function Show-Section([string]$Title, [scriptblock]$Get) {
|
||||||
|
"== $Title"
|
||||||
|
try {
|
||||||
|
$result = @(& $Get)
|
||||||
|
if ($result.Count -eq 0 -or ($result.Count -eq 1 -and $null -eq $result[0])) { ' (nothing returned)' }
|
||||||
|
$n = 0
|
||||||
|
foreach ($item in $result) {
|
||||||
|
if ($null -eq $item) { continue }
|
||||||
|
" [$n] ($($item.GetType().FullName))"
|
||||||
|
Show-Object $item ' '
|
||||||
|
$n++
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
" error: $($_.Exception.Message)"
|
||||||
|
}
|
||||||
|
''
|
||||||
|
}
|
||||||
|
|
||||||
|
$report = & {
|
||||||
|
"Generated $(Get-Date -Format o) on $env:COMPUTERNAME, PowerShell $($PSVersionTable.PSVersion)"
|
||||||
|
"Local time zone: $([TimeZoneInfo]::Local.Id), UTC offset now $([TimeZoneInfo]::Local.GetUtcOffset((Get-Date)))"
|
||||||
|
''
|
||||||
|
Show-Section 'Get-WBSummary' { Get-WBSummary }
|
||||||
|
Show-Section 'Get-WBJob (in progress)' { Get-WBJob }
|
||||||
|
Show-Section "Get-WBJob -Previous $Jobs" { Get-WBJob -Previous $Jobs }
|
||||||
|
Show-Section 'Get-WBPolicy | Get-WBSchedule' {
|
||||||
|
$policy = Get-WBPolicy
|
||||||
|
if ($policy) { Get-WBSchedule -Policy $policy } else { '(no scheduled backup policy)' }
|
||||||
|
}
|
||||||
|
"== Microsoft-Windows-Backup event log (last $Events, local times)"
|
||||||
|
try {
|
||||||
|
Get-WinEvent -LogName 'Microsoft-Windows-Backup' -MaxEvents $Events |
|
||||||
|
Select-Object TimeCreated, Id, LevelDisplayName, @{ n = 'Message'; e = { ($_.Message -split "`r?`n")[0] } } |
|
||||||
|
Format-Table -AutoSize -Wrap | Out-String -Width 300
|
||||||
|
} catch {
|
||||||
|
" error: $($_.Exception.Message)"
|
||||||
|
}
|
||||||
|
} | Out-String -Width 300
|
||||||
|
|
||||||
|
$report
|
||||||
|
$report | Set-Content -Path $OutFile -Encoding UTF8
|
||||||
|
Write-Host "Saved to $OutFile"
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Installs the Windows Server Backup collector as a SYSTEM scheduled task.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
- Copies Send-WsbBackupStatus.ps1 to Program Files (only admins can modify it, which
|
||||||
|
matters because it runs as SYSTEM).
|
||||||
|
- Writes the config to ProgramData\BackupDashboard and restricts that folder to SYSTEM and
|
||||||
|
Administrators, since it contains the API key.
|
||||||
|
- Registers a scheduled task that runs every IntervalMinutes under Windows PowerShell 5.1
|
||||||
|
(powershell.exe), which the WindowsServerBackup module targets.
|
||||||
|
Safe to re-run to change settings or update the script. Can sit alongside the other collectors.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Install-WsbCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||||
|
(prompts for the API key)
|
||||||
|
#>
|
||||||
|
#Requires -RunAsAdministrator
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[Parameter(Mandatory)][string]$ApiUrl,
|
||||||
|
[Parameter(Mandatory)][string]$Tenant,
|
||||||
|
# Name shown on the dashboard; defaults to "<server> - Windows Server Backup"
|
||||||
|
[string]$JobName,
|
||||||
|
[string]$SourceSystem = 'Windows Server Backup',
|
||||||
|
[ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
|
||||||
|
[ValidateRange(1, 720)][int]$LookbackHours = 48,
|
||||||
|
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard')
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
$TaskName = 'Backup Dashboard - WSB collector'
|
||||||
|
$DataDir = Join-Path $env:ProgramData 'BackupDashboard'
|
||||||
|
$ConfigPath = Join-Path $DataDir 'wsb-collector.json'
|
||||||
|
$WindowsPowerShell = Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\powershell.exe'
|
||||||
|
|
||||||
|
# Check Windows Server Backup is here before changing anything
|
||||||
|
if (-not (Get-Module -ListAvailable -Name WindowsServerBackup)) {
|
||||||
|
throw 'The WindowsServerBackup PowerShell module is not available. Install the Windows Server Backup feature (Install-WindowsFeature Windows-Server-Backup).'
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($ApiUrl -notmatch '^https://') {
|
||||||
|
Write-Warning 'ApiUrl is not HTTPS: the API key would be sent in clear text.'
|
||||||
|
}
|
||||||
|
|
||||||
|
# Read the key as a SecureString so it isn't echoed to the console or kept in shell history
|
||||||
|
$secure = Read-Host -AsSecureString 'API key (from: python manage.py create-key <name>)'
|
||||||
|
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
|
||||||
|
try { $apiKey = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) }
|
||||||
|
finally { [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr) }
|
||||||
|
if (-not $apiKey.StartsWith('bdk_')) { throw "That doesn't look like a dashboard API key (expected 'bdk_...')." }
|
||||||
|
|
||||||
|
# Config folder: SYSTEM + Administrators only (SIDs so this works on non-English Windows)
|
||||||
|
New-Item -ItemType Directory -Path $DataDir -Force | Out-Null
|
||||||
|
& icacls.exe $DataDir /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' | Out-Null
|
||||||
|
if ($LASTEXITCODE) { throw "icacls failed on $DataDir" }
|
||||||
|
|
||||||
|
$config = [ordered]@{
|
||||||
|
ApiUrl = $ApiUrl.TrimEnd('/')
|
||||||
|
ApiKey = $apiKey
|
||||||
|
Tenant = $Tenant
|
||||||
|
SourceSystem = $SourceSystem
|
||||||
|
LookbackHours = $LookbackHours
|
||||||
|
}
|
||||||
|
if ($JobName) { $config.JobName = $JobName }
|
||||||
|
$config | ConvertTo-Json | Set-Content -Path $ConfigPath -Encoding UTF8
|
||||||
|
|
||||||
|
New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null
|
||||||
|
$scriptPath = Join-Path $InstallDir 'Send-WsbBackupStatus.ps1'
|
||||||
|
Copy-Item -Path (Join-Path $PSScriptRoot 'Send-WsbBackupStatus.ps1') -Destination $scriptPath -Force
|
||||||
|
Unblock-File -Path $scriptPath
|
||||||
|
|
||||||
|
$action = New-ScheduledTaskAction -Execute $WindowsPowerShell `
|
||||||
|
-Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$scriptPath`""
|
||||||
|
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date).AddMinutes(1) `
|
||||||
|
-RepetitionInterval (New-TimeSpan -Minutes $IntervalMinutes)
|
||||||
|
$principal = New-ScheduledTaskPrincipal -UserId 'S-1-5-18' -LogonType ServiceAccount -RunLevel Highest
|
||||||
|
$settings = New-ScheduledTaskSettingsSet -ExecutionTimeLimit (New-TimeSpan -Minutes 30) `
|
||||||
|
-MultipleInstances IgnoreNew -StartWhenAvailable
|
||||||
|
Register-ScheduledTask -TaskName $TaskName -Action $action -Trigger $trigger -Principal $principal `
|
||||||
|
-Settings $settings -Description 'Sends Windows Server Backup results to the Backup Dashboard' -Force | Out-Null
|
||||||
|
|
||||||
|
Write-Host "Installed. Config: $ConfigPath"
|
||||||
|
Write-Host "Task '$TaskName' runs every $IntervalMinutes minutes as SYSTEM, using Windows PowerShell."
|
||||||
|
Write-Host ''
|
||||||
|
Write-Host 'Next steps:'
|
||||||
|
Write-Host " 1. Preview what will be sent: & '$WindowsPowerShell' -File '$scriptPath' -DryRun"
|
||||||
|
Write-Host " 2. Backfill 30 days of history: & '$WindowsPowerShell' -File '$scriptPath' -LookbackHours 720"
|
||||||
|
Write-Host " 3. Check the log: Get-Content '$DataDir\wsb-collector.log' -Tail 20"
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
# Windows Server Backup collector
|
||||||
|
|
||||||
|
A PowerShell script that runs on each server using Windows Server Backup (WSB) and reports its
|
||||||
|
backups to the dashboard. Each server appears as one job, named `<SERVER> - Windows Server Backup`
|
||||||
|
by default.
|
||||||
|
|
||||||
|
> **Checked against:** Server 2019 Hyper-V host backing up VMs, successful backups only.
|
||||||
|
> Not yet seen on a real server: a failed backup, a failed VM or volume, or a backup in progress.
|
||||||
|
> Those paths are written to the same object model, but if one looks wrong, run
|
||||||
|
> `Get-WsbCollectorDiagnostics.ps1` while it's in that state and send the output.
|
||||||
|
|
||||||
|
## Install (on the server, as an administrator)
|
||||||
|
|
||||||
|
Requires the Windows Server Backup feature with its PowerShell module
|
||||||
|
(`Install-WindowsFeature Windows-Server-Backup`). Runs under Windows PowerShell 5.1
|
||||||
|
(`powershell.exe`), which that module targets.
|
||||||
|
|
||||||
|
1. On the dashboard host, create a key for this server:
|
||||||
|
```bash
|
||||||
|
python manage.py create-key wsb-acme-dc01
|
||||||
|
```
|
||||||
|
2. Copy this folder to the server and run:
|
||||||
|
```powershell
|
||||||
|
.\Install-WsbCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||||
|
```
|
||||||
|
Add `-JobName "Domain Controller"` to choose the name shown on the dashboard. It prompts for the
|
||||||
|
key, then:
|
||||||
|
- writes `C:\ProgramData\BackupDashboard\wsb-collector.json` (readable only by SYSTEM and Administrators)
|
||||||
|
- copies the script to `C:\Program Files\BackupDashboard\`
|
||||||
|
- registers the scheduled task **Backup Dashboard - WSB collector** (SYSTEM, every 60 minutes)
|
||||||
|
3. Preview the payloads, then backfill 30 days so the history view is populated straight away:
|
||||||
|
```powershell
|
||||||
|
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-WsbBackupStatus.ps1' -DryRun
|
||||||
|
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-WsbBackupStatus.ps1' -LookbackHours 720
|
||||||
|
```
|
||||||
|
|
||||||
|
It can be installed alongside the other collectors: it uses its own config, state, log and task.
|
||||||
|
Re-run the installer to change settings or update the script.
|
||||||
|
|
||||||
|
## What it sends
|
||||||
|
|
||||||
|
| Dashboard | Windows Server Backup |
|
||||||
|
|---|---|
|
||||||
|
| Company | `Tenant` from the config |
|
||||||
|
| Job | One per server, ID `wsb:<computer name>`, so changing `JobName` renames rather than duplicates it |
|
||||||
|
| Run | One per backup job from `Get-WBJob`, identified by its start time (WSB jobs have no ID). Restores are ignored. `details.items` lists what was backed up: the VMs for a Hyper-V backup, otherwise volumes |
|
||||||
|
| Status | Error HRESULT → **Failed**. Success, but a detailed error or a failed item (e.g. one volume) → **Warning**. Cancelled or unrecognised → **Warning**. In progress → **Running** |
|
||||||
|
| Started / finished | Job start / end time. WSB gives these as local-time text to the minute, in the server's date format; the collector parses them with that format |
|
||||||
|
| Next scheduled | `Get-WBSummary`'s `NextBackupTime`. Empty when there's no scheduled backup |
|
||||||
|
| Message | Error description, error codes (as `0x8XXXXXXX`), failed VMs/volumes, and the failure log path |
|
||||||
|
| Size | Bytes processed across the backup's items |
|
||||||
|
|
||||||
|
Each run covers the last 48 hours (`LookbackHours`). Runs already delivered are recorded in
|
||||||
|
`wsb-collector.state.json`.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
- Log: `C:\ProgramData\BackupDashboard\wsb-collector.log`. Exit codes: `0` OK, `1` some runs
|
||||||
|
failed to send (retried next run), `2` setup failed (config or WindowsServerBackup module).
|
||||||
|
- Wrong status, times or messages: run `.\Get-WsbCollectorDiagnostics.ps1` (read-only). It prints
|
||||||
|
the real structure of what the cmdlets return, plus recent `Microsoft-Windows-Backup` events to
|
||||||
|
compare times against.
|
||||||
|
|
||||||
|
## Testing without Windows Server Backup
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
. ./tests/MockWsb.ps1 # or: . ./tests/MockWsb.ps1 -WithRunningJob
|
||||||
|
./Send-WsbBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||||
|
```
|
||||||
@@ -0,0 +1,395 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Reports Windows Server Backup (WSB) results to the Backup Dashboard.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
Runs on a server with the Windows Server Backup feature, normally as a SYSTEM scheduled task
|
||||||
|
every 60 minutes (see Install-WsbCollector.ps1). The server appears on the dashboard as one job.
|
||||||
|
Each run sends:
|
||||||
|
- the next scheduled backup (Get-WBSummary's NextBackupTime), which drives the dashboard's
|
||||||
|
Overdue flag, and
|
||||||
|
- every backup from the last LookbackHours that the dashboard hasn't already received in its
|
||||||
|
final state. A backup in progress is sent as Running until it finishes.
|
||||||
|
|
||||||
|
WSB jobs have no ID, so a run is identified by its start time. Already-reported runs are
|
||||||
|
tracked in a state file, and the ingest API is idempotent, so a failed or repeated run is
|
||||||
|
harmless: anything not confirmed is retried next time.
|
||||||
|
|
||||||
|
Written for Windows PowerShell 5.1, which the WindowsServerBackup module targets.
|
||||||
|
|
||||||
|
.PARAMETER ConfigPath
|
||||||
|
JSON config with ApiUrl, ApiKey and Tenant (optional: JobName, SourceSystem, LookbackHours).
|
||||||
|
|
||||||
|
.PARAMETER LookbackHours
|
||||||
|
Overrides the config value. Use 720 once to backfill 30 days of history.
|
||||||
|
|
||||||
|
.PARAMETER DryRun
|
||||||
|
Print the payloads instead of sending them. Doesn't touch the state or log files.
|
||||||
|
|
||||||
|
.EXAMPLE
|
||||||
|
.\Send-WsbBackupStatus.ps1 -DryRun
|
||||||
|
.EXAMPLE
|
||||||
|
.\Send-WsbBackupStatus.ps1 -LookbackHours 720
|
||||||
|
#>
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[string]$ConfigPath = (Join-Path $env:ProgramData 'BackupDashboard\wsb-collector.json'),
|
||||||
|
[int]$LookbackHours = 0,
|
||||||
|
[switch]$DryRun
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
|
||||||
|
$DataDir = Split-Path -Parent $ConfigPath
|
||||||
|
$LogPath = Join-Path $DataDir 'wsb-collector.log'
|
||||||
|
$StatePath = Join-Path $DataDir 'wsb-collector.state.json'
|
||||||
|
$MaxLogBytes = 5MB
|
||||||
|
# Get-WBJob -Previous accepts at most 100. That covers a 30-day backfill at up to 3 backups a day;
|
||||||
|
# servers backing up more often only get their latest 100 runs backfilled.
|
||||||
|
$MaxPreviousJobs = 100
|
||||||
|
$Invariant = [Globalization.CultureInfo]::InvariantCulture
|
||||||
|
|
||||||
|
# ---------- Helpers ----------
|
||||||
|
|
||||||
|
function Write-Log {
|
||||||
|
param([string]$Message, [string]$Level = 'INFO')
|
||||||
|
$line = '{0} [{1}] {2}' -f (Get-Date).ToString('yyyy-MM-dd HH:mm:ss'), $Level, $Message
|
||||||
|
Write-Host $line
|
||||||
|
if ($DryRun) { return }
|
||||||
|
if ((Test-Path $LogPath) -and (Get-Item $LogPath).Length -gt $MaxLogBytes) {
|
||||||
|
Move-Item $LogPath "$LogPath.1" -Force
|
||||||
|
}
|
||||||
|
Add-Content -Path $LogPath -Value $line -Encoding UTF8
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-ErrorText($ErrorRecord) {
|
||||||
|
# Windows PowerShell puts the HTTP response body (e.g. the API's 422 explanation) here
|
||||||
|
if ($ErrorRecord.ErrorDetails -and $ErrorRecord.ErrorDetails.Message) {
|
||||||
|
return "$($ErrorRecord.Exception.Message) $($ErrorRecord.ErrorDetails.Message)"
|
||||||
|
}
|
||||||
|
return $ErrorRecord.Exception.Message
|
||||||
|
}
|
||||||
|
|
||||||
|
# The API needs an explicit UTC offset. DateTimeOffset keeps a Local/Unspecified time as local and a
|
||||||
|
# Utc time as +00:00.
|
||||||
|
function ConvertTo-IsoTime($Value) {
|
||||||
|
if ($null -eq $Value) { return $null }
|
||||||
|
$dt = [datetime]$Value
|
||||||
|
if ($dt.Year -le 1900) { return $null } # "not set" (DateTime.MinValue and friends)
|
||||||
|
return ([DateTimeOffset]$dt).ToString("yyyy-MM-dd'T'HH:mm:sszzz", $Invariant)
|
||||||
|
}
|
||||||
|
|
||||||
|
# First property of $Object, out of $Names, that exists and isn't empty. The WSB object model isn't
|
||||||
|
# fully documented, so reads go through this rather than assuming one spelling.
|
||||||
|
function Get-FirstValue($Object, [string[]]$Names) {
|
||||||
|
if ($null -eq $Object) { return $null }
|
||||||
|
foreach ($name in $Names) {
|
||||||
|
try {
|
||||||
|
$prop = $Object.PSObject.Properties[$name]
|
||||||
|
if (-not $prop) { continue }
|
||||||
|
$value = $prop.Value
|
||||||
|
# Check emptiness by type: casting to [string] would make a one-item list of objects ''
|
||||||
|
if ($null -eq $value) { continue }
|
||||||
|
if ($value -is [string] -and $value -eq '') { continue }
|
||||||
|
if ($value -is [Collections.ICollection] -and $value.Count -eq 0) { continue }
|
||||||
|
return $value
|
||||||
|
} catch { }
|
||||||
|
}
|
||||||
|
return $null
|
||||||
|
}
|
||||||
|
|
||||||
|
function Import-Wsb {
|
||||||
|
if (Get-Command Get-WBJob -ErrorAction SilentlyContinue) { return } # already loaded
|
||||||
|
# Present only when the Windows Server Backup feature (with its command-line tools) is installed
|
||||||
|
Import-Module WindowsServerBackup
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------- Jobs ----------
|
||||||
|
|
||||||
|
# WBJob times are strings in local time, to the minute, formatted with the current culture
|
||||||
|
# ("10/6/2026 9:00 PM" on en-US). Parse with that same culture: PowerShell's [datetime] cast would
|
||||||
|
# use the invariant (US) format and misread dd/MM dates. Empty while not set (e.g. EndTime of a
|
||||||
|
# running backup).
|
||||||
|
function Get-JobTime($Job, [string]$Field) {
|
||||||
|
$value = Get-FirstValue $Job @($Field)
|
||||||
|
if ($null -eq $value) { return $null }
|
||||||
|
if ($value -is [datetime]) {
|
||||||
|
$dt = $value
|
||||||
|
} else {
|
||||||
|
$dt = [datetime]::MinValue
|
||||||
|
if (-not [datetime]::TryParse([string]$value, [Globalization.CultureInfo]::CurrentCulture,
|
||||||
|
[Globalization.DateTimeStyles]::AssumeLocal, [ref]$dt)) {
|
||||||
|
Write-Log "Couldn't parse $Field '$value'" 'WARN'
|
||||||
|
return $null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if ($dt.Year -le 1900) { return $null }
|
||||||
|
return $dt
|
||||||
|
}
|
||||||
|
|
||||||
|
# HRESULTs as numbers; 0 = OK. $null when the object doesn't have the property. WSB returns them
|
||||||
|
# as strings ("0"); PowerShell's conversion also accepts hex ("0x80042302") and negative decimals.
|
||||||
|
function Get-HResult($Object, [string]$Field) {
|
||||||
|
$value = Get-FirstValue $Object @($Field)
|
||||||
|
if ($null -eq $value) { return $null }
|
||||||
|
try { return [long]$value } catch { return $null }
|
||||||
|
}
|
||||||
|
|
||||||
|
function Format-HResult([long]$Value) {
|
||||||
|
if ($Value -lt 0) { $Value += 4294967296 } # show as the familiar unsigned 0x8XXXXXXX
|
||||||
|
return '0x{0:X8}' -f $Value
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-JobItems($Job) {
|
||||||
|
return @(Get-FirstValue $Job @('JobItems', 'Items') | Where-Object { $_ })
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-ItemName($Item) {
|
||||||
|
$name = Get-FirstValue $Item @('Name', 'VolumeLabel', 'MountPath', 'Path', 'Type')
|
||||||
|
if ($name) { return [string]$name }
|
||||||
|
return 'Item'
|
||||||
|
}
|
||||||
|
|
||||||
|
# The things actually backed up. Items can nest: a Hyper-V backup has one VirtualMachineList item
|
||||||
|
# whose SubItemList holds each VM (and "Host Component").
|
||||||
|
function Get-LeafItems($Items) {
|
||||||
|
foreach ($item in $Items) {
|
||||||
|
$subItems = @(Get-FirstValue $item @('SubItemList') | Where-Object { $_ })
|
||||||
|
if ($subItems.Count) { Get-LeafItems $subItems } else { $item }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function Test-ItemFailed($Item) {
|
||||||
|
$hr = Get-HResult $Item 'HResult'
|
||||||
|
$state = [string](Get-FirstValue $Item @('State', 'JobState'))
|
||||||
|
return ($hr -and $hr -ne 0) -or $state -match 'Fail'
|
||||||
|
}
|
||||||
|
|
||||||
|
# Items that reported an error: failed leaves, or a parent that counts failed sub-items without
|
||||||
|
# saying which
|
||||||
|
function Get-FailedItems($Job) {
|
||||||
|
$items = Get-JobItems $Job
|
||||||
|
$failed = @(Get-LeafItems $items | Where-Object { Test-ItemFailed $_ })
|
||||||
|
if ($failed.Count) { return $failed }
|
||||||
|
return @($items | Where-Object {
|
||||||
|
(Test-ItemFailed $_) -or ((Get-FirstValue $_ @('SubItemFailed')) -and [long](Get-FirstValue $_ @('SubItemFailed')) -gt 0)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
# Bytes processed, from the top-level items (sub-items are already counted in their parent)
|
||||||
|
function Get-BytesProcessed($Job) {
|
||||||
|
$sum = [long]0
|
||||||
|
foreach ($item in Get-JobItems $Job) {
|
||||||
|
$value = Get-FirstValue $item @('BytesProcessed')
|
||||||
|
if ($value) { $sum += [long]$value }
|
||||||
|
}
|
||||||
|
return $sum
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-RunStatus($Job) {
|
||||||
|
$state = [string](Get-FirstValue $Job @('JobState', 'State'))
|
||||||
|
if (-not (Get-JobTime $Job 'EndTime') -and $state -notmatch 'Complet|Fail|Cancel|Abort|Stop|Succe') {
|
||||||
|
return 'running'
|
||||||
|
}
|
||||||
|
if ($state -match 'Fail') { return 'failed' }
|
||||||
|
if ($state -match 'Cancel|Abort|Stop') { return 'warning' }
|
||||||
|
$hr = Get-HResult $Job 'HResult'
|
||||||
|
if ($hr -and $hr -ne 0) { return 'failed' }
|
||||||
|
# Overall success but something underneath went wrong (e.g. a volume or files skipped)
|
||||||
|
$detailed = Get-HResult $Job 'DetailedHResult'
|
||||||
|
if (($detailed -and $detailed -ne 0) -or (Get-FailedItems $Job).Count) { return 'warning' }
|
||||||
|
if (-not $state -or $state -match 'Complet|Succe') { return 'success' }
|
||||||
|
return 'warning' # finished in a state we don't recognise: surface it rather than hide it
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-RunMessage($Job, [string]$Status) {
|
||||||
|
if ($Status -eq 'running') { return '' }
|
||||||
|
$lines = New-Object 'System.Collections.Generic.List[string]'
|
||||||
|
$state = [string](Get-FirstValue $Job @('JobState', 'State'))
|
||||||
|
$description = Get-FirstValue $Job @('ErrorDescription', 'DetailedMessage')
|
||||||
|
if ($description) { $lines.Add(([string]$description).Trim()) }
|
||||||
|
|
||||||
|
$hr = Get-HResult $Job 'HResult'
|
||||||
|
$detailed = Get-HResult $Job 'DetailedHResult'
|
||||||
|
if (($hr -and $hr -ne 0) -or ($detailed -and $detailed -ne 0)) {
|
||||||
|
$codes = @()
|
||||||
|
if ($hr -and $hr -ne 0) { $codes += "error $(Format-HResult $hr)" }
|
||||||
|
if ($detailed -and $detailed -ne 0) { $codes += "detailed error $(Format-HResult $detailed)" }
|
||||||
|
$lines.Add('Windows Server Backup reported ' + ($codes -join ', ') + '.')
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach ($item in Get-FailedItems $Job) {
|
||||||
|
$itemError = Get-FirstValue $item @('ErrorDescription', 'DetailedMessage')
|
||||||
|
$itemHr = Get-HResult $item 'HResult'
|
||||||
|
$subFailed = Get-FirstValue $item @('SubItemFailed')
|
||||||
|
$line = "$(Get-ItemName $item) - " # names are often volumes like "D:", so no colon
|
||||||
|
if ($itemError) { $line += ([string]$itemError).Trim() }
|
||||||
|
elseif ($itemHr) { $line += "error $(Format-HResult $itemHr)" }
|
||||||
|
elseif ($subFailed -and [long]$subFailed -gt 0) { $line += "$subFailed sub-item(s) failed" }
|
||||||
|
else { $line += [string](Get-FirstValue $item @('State', 'JobState')) }
|
||||||
|
$lines.Add($line)
|
||||||
|
}
|
||||||
|
|
||||||
|
# WSB fills in FailureLogPath even for successful backups, so only point at it when relevant
|
||||||
|
if ($Status -ne 'success') {
|
||||||
|
$failureLog = Get-FirstValue $Job @('FailureLogPath')
|
||||||
|
if ($failureLog) { $lines.Add("Failure log: $failureLog") }
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($lines.Count -eq 0) {
|
||||||
|
if ($Status -eq 'success') { return 'Backup completed successfully.' }
|
||||||
|
return "Backup finished with state '$state'."
|
||||||
|
}
|
||||||
|
# WSB reports "Completed" even for a backup that failed with an error code, so only mention
|
||||||
|
# the state when it says something else (e.g. cancelled)
|
||||||
|
if ($state -and $Status -ne 'success' -and $state -notmatch 'Complet') { $lines.Insert(0, "Job state: $state") }
|
||||||
|
return ($lines -join "`n")
|
||||||
|
}
|
||||||
|
|
||||||
|
function ConvertTo-Run($Job) {
|
||||||
|
$status = Get-RunStatus $Job
|
||||||
|
$details = [ordered]@{ server = $env:COMPUTERNAME }
|
||||||
|
$items = @(Get-LeafItems (Get-JobItems $Job) | ForEach-Object { Get-ItemName $_ })
|
||||||
|
if ($items.Count) { $details.items = $items }
|
||||||
|
$state = Get-FirstValue $Job @('JobState', 'State')
|
||||||
|
if ($state) { $details.wsb_state = [string]$state }
|
||||||
|
$version = Get-FirstValue $Job @('VersionId')
|
||||||
|
if ($version) { $details.version_id = [string]$version }
|
||||||
|
|
||||||
|
# No external_id: WSB jobs have no ID, so the API keys the run on started_at
|
||||||
|
$run = [ordered]@{
|
||||||
|
status = $status
|
||||||
|
started_at = ConvertTo-IsoTime (Get-JobTime $Job 'StartTime')
|
||||||
|
message = Get-RunMessage $Job $status
|
||||||
|
details = $details
|
||||||
|
}
|
||||||
|
if ($status -ne 'running') {
|
||||||
|
$finished = ConvertTo-IsoTime (Get-JobTime $Job 'EndTime')
|
||||||
|
if ($finished) { $run.finished_at = $finished } # otherwise the API uses receipt time
|
||||||
|
$bytes = Get-BytesProcessed $Job
|
||||||
|
if ($bytes -gt 0) { $run.size_bytes = $bytes }
|
||||||
|
}
|
||||||
|
return $run
|
||||||
|
}
|
||||||
|
|
||||||
|
function Test-IsBackupJob($Job) {
|
||||||
|
# Get-WBJob also lists recoveries; only backups belong on the dashboard
|
||||||
|
$type = [string](Get-FirstValue $Job @('JobType', 'Type'))
|
||||||
|
return (-not $type) -or $type -match 'Backup'
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-NextRun {
|
||||||
|
try {
|
||||||
|
$summary = Get-WBSummary
|
||||||
|
return ConvertTo-IsoTime (Get-FirstValue $summary @('NextBackupTime'))
|
||||||
|
} catch {
|
||||||
|
Write-Log "Couldn't read the backup summary: $(Get-ErrorText $_)" 'WARN'
|
||||||
|
return $null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function Send-Report($Body) {
|
||||||
|
$json = $Body | ConvertTo-Json -Depth 6 -Compress
|
||||||
|
if ($DryRun) {
|
||||||
|
Write-Host $json
|
||||||
|
return
|
||||||
|
}
|
||||||
|
# Send bytes: Windows PowerShell would otherwise encode a string body as Latin-1
|
||||||
|
$params = @{
|
||||||
|
Uri = "$($Config.ApiUrl.TrimEnd('/'))/api/v1/ingest"
|
||||||
|
Method = 'Post'
|
||||||
|
Headers = @{ Authorization = "Bearer $($Config.ApiKey)" }
|
||||||
|
ContentType = 'application/json; charset=utf-8'
|
||||||
|
Body = [Text.Encoding]::UTF8.GetBytes($json)
|
||||||
|
TimeoutSec = 30
|
||||||
|
}
|
||||||
|
Invoke-RestMethod @params | Out-Null
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------- Main ----------
|
||||||
|
|
||||||
|
try {
|
||||||
|
$Config = Get-Content -Path $ConfigPath -Raw | ConvertFrom-Json
|
||||||
|
foreach ($key in 'ApiUrl', 'ApiKey', 'Tenant') {
|
||||||
|
if (-not $Config.$key) { throw "Config $ConfigPath is missing '$key'" }
|
||||||
|
}
|
||||||
|
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Windows Server Backup' }
|
||||||
|
$JobName = if ($Config.JobName) { [string]$Config.JobName } else { "$env:COMPUTERNAME - Windows Server Backup" }
|
||||||
|
if ($LookbackHours -le 0) {
|
||||||
|
$LookbackHours = if ($Config.LookbackHours) { [int]$Config.LookbackHours } else { 48 }
|
||||||
|
}
|
||||||
|
|
||||||
|
# Windows PowerShell 5.1 may default to TLS 1.0
|
||||||
|
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
|
||||||
|
|
||||||
|
Import-Wsb
|
||||||
|
|
||||||
|
# Run start time (ISO, as sent) -> Unix seconds, for runs already delivered in their final state
|
||||||
|
$sent = @{}
|
||||||
|
if (Test-Path $StatePath) {
|
||||||
|
$saved = Get-Content -Path $StatePath -Raw | ConvertFrom-Json
|
||||||
|
if ($saved) { foreach ($p in $saved.PSObject.Properties) { $sent[$p.Name] = [long]$p.Value } }
|
||||||
|
}
|
||||||
|
|
||||||
|
$since = (Get-Date).AddHours(-$LookbackHours)
|
||||||
|
# Get-WBJob with no arguments returns the job in progress (if any); -Previous lists finished ones
|
||||||
|
$jobs = @(@(Get-WBJob) + @(Get-WBJob -Previous $MaxPreviousJobs) | Where-Object {
|
||||||
|
$_ -and (Test-IsBackupJob $_) -and (Get-JobTime $_ 'StartTime')
|
||||||
|
})
|
||||||
|
$byStart = [ordered]@{}
|
||||||
|
foreach ($job in $jobs) {
|
||||||
|
$start = Get-JobTime $job 'StartTime'
|
||||||
|
if ($start -lt $since) { continue }
|
||||||
|
$byStart[(ConvertTo-IsoTime $start)] = $job # a job in both lists: the finished entry (later) wins
|
||||||
|
}
|
||||||
|
$pending = @($byStart.Keys | Sort-Object | Where-Object { -not $sent.ContainsKey($_) } |
|
||||||
|
ForEach-Object { $byStart[$_] })
|
||||||
|
|
||||||
|
$jobInfo = [ordered]@{
|
||||||
|
# Stable per server, so renaming the dashboard job via JobName doesn't create a new one
|
||||||
|
external_id = "wsb:$($env:COMPUTERNAME.ToLowerInvariant())"
|
||||||
|
name = $JobName
|
||||||
|
# Always sent (null when unscheduled) so the dashboard's value tracks WSB's
|
||||||
|
next_run_at = Get-NextRun
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
Write-Log "Collector setup failed: $(Get-ErrorText $_)" 'ERROR'
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
$failures = 0
|
||||||
|
$runsSent = 0
|
||||||
|
try {
|
||||||
|
if ($pending.Count -eq 0) {
|
||||||
|
# Keep next_run_at fresh even when nothing new ran
|
||||||
|
Send-Report ([ordered]@{ tenant = $Config.Tenant; source_system = $SourceSystem; job = $jobInfo })
|
||||||
|
}
|
||||||
|
foreach ($job in $pending) {
|
||||||
|
$run = ConvertTo-Run $job
|
||||||
|
Send-Report ([ordered]@{ tenant = $Config.Tenant; source_system = $SourceSystem; job = $jobInfo; run = $run })
|
||||||
|
$runsSent++
|
||||||
|
if ($run.status -ne 'running') {
|
||||||
|
$sent[$run.started_at] = ([DateTimeOffset](Get-JobTime $job 'StartTime')).ToUnixTimeSeconds()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
$failures++
|
||||||
|
Write-Log "Failed to report: $(Get-ErrorText $_)" 'ERROR'
|
||||||
|
}
|
||||||
|
|
||||||
|
if (-not $DryRun) {
|
||||||
|
# Forget runs that have aged out of the lookback window (plus a day of slack)
|
||||||
|
$cutoff = [DateTimeOffset]::Now.AddHours(-$LookbackHours - 24).ToUnixTimeSeconds()
|
||||||
|
$keep = [ordered]@{}
|
||||||
|
foreach ($key in $sent.Keys) {
|
||||||
|
if ($sent[$key] -ge $cutoff) { $keep[$key] = $sent[$key] }
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
$keep | ConvertTo-Json -Compress | Set-Content -Path $StatePath -Encoding UTF8
|
||||||
|
} catch {
|
||||||
|
Write-Log "Couldn't save state file: $(Get-ErrorText $_)" 'WARN'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Write-Log ("Reported {0} runs, {1} failures" -f $runsSent, $failures)
|
||||||
|
if ($failures) { exit 1 }
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
# Stand-ins for the WindowsServerBackup cmdlets. Object shapes follow Get-WsbCollectorDiagnostics.ps1
|
||||||
|
# output from a real Hyper-V host (Server 2019): job times are culture-formatted local-time strings
|
||||||
|
# to the minute ("10/6/2026 9:00 PM" on en-US), HRESULTs are strings, and a Hyper-V backup has one
|
||||||
|
# VirtualMachineList item whose SubItemList holds the VMs. When nothing is running, Get-WBJob
|
||||||
|
# returns an empty job (JobType None).
|
||||||
|
#
|
||||||
|
# . ./tests/MockWsb.ps1 # nothing running
|
||||||
|
# . ./tests/MockWsb.ps1 -WithRunningJob # a backup in progress
|
||||||
|
# ./Send-WsbBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||||
|
param([switch]$WithRunningJob)
|
||||||
|
|
||||||
|
# Fixed to the top of the hour so repeated runs see the same jobs
|
||||||
|
$now = Get-Date -Minute 0 -Second 0 -Millisecond 0
|
||||||
|
|
||||||
|
# Same format as the real cmdlet: short date + short time in the current culture
|
||||||
|
function Format-WbTime($Value) {
|
||||||
|
if ($null -eq $Value) { return '' }
|
||||||
|
return ([datetime]$Value).ToString('g', [Globalization.CultureInfo]::CurrentCulture)
|
||||||
|
}
|
||||||
|
|
||||||
|
function New-MockItem($Name, $Type = 'Volume', [long]$Bytes = 0, [string]$HResult = '0', $ErrorDescription = '', $SubItems = $null) {
|
||||||
|
$failedSubs = @($SubItems | Where-Object { $_ -and $_.HResult -ne '0' }).Count
|
||||||
|
[pscustomobject]@{
|
||||||
|
Name = $Name
|
||||||
|
Type = $Type
|
||||||
|
State = $(if ($HResult -ne '0') { 'Failed' } else { 'Completed' })
|
||||||
|
HResult = $HResult
|
||||||
|
DetailedHResult = '0'
|
||||||
|
ErrorDescription = $ErrorDescription
|
||||||
|
BytesProcessed = $Bytes
|
||||||
|
TotalBytes = $Bytes
|
||||||
|
CurrentItem = $null
|
||||||
|
SubItemProcessed = [long]@($SubItems | Where-Object { $_ }).Count
|
||||||
|
SubItemFailed = [long]$failedSubs
|
||||||
|
TotalSubItem = [long]@($SubItems | Where-Object { $_ }).Count
|
||||||
|
SubItemList = $SubItems
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function New-VmList([long]$VmBytes, [string]$VmHResult = '0', $VmError = '') {
|
||||||
|
$vms = @(
|
||||||
|
(New-MockItem 'AAOG-EDM-AD1' 'Component' $VmBytes $VmHResult $VmError),
|
||||||
|
(New-MockItem 'Host Component' 'Component' 447729)
|
||||||
|
)
|
||||||
|
New-MockItem 'VirtualMachineList' 'VirtualmachineList' ($VmBytes + 447729) -SubItems $vms
|
||||||
|
}
|
||||||
|
|
||||||
|
function New-MockJob($Type, $Start, $End, $State, [string]$HResult = '0', [string]$DetailedHResult = '0', $ErrorDescription = '', $Items = $null) {
|
||||||
|
[pscustomobject]@{
|
||||||
|
JobType = $Type
|
||||||
|
StartTime = Format-WbTime $Start
|
||||||
|
EndTime = Format-WbTime $End
|
||||||
|
JobState = $State
|
||||||
|
CurrentOperation = ''
|
||||||
|
HResult = $HResult
|
||||||
|
DetailedHResult = $DetailedHResult
|
||||||
|
ErrorDescription = $ErrorDescription
|
||||||
|
JobItems = $Items
|
||||||
|
VersionId = $(if ($End) { $Start.ToUniversalTime().ToString('MM/dd/yyyy-HH:mm', [Globalization.CultureInfo]::InvariantCulture) } else { '' })
|
||||||
|
SuccessLogPath = 'C:\Windows\Logs\WindowsServerBackup\Backup.log'
|
||||||
|
# Populated even for successful backups, as on the real server
|
||||||
|
FailureLogPath = 'C:\Windows\Logs\WindowsServerBackup\Backup_Error.log'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
$global:MockWsbRunning = if ($WithRunningJob) {
|
||||||
|
@(New-MockJob 'Backup' $now.AddMinutes(-10) $null 'Running' -Items @(New-VmList 10GB))
|
||||||
|
} else {
|
||||||
|
@(New-MockJob 'None' $null $null 'Unknown')
|
||||||
|
}
|
||||||
|
|
||||||
|
$global:MockWsbPrevious = @(
|
||||||
|
(New-MockJob 'Backup' $now.AddHours(-12) $now.AddHours(-11.6) 'Completed' -Items @(New-VmList 124756706036)),
|
||||||
|
# Completed overall, but one VM failed
|
||||||
|
(New-MockJob 'Backup' $now.AddHours(-36) $now.AddHours(-35.6) 'Completed' '0' '0' '' @(
|
||||||
|
New-VmList 0 '-2147024891' 'Access is denied.')),
|
||||||
|
# Failed outright: backup target not found (0x80042302)
|
||||||
|
(New-MockJob 'Backup' $now.AddHours(-30) $now.AddHours(-29.9) 'Completed' '-2147212542' '0' `
|
||||||
|
'The backup storage location could not be found.'),
|
||||||
|
# A volume-based backup, for servers that aren't Hyper-V hosts
|
||||||
|
(New-MockJob 'Backup' $now.AddHours(-20) $now.AddHours(-19.5) 'Completed' -Items @(
|
||||||
|
(New-MockItem 'C:' 'Volume' 40GB), (New-MockItem 'D:' 'Volume' 200GB))),
|
||||||
|
(New-MockJob 'Recovery' $now.AddHours(-5) $now.AddHours(-4) 'Completed'), # a restore: must be ignored
|
||||||
|
(New-MockJob 'Backup' $now.AddDays(-5) $now.AddDays(-5).AddHours(1) 'Completed') # outside 48h lookback
|
||||||
|
)
|
||||||
|
|
||||||
|
$global:MockWsbSummary = [pscustomobject]@{
|
||||||
|
NextBackupTime = $now.Date.AddHours(21).AddDays($(if ($now.Hour -ge 21) { 1 } else { 0 }))
|
||||||
|
NumberOfVersions = 1
|
||||||
|
LastSuccessfulBackupTime = $now.AddHours(-12)
|
||||||
|
LastBackupResultHR = [uint32]0
|
||||||
|
CurrentOperationStatus = $(if ($WithRunningJob) { 'BackupInProgress' } else { 'NoOperationInProgress' })
|
||||||
|
}
|
||||||
|
|
||||||
|
# Same limit as the real cmdlet
|
||||||
|
function Get-WBJob { param([ValidateRange(1, 100)][int]$Previous) if ($PSBoundParameters.ContainsKey('Previous')) { $global:MockWsbPrevious } else { $global:MockWsbRunning } }
|
||||||
|
function Get-WBSummary { $global:MockWsbSummary }
|
||||||
|
function Get-WBPolicy { [pscustomobject]@{ PolicyName = 'Scheduled' } }
|
||||||
|
function Get-WBSchedule { param($Policy) @($global:MockWsbSummary.NextBackupTime) }
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Backup Dashboard API
|
||||||
|
After=network.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
User=backupdash
|
||||||
|
Group=backupdash
|
||||||
|
WorkingDirectory=/opt/backup-dashboard/backend
|
||||||
|
# Settings live in this file so manage.py reads exactly the same ones (see deploy/backup-dashboard.env)
|
||||||
|
EnvironmentFile=/etc/backup-dashboard/backup-dashboard.env
|
||||||
|
# Bound to localhost: only nginx can reach it, so nginx's auth can't be bypassed from outside.
|
||||||
|
# One worker is plenty at this scale and keeps SQLite writes simple.
|
||||||
|
ExecStart=/opt/backup-dashboard/venv/bin/uvicorn --factory app.main:create_app --host 127.0.0.1 --port 8000 --proxy-headers
|
||||||
|
Restart=on-failure
|
||||||
|
|
||||||
|
StateDirectory=backup-dashboard
|
||||||
|
NoNewPrivileges=true
|
||||||
|
ProtectSystem=strict
|
||||||
|
ProtectHome=true
|
||||||
|
PrivateTmp=true
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
# Install as /etc/backup-dashboard/backup-dashboard.env
|
||||||
|
# Read by both the API service and manage.py, so they always use the same database.
|
||||||
|
BACKUP_DASH_DB=/var/lib/backup-dashboard/backup-dashboard.db
|
||||||
|
BACKUP_DASH_RETENTION_DAYS=90
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
server {
|
||||||
|
listen 80;
|
||||||
|
server_name backups.example.com;
|
||||||
|
# Basic auth sends the password with every request — never serve this over plain HTTP
|
||||||
|
return 301 https://$host$request_uri;
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 443 ssl;
|
||||||
|
http2 on;
|
||||||
|
server_name backups.example.com;
|
||||||
|
|
||||||
|
# Paths as created by `certbot certonly --nginx -d backups.example.com`
|
||||||
|
ssl_certificate /etc/letsencrypt/live/backups.example.com/fullchain.pem;
|
||||||
|
ssl_certificate_key /etc/letsencrypt/live/backups.example.com/privkey.pem;
|
||||||
|
|
||||||
|
root /var/www/backup-dashboard/public;
|
||||||
|
index index.html;
|
||||||
|
|
||||||
|
# Strict CSP works because there is no inline JS/CSS
|
||||||
|
add_header Content-Security-Policy "default-src 'self'; object-src 'none'; base-uri 'self'; frame-ancestors 'none'" always;
|
||||||
|
add_header X-Content-Type-Options "nosniff" always;
|
||||||
|
add_header Referrer-Policy "no-referrer" always;
|
||||||
|
add_header Strict-Transport-Security "max-age=31536000" always;
|
||||||
|
|
||||||
|
# Dashboard users. Create with: htpasswd -B -c /etc/nginx/backup-dashboard.htpasswd <user>
|
||||||
|
# To move to Microsoft 365 sign-in later, replace these two lines with an auth_request to
|
||||||
|
# oauth2-proxy (provider=entra-id); the app itself doesn't need to change.
|
||||||
|
auth_basic "Backup Monitor";
|
||||||
|
auth_basic_user_file /etc/nginx/backup-dashboard.htpasswd;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
try_files $uri $uri/ /index.html;
|
||||||
|
}
|
||||||
|
|
||||||
|
location /api/ {
|
||||||
|
proxy_pass http://127.0.0.1:8000;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
}
|
||||||
|
|
||||||
|
# Collectors authenticate with their own API key (Authorization: Bearer …), checked by the app
|
||||||
|
location = /api/v1/ingest {
|
||||||
|
auth_basic off;
|
||||||
|
client_max_body_size 256k;
|
||||||
|
proxy_pass http://127.0.0.1:8000;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,341 @@
|
|||||||
|
:root {
|
||||||
|
color-scheme: light dark;
|
||||||
|
--bg: #f6f7f9;
|
||||||
|
--surface: #ffffff;
|
||||||
|
--surface-alt: #f0f2f5;
|
||||||
|
--border: #d9dde3;
|
||||||
|
--text: #1f2328;
|
||||||
|
--muted: #656d76;
|
||||||
|
--accent: #0969da;
|
||||||
|
|
||||||
|
/* Status palette: each has fg/bg/border so badges stay legible in both themes */
|
||||||
|
--success-fg: #116329; --success-bg: #dafbe1; --success-bd: #4ac26b;
|
||||||
|
--warning-fg: #7d4e00; --warning-bg: #fff8c5; --warning-bd: #d4a72c;
|
||||||
|
--failed-fg: #a40e26; --failed-bg: #ffebe9; --failed-bd: #ff8182;
|
||||||
|
--unknown-fg: #57606a; --unknown-bg: #eaeef2; --unknown-bd: #afb8c1;
|
||||||
|
--running-fg: #0550ae; --running-bg: #ddf4ff; --running-bd: #54aeff;
|
||||||
|
|
||||||
|
--radius: 8px;
|
||||||
|
--shadow: 0 1px 2px rgb(0 0 0 / 6%), 0 4px 12px rgb(0 0 0 / 4%);
|
||||||
|
font-family: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
|
||||||
|
font-size: 15px;
|
||||||
|
line-height: 1.45;
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (prefers-color-scheme: dark) {
|
||||||
|
:root {
|
||||||
|
--bg: #0d1117;
|
||||||
|
--surface: #161b22;
|
||||||
|
--surface-alt: #1c2128;
|
||||||
|
--border: #30363d;
|
||||||
|
--text: #e6edf3;
|
||||||
|
--muted: #8d96a0;
|
||||||
|
--accent: #4493f8;
|
||||||
|
--success-fg: #6fdd8b; --success-bg: #12261e; --success-bd: #2ea043;
|
||||||
|
--warning-fg: #f2cc60; --warning-bg: #272115; --warning-bd: #9e6a03;
|
||||||
|
--failed-fg: #ffa198; --failed-bg: #2d1214; --failed-bd: #da3633;
|
||||||
|
--unknown-fg: #9da7b3; --unknown-bg: #21262d; --unknown-bd: #484f58;
|
||||||
|
--running-fg: #79c0ff; --running-bg: #0c2d4b; --running-bd: #1f6feb;
|
||||||
|
--shadow: none;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
*, *::before, *::after { box-sizing: border-box; }
|
||||||
|
|
||||||
|
body {
|
||||||
|
margin: 0;
|
||||||
|
background: var(--bg);
|
||||||
|
color: var(--text);
|
||||||
|
}
|
||||||
|
|
||||||
|
.muted { color: var(--muted); }
|
||||||
|
|
||||||
|
.visually-hidden {
|
||||||
|
position: absolute !important;
|
||||||
|
width: 1px; height: 1px;
|
||||||
|
overflow: hidden;
|
||||||
|
clip: rect(0 0 0 0);
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---------- Layout ---------- */
|
||||||
|
.topbar {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: space-between;
|
||||||
|
gap: 1rem;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
padding: 0.75rem 1.5rem;
|
||||||
|
background: var(--surface);
|
||||||
|
border-bottom: 1px solid var(--border);
|
||||||
|
}
|
||||||
|
.topbar__title { margin: 0; font-size: 1.25rem; }
|
||||||
|
.topbar__meta { display: flex; align-items: center; gap: 0.75rem; font-size: 0.875rem; }
|
||||||
|
|
||||||
|
.container {
|
||||||
|
max-width: 1400px;
|
||||||
|
margin: 0 auto;
|
||||||
|
padding: 1.25rem 1.5rem 3rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---------- Controls ---------- */
|
||||||
|
.btn {
|
||||||
|
font: inherit;
|
||||||
|
padding: 0.4rem 0.9rem;
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: 6px;
|
||||||
|
background: var(--surface-alt);
|
||||||
|
color: var(--text);
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.btn:hover { border-color: var(--muted); }
|
||||||
|
.btn:disabled { opacity: 0.6; cursor: progress; }
|
||||||
|
.btn--icon { padding: 0.25rem 0.55rem; line-height: 1; }
|
||||||
|
|
||||||
|
.input {
|
||||||
|
font: inherit;
|
||||||
|
width: min(100%, 360px);
|
||||||
|
padding: 0.45rem 0.75rem;
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: 6px;
|
||||||
|
background: var(--surface);
|
||||||
|
color: var(--text);
|
||||||
|
}
|
||||||
|
|
||||||
|
:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
|
||||||
|
|
||||||
|
.toolbar { margin: 1rem 0; }
|
||||||
|
|
||||||
|
/* ---------- Banner ---------- */
|
||||||
|
.banner {
|
||||||
|
padding: 0.75rem 1rem;
|
||||||
|
border-radius: var(--radius);
|
||||||
|
margin-bottom: 1rem;
|
||||||
|
border: 1px solid;
|
||||||
|
}
|
||||||
|
.banner--error { background: var(--failed-bg); color: var(--failed-fg); border-color: var(--failed-bd); }
|
||||||
|
|
||||||
|
/* ---------- Summary cards ---------- */
|
||||||
|
.summary {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
|
||||||
|
gap: 0.75rem;
|
||||||
|
}
|
||||||
|
.summary-card {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
align-items: flex-start;
|
||||||
|
gap: 0.15rem;
|
||||||
|
padding: 0.9rem 1rem;
|
||||||
|
font: inherit;
|
||||||
|
text-align: left;
|
||||||
|
color: var(--text);
|
||||||
|
background: var(--surface);
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-left-width: 4px;
|
||||||
|
border-radius: var(--radius);
|
||||||
|
box-shadow: var(--shadow);
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.summary-card[aria-pressed="true"] { outline: 2px solid var(--accent); outline-offset: -1px; }
|
||||||
|
.summary-card__count { font-size: 1.75rem; font-weight: 600; font-variant-numeric: tabular-nums; }
|
||||||
|
.summary-card__label { font-size: 0.85rem; color: var(--muted); }
|
||||||
|
.summary-card--failed { border-left-color: var(--failed-bd); }
|
||||||
|
.summary-card--warning { border-left-color: var(--warning-bd); }
|
||||||
|
.summary-card--running { border-left-color: var(--running-bd); }
|
||||||
|
.summary-card--success { border-left-color: var(--success-bd); }
|
||||||
|
.summary-card--overdue { border-left-color: var(--unknown-bd); }
|
||||||
|
|
||||||
|
/* ---------- Table ---------- */
|
||||||
|
.table-wrap {
|
||||||
|
overflow-x: auto;
|
||||||
|
background: var(--surface);
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: var(--radius);
|
||||||
|
box-shadow: var(--shadow);
|
||||||
|
}
|
||||||
|
.jobs { width: 100%; border-collapse: collapse; }
|
||||||
|
.jobs th, .jobs td {
|
||||||
|
padding: 0.65rem 1rem;
|
||||||
|
text-align: left;
|
||||||
|
border-bottom: 1px solid var(--border);
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
.jobs tbody tr:last-child td { border-bottom: none; }
|
||||||
|
.jobs tbody tr:hover { background: var(--surface-alt); }
|
||||||
|
.jobs th { background: var(--surface-alt); font-size: 0.8rem; text-transform: uppercase; letter-spacing: 0.03em; }
|
||||||
|
|
||||||
|
.sort {
|
||||||
|
all: unset;
|
||||||
|
cursor: pointer;
|
||||||
|
color: var(--muted);
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
.sort::after { content: " ↕"; opacity: 0.4; }
|
||||||
|
th[aria-sort="ascending"] .sort::after { content: " ↑"; opacity: 1; }
|
||||||
|
th[aria-sort="descending"] .sort::after { content: " ↓"; opacity: 1; }
|
||||||
|
th[aria-sort="ascending"] .sort,
|
||||||
|
th[aria-sort="descending"] .sort { color: var(--text); }
|
||||||
|
|
||||||
|
.time { display: flex; flex-direction: column; }
|
||||||
|
.time__rel { font-size: 0.8rem; color: var(--muted); }
|
||||||
|
|
||||||
|
.empty { padding: 2rem; text-align: center; color: var(--muted); margin: 0; }
|
||||||
|
|
||||||
|
/* ---------- Status badges ---------- */
|
||||||
|
.badge {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 0.35rem;
|
||||||
|
padding: 0.2rem 0.65rem;
|
||||||
|
font: inherit;
|
||||||
|
font-size: 0.85rem;
|
||||||
|
font-weight: 600;
|
||||||
|
border: 1px solid;
|
||||||
|
border-radius: 999px;
|
||||||
|
}
|
||||||
|
button.badge { cursor: pointer; }
|
||||||
|
button.badge:hover { filter: brightness(0.95); }
|
||||||
|
/* Icon + text, not colour alone, so it's readable for colour-blind users */
|
||||||
|
.badge__icon { font-size: 0.75rem; }
|
||||||
|
.badge--success { color: var(--success-fg); background: var(--success-bg); border-color: var(--success-bd); }
|
||||||
|
.badge--warning { color: var(--warning-fg); background: var(--warning-bg); border-color: var(--warning-bd); }
|
||||||
|
.badge--failed { color: var(--failed-fg); background: var(--failed-bg); border-color: var(--failed-bd); }
|
||||||
|
.badge--unknown { color: var(--unknown-fg); background: var(--unknown-bg); border-color: var(--unknown-bd); }
|
||||||
|
.badge--running { color: var(--running-fg); background: var(--running-bg); border-color: var(--running-bd); }
|
||||||
|
@media (prefers-reduced-motion: no-preference) {
|
||||||
|
.badge--running .badge__icon { animation: spin 2s linear infinite; }
|
||||||
|
}
|
||||||
|
@keyframes spin { to { transform: rotate(360deg); } }
|
||||||
|
|
||||||
|
.tag-overdue {
|
||||||
|
margin-left: 0.5rem;
|
||||||
|
padding: 0.05rem 0.45rem;
|
||||||
|
font-size: 0.75rem;
|
||||||
|
font-weight: 600;
|
||||||
|
border-radius: 4px;
|
||||||
|
color: var(--failed-fg);
|
||||||
|
background: var(--failed-bg);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---------- Modal ---------- */
|
||||||
|
.modal {
|
||||||
|
width: min(720px, calc(100vw - 2rem));
|
||||||
|
max-height: calc(100vh - 4rem);
|
||||||
|
padding: 0;
|
||||||
|
color: var(--text);
|
||||||
|
background: var(--surface);
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: 12px;
|
||||||
|
}
|
||||||
|
.modal::backdrop { background: rgb(0 0 0 / 45%); }
|
||||||
|
.modal__header {
|
||||||
|
display: flex;
|
||||||
|
justify-content: space-between;
|
||||||
|
align-items: flex-start;
|
||||||
|
gap: 1rem;
|
||||||
|
padding: 1rem 1.25rem;
|
||||||
|
border-bottom: 1px solid var(--border);
|
||||||
|
}
|
||||||
|
.modal__title { margin: 0; font-size: 1.15rem; }
|
||||||
|
.modal__header p { margin: 0.15rem 0 0; }
|
||||||
|
.modal__body { padding: 1rem 1.25rem 1.25rem; }
|
||||||
|
|
||||||
|
.detail-grid {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: max-content 1fr;
|
||||||
|
gap: 0.4rem 1.25rem;
|
||||||
|
margin: 1rem 0;
|
||||||
|
}
|
||||||
|
.detail-grid dt { color: var(--muted); }
|
||||||
|
.detail-grid dd { margin: 0; overflow-wrap: anywhere; }
|
||||||
|
|
||||||
|
.detail-section-title { font-size: 0.85rem; text-transform: uppercase; color: var(--muted); margin: 1.25rem 0 0.5rem; }
|
||||||
|
|
||||||
|
.message {
|
||||||
|
margin: 0;
|
||||||
|
padding: 0.75rem 1rem;
|
||||||
|
max-height: 320px;
|
||||||
|
overflow: auto;
|
||||||
|
white-space: pre-wrap; /* preserve backup-software line breaks */
|
||||||
|
overflow-wrap: anywhere;
|
||||||
|
font-family: ui-monospace, SFMono-Regular, Consolas, monospace;
|
||||||
|
font-size: 0.85rem;
|
||||||
|
background: var(--surface-alt);
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-left-width: 4px;
|
||||||
|
border-radius: 6px;
|
||||||
|
}
|
||||||
|
.message--failed { border-left-color: var(--failed-bd); }
|
||||||
|
.message--warning { border-left-color: var(--warning-bd); }
|
||||||
|
.message--success { border-left-color: var(--success-bd); }
|
||||||
|
.message--running { border-left-color: var(--running-bd); }
|
||||||
|
|
||||||
|
.detail-status { display: flex; align-items: center; gap: 0.6rem; flex-wrap: wrap; }
|
||||||
|
|
||||||
|
.link {
|
||||||
|
all: unset;
|
||||||
|
color: var(--accent);
|
||||||
|
cursor: pointer;
|
||||||
|
text-decoration: underline;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---------- Run history ---------- */
|
||||||
|
.history-stats { margin: 0 0 0.6rem; font-size: 0.9rem; }
|
||||||
|
|
||||||
|
/* One cell per day, oldest → today. Colour + tooltip; the runs table below is the accessible detail. */
|
||||||
|
.day-strip {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(30, 1fr);
|
||||||
|
gap: 3px;
|
||||||
|
}
|
||||||
|
.day {
|
||||||
|
height: 28px;
|
||||||
|
padding: 0;
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: 3px;
|
||||||
|
background: var(--surface-alt);
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.day:disabled { cursor: default; opacity: 0.6; }
|
||||||
|
.day:not(:disabled):hover { filter: brightness(0.9); }
|
||||||
|
.day[aria-pressed="true"] { outline: 2px solid var(--accent); outline-offset: 1px; }
|
||||||
|
.day--success { background: var(--success-bd); border-color: var(--success-bd); }
|
||||||
|
.day--warning { background: var(--warning-bd); border-color: var(--warning-bd); }
|
||||||
|
.day--failed { background: var(--failed-bd); border-color: var(--failed-bd); }
|
||||||
|
.day--running { background: var(--running-bd); border-color: var(--running-bd); }
|
||||||
|
.day--unknown { background: var(--unknown-bd); border-color: var(--unknown-bd); }
|
||||||
|
.day-strip__ticks {
|
||||||
|
display: flex;
|
||||||
|
justify-content: space-between;
|
||||||
|
margin-top: 0.25rem;
|
||||||
|
font-size: 0.75rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.history-filter { margin: 0.75rem 0 0; font-size: 0.85rem; }
|
||||||
|
|
||||||
|
.runs-wrap {
|
||||||
|
margin-top: 0.75rem;
|
||||||
|
max-height: 300px;
|
||||||
|
overflow: auto;
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: 6px;
|
||||||
|
}
|
||||||
|
.runs { width: 100%; border-collapse: collapse; font-size: 0.85rem; }
|
||||||
|
.runs th, .runs td {
|
||||||
|
padding: 0.4rem 0.75rem;
|
||||||
|
text-align: left;
|
||||||
|
white-space: nowrap;
|
||||||
|
border-bottom: 1px solid var(--border);
|
||||||
|
}
|
||||||
|
.runs th {
|
||||||
|
position: sticky;
|
||||||
|
top: 0;
|
||||||
|
background: var(--surface-alt);
|
||||||
|
color: var(--muted);
|
||||||
|
font-size: 0.75rem;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
.runs tbody tr:last-child td { border-bottom: none; }
|
||||||
|
.runs tr.is-selected td { background: color-mix(in srgb, var(--accent) 16%, transparent); }
|
||||||
|
.runs .badge { font-size: 0.75rem; padding: 0.1rem 0.5rem; }
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<title>Backup Monitor</title>
|
||||||
|
<link rel="stylesheet" href="/css/styles.css">
|
||||||
|
<!-- type="module" is deferred by default; no inline JS so a strict CSP works -->
|
||||||
|
<script type="module" src="/js/app.js"></script>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header class="topbar">
|
||||||
|
<h1 class="topbar__title">Backup Monitor</h1>
|
||||||
|
<div class="topbar__meta">
|
||||||
|
<span id="last-updated" class="muted" aria-live="polite">Loading…</span>
|
||||||
|
<button id="refresh-btn" class="btn" type="button">Refresh</button>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<main class="container">
|
||||||
|
<div id="error-banner" class="banner banner--error" role="alert" hidden></div>
|
||||||
|
|
||||||
|
<!-- Summary cards double as status filters -->
|
||||||
|
<section class="summary" aria-label="Status summary and filters">
|
||||||
|
<button type="button" class="summary-card" data-filter="all" aria-pressed="true">
|
||||||
|
<span class="summary-card__count" data-count="all">–</span>
|
||||||
|
<span class="summary-card__label">All jobs</span>
|
||||||
|
</button>
|
||||||
|
<button type="button" class="summary-card summary-card--failed" data-filter="failed" aria-pressed="false">
|
||||||
|
<span class="summary-card__count" data-count="failed">–</span>
|
||||||
|
<span class="summary-card__label">Failed</span>
|
||||||
|
</button>
|
||||||
|
<button type="button" class="summary-card summary-card--warning" data-filter="warning" aria-pressed="false">
|
||||||
|
<span class="summary-card__count" data-count="warning">–</span>
|
||||||
|
<span class="summary-card__label">Warning</span>
|
||||||
|
</button>
|
||||||
|
<button type="button" class="summary-card summary-card--running" data-filter="running" aria-pressed="false">
|
||||||
|
<span class="summary-card__count" data-count="running">–</span>
|
||||||
|
<span class="summary-card__label">Running</span>
|
||||||
|
</button>
|
||||||
|
<button type="button" class="summary-card summary-card--success" data-filter="success" aria-pressed="false">
|
||||||
|
<span class="summary-card__count" data-count="success">–</span>
|
||||||
|
<span class="summary-card__label">Success</span>
|
||||||
|
</button>
|
||||||
|
<button type="button" class="summary-card summary-card--overdue" data-filter="overdue" aria-pressed="false">
|
||||||
|
<span class="summary-card__count" data-count="overdue">–</span>
|
||||||
|
<span class="summary-card__label">Overdue</span>
|
||||||
|
</button>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<div class="toolbar">
|
||||||
|
<label for="search" class="visually-hidden">Search jobs</label>
|
||||||
|
<input id="search" class="input" type="search" placeholder="Search company or job…" autocomplete="off">
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="table-wrap">
|
||||||
|
<table class="jobs">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th scope="col" aria-sort="none"><button type="button" class="sort" data-sort="company">Company</button></th>
|
||||||
|
<th scope="col" aria-sort="none"><button type="button" class="sort" data-sort="name">Backup Job</button></th>
|
||||||
|
<th scope="col" aria-sort="ascending"><button type="button" class="sort" data-sort="status">Status</button></th>
|
||||||
|
<th scope="col" aria-sort="none"><button type="button" class="sort" data-sort="lastRun">Last Run</button></th>
|
||||||
|
<th scope="col" aria-sort="none"><button type="button" class="sort" data-sort="nextRun">Next Scheduled</button></th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody id="jobs-body"></tbody>
|
||||||
|
</table>
|
||||||
|
<p id="empty-state" class="empty" hidden>No backup jobs match the current filter.</p>
|
||||||
|
</div>
|
||||||
|
</main>
|
||||||
|
|
||||||
|
<!-- Native <dialog>: built-in focus trap, Esc-to-close, and top-layer rendering -->
|
||||||
|
<dialog id="detail-dialog" class="modal" aria-labelledby="detail-title">
|
||||||
|
<div class="modal__header">
|
||||||
|
<div>
|
||||||
|
<h2 id="detail-title" class="modal__title"></h2>
|
||||||
|
<p id="detail-subtitle" class="muted"></p>
|
||||||
|
</div>
|
||||||
|
<button type="button" class="btn btn--icon" data-close aria-label="Close">✕</button>
|
||||||
|
</div>
|
||||||
|
<div id="detail-body" class="modal__body"></div>
|
||||||
|
</dialog>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
import { CONFIG } from './config.js';
|
||||||
|
|
||||||
|
export class ApiError extends Error {
|
||||||
|
constructor(message, status = null) {
|
||||||
|
super(message);
|
||||||
|
this.name = 'ApiError';
|
||||||
|
this.status = status;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function request(path) {
|
||||||
|
const controller = new AbortController();
|
||||||
|
const timer = setTimeout(() => controller.abort(), CONFIG.requestTimeoutMs);
|
||||||
|
try {
|
||||||
|
const res = await fetch(`${CONFIG.apiBaseUrl}${path}`, {
|
||||||
|
headers: { Accept: 'application/json' },
|
||||||
|
credentials: 'same-origin', // ready for cookie/session auth behind nginx
|
||||||
|
signal: controller.signal,
|
||||||
|
});
|
||||||
|
// 401 means the nginx/SSO session lapsed; a reload sends the user back through sign-in
|
||||||
|
if (res.status === 401) throw new ApiError('Your session has expired. Reload the page to sign in again', 401);
|
||||||
|
if (!res.ok) throw new ApiError(`Server returned HTTP ${res.status}`, res.status);
|
||||||
|
return await res.json();
|
||||||
|
} catch (err) {
|
||||||
|
if (err.name === 'AbortError') throw new ApiError('Request timed out');
|
||||||
|
if (err instanceof ApiError) throw err;
|
||||||
|
throw new ApiError(`Network error: ${err.message}`);
|
||||||
|
} finally {
|
||||||
|
clearTimeout(timer);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const httpApi = {
|
||||||
|
/** GET /jobs → { generated_at, jobs: Job[] } */
|
||||||
|
async listJobs() {
|
||||||
|
const data = await request('/jobs');
|
||||||
|
if (!Array.isArray(data?.jobs)) throw new ApiError('Unexpected response shape from /jobs');
|
||||||
|
return data;
|
||||||
|
},
|
||||||
|
/** GET /jobs/:id → JobDetail */
|
||||||
|
getJob(id) {
|
||||||
|
return request(`/jobs/${encodeURIComponent(id)}`);
|
||||||
|
},
|
||||||
|
/** GET /jobs/:id/runs?days=N → { job_id, since, runs: RunSummary[] } (newest first, no messages) */
|
||||||
|
listRuns(id, days) {
|
||||||
|
return request(`/jobs/${encodeURIComponent(id)}/runs?days=${encodeURIComponent(days)}`);
|
||||||
|
},
|
||||||
|
/** GET /runs/:id → Run (with message and details) */
|
||||||
|
getRun(id) {
|
||||||
|
return request(`/runs/${encodeURIComponent(id)}`);
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
// Mock is loaded lazily so it never ships in the production request path.
|
||||||
|
export async function getApi() {
|
||||||
|
if (CONFIG.useMock) {
|
||||||
|
const { mockApi } = await import('./mock.js');
|
||||||
|
return mockApi;
|
||||||
|
}
|
||||||
|
return httpApi;
|
||||||
|
}
|
||||||
@@ -0,0 +1,528 @@
|
|||||||
|
import { CONFIG } from './config.js';
|
||||||
|
import { getApi } from './api.js';
|
||||||
|
import {
|
||||||
|
parseDate, formatDateTime, formatTime, formatRelative, formatDuration, formatBytes,
|
||||||
|
} from './format.js';
|
||||||
|
|
||||||
|
const api = await getApi();
|
||||||
|
|
||||||
|
// ---------- Domain ----------
|
||||||
|
// rank drives default sort: worst first
|
||||||
|
const STATUS = {
|
||||||
|
failed: { label: 'Failed', icon: '✕', rank: 0 },
|
||||||
|
warning: { label: 'Warning', icon: '!', rank: 1 },
|
||||||
|
unknown: { label: 'Unknown', icon: '?', rank: 2 },
|
||||||
|
running: { label: 'Running', icon: '↻', rank: 3 },
|
||||||
|
success: { label: 'Success', icon: '✓', rank: 4 },
|
||||||
|
};
|
||||||
|
|
||||||
|
// Anything the backend sends that we don't recognise becomes "unknown" rather than breaking the UI
|
||||||
|
function normalizeStatus(raw) {
|
||||||
|
const s = String(raw ?? '').toLowerCase();
|
||||||
|
return Object.hasOwn(STATUS, s) ? s : 'unknown';
|
||||||
|
}
|
||||||
|
|
||||||
|
function normalizeJob(j) {
|
||||||
|
return {
|
||||||
|
id: String(j.id),
|
||||||
|
company: j.company_name ?? '',
|
||||||
|
name: j.job_name ?? '',
|
||||||
|
status: normalizeStatus(j.status),
|
||||||
|
message: j.message ?? '',
|
||||||
|
runningSince: parseDate(j.running_since),
|
||||||
|
// When the most recent run *finished*; a job that's running keeps showing its previous finish
|
||||||
|
lastRun: parseDate(j.last_run_at),
|
||||||
|
nextRun: parseDate(j.next_run_at),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// A running job isn't silently dead, so it's never overdue (a long run can pass its next slot)
|
||||||
|
const isOverdue = (job, now = Date.now()) =>
|
||||||
|
job.status !== 'running' && !!job.nextRun && job.nextRun.getTime() + CONFIG.overdueGraceMinutes * 60_000 < now;
|
||||||
|
|
||||||
|
// ---------- State ----------
|
||||||
|
const state = {
|
||||||
|
jobs: [],
|
||||||
|
filter: 'all',
|
||||||
|
search: '',
|
||||||
|
sort: { key: 'status', dir: 1 },
|
||||||
|
lastSuccess: null,
|
||||||
|
};
|
||||||
|
|
||||||
|
// ---------- DOM ----------
|
||||||
|
const $ = (sel) => document.querySelector(sel);
|
||||||
|
const dom = {
|
||||||
|
tbody: $('#jobs-body'),
|
||||||
|
empty: $('#empty-state'),
|
||||||
|
search: $('#search'),
|
||||||
|
refreshBtn: $('#refresh-btn'),
|
||||||
|
lastUpdated: $('#last-updated'),
|
||||||
|
errorBanner: $('#error-banner'),
|
||||||
|
dialog: $('#detail-dialog'),
|
||||||
|
detailTitle: $('#detail-title'),
|
||||||
|
detailSubtitle: $('#detail-subtitle'),
|
||||||
|
detailBody: $('#detail-body'),
|
||||||
|
};
|
||||||
|
|
||||||
|
// Tiny element builder. All text goes through append() → text nodes, so API data can never inject HTML.
|
||||||
|
function el(tag, props = {}, ...children) {
|
||||||
|
const node = document.createElement(tag);
|
||||||
|
for (const [k, v] of Object.entries(props)) {
|
||||||
|
if (v == null || v === false) continue;
|
||||||
|
if (k === 'class') node.className = v;
|
||||||
|
else if (k === 'dataset') Object.assign(node.dataset, v);
|
||||||
|
else node.setAttribute(k, v === true ? '' : v);
|
||||||
|
}
|
||||||
|
node.append(...children.flat().filter((c) => c != null && c !== false));
|
||||||
|
return node;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------- Rendering ----------
|
||||||
|
function statusBadge(status, { asButton = false, opensDialog = false, dataset = null, title = null } = {}) {
|
||||||
|
const { label, icon } = STATUS[status];
|
||||||
|
const content = [el('span', { class: 'badge__icon', 'aria-hidden': 'true' }, icon), label];
|
||||||
|
return asButton
|
||||||
|
? el('button', {
|
||||||
|
type: 'button',
|
||||||
|
class: `badge badge--${status}`,
|
||||||
|
'aria-haspopup': opensDialog ? 'dialog' : null,
|
||||||
|
'aria-label': `${label} – view details`,
|
||||||
|
title,
|
||||||
|
dataset,
|
||||||
|
}, content)
|
||||||
|
: el('span', { class: `badge badge--${status}`, title }, content);
|
||||||
|
}
|
||||||
|
|
||||||
|
const runningTitle = (since) => (since ? `Running since ${formatDateTime(since)}` : null);
|
||||||
|
|
||||||
|
function timeCell(date, { extra = null, emptyLabel = 'Never' } = {}) {
|
||||||
|
if (!date) return el('td', {}, el('span', { class: 'muted' }, emptyLabel));
|
||||||
|
return el('td', {},
|
||||||
|
el('time', { class: 'time', datetime: date.toISOString(), title: date.toString() },
|
||||||
|
el('span', {}, formatDateTime(date), extra),
|
||||||
|
el('span', { class: 'time__rel' }, formatRelative(date))));
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderRow(job) {
|
||||||
|
const overdueTag = isOverdue(job) ? el('span', { class: 'tag-overdue' }, 'Overdue') : null;
|
||||||
|
return el('tr', {},
|
||||||
|
el('td', {}, job.company),
|
||||||
|
el('td', {}, job.name),
|
||||||
|
el('td', {}, statusBadge(job.status, {
|
||||||
|
asButton: true, opensDialog: true, dataset: { jobId: job.id }, title: runningTitle(job.runningSince),
|
||||||
|
})),
|
||||||
|
timeCell(job.lastRun),
|
||||||
|
// No next run: unscheduled, or e.g. a Veeam copy job in immediate mode
|
||||||
|
timeCell(job.nextRun, { extra: overdueTag, emptyLabel: 'Not scheduled' }));
|
||||||
|
}
|
||||||
|
|
||||||
|
const collator = new Intl.Collator(undefined, { sensitivity: 'base', numeric: true });
|
||||||
|
|
||||||
|
function cmpDate(a, b) {
|
||||||
|
if (!a && !b) return 0;
|
||||||
|
if (!a) return 1;
|
||||||
|
if (!b) return -1;
|
||||||
|
return a - b;
|
||||||
|
}
|
||||||
|
|
||||||
|
const SORTERS = {
|
||||||
|
company: (a, b) => collator.compare(a.company, b.company),
|
||||||
|
name: (a, b) => collator.compare(a.name, b.name),
|
||||||
|
status: (a, b) => STATUS[a.status].rank - STATUS[b.status].rank,
|
||||||
|
lastRun: (a, b) => cmpDate(a.lastRun, b.lastRun),
|
||||||
|
nextRun: (a, b) => cmpDate(a.nextRun, b.nextRun),
|
||||||
|
};
|
||||||
|
|
||||||
|
function visibleJobs() {
|
||||||
|
const q = state.search.trim().toLowerCase();
|
||||||
|
const now = Date.now();
|
||||||
|
const { key, dir } = state.sort;
|
||||||
|
return state.jobs
|
||||||
|
.filter((j) => {
|
||||||
|
if (state.filter === 'overdue') return isOverdue(j, now);
|
||||||
|
return state.filter === 'all' || j.status === state.filter;
|
||||||
|
})
|
||||||
|
.filter((j) => !q || j.company.toLowerCase().includes(q) || j.name.toLowerCase().includes(q))
|
||||||
|
// Stable tie-breakers keep row order from jumping around on each refresh
|
||||||
|
.sort((a, b) => SORTERS[key](a, b) * dir || SORTERS.company(a, b) || SORTERS.name(a, b));
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderSummary() {
|
||||||
|
const now = Date.now();
|
||||||
|
const counts = { all: state.jobs.length, failed: 0, warning: 0, running: 0, success: 0, overdue: 0 };
|
||||||
|
for (const j of state.jobs) {
|
||||||
|
if (j.status in counts) counts[j.status]++;
|
||||||
|
if (isOverdue(j, now)) counts.overdue++;
|
||||||
|
}
|
||||||
|
for (const [k, v] of Object.entries(counts)) {
|
||||||
|
const node = document.querySelector(`[data-count="${k}"]`);
|
||||||
|
if (node) node.textContent = v;
|
||||||
|
}
|
||||||
|
document.querySelectorAll('.summary-card').forEach((card) => {
|
||||||
|
card.setAttribute('aria-pressed', String(card.dataset.filter === state.filter));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderSortHeaders() {
|
||||||
|
document.querySelectorAll('th[aria-sort]').forEach((th) => {
|
||||||
|
const key = th.querySelector('.sort').dataset.sort;
|
||||||
|
th.setAttribute('aria-sort',
|
||||||
|
key === state.sort.key ? (state.sort.dir === 1 ? 'ascending' : 'descending') : 'none');
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function render() {
|
||||||
|
const rows = visibleJobs();
|
||||||
|
dom.tbody.replaceChildren(...rows.map(renderRow));
|
||||||
|
dom.empty.hidden = rows.length > 0;
|
||||||
|
renderSummary();
|
||||||
|
renderSortHeaders();
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------- Data loading ----------
|
||||||
|
let inFlight = null;
|
||||||
|
let pollTimer = null;
|
||||||
|
|
||||||
|
function refresh() {
|
||||||
|
if (inFlight) return inFlight; // de-dupe overlapping manual + scheduled refreshes
|
||||||
|
dom.refreshBtn.disabled = true;
|
||||||
|
inFlight = (async () => {
|
||||||
|
try {
|
||||||
|
const data = await api.listJobs();
|
||||||
|
state.jobs = data.jobs.map(normalizeJob);
|
||||||
|
state.lastSuccess = new Date();
|
||||||
|
dom.errorBanner.hidden = true;
|
||||||
|
dom.lastUpdated.textContent = `Updated ${formatTime(state.lastSuccess)}`;
|
||||||
|
} catch (err) {
|
||||||
|
// Keep showing stale data, but make it obvious it's stale
|
||||||
|
dom.errorBanner.textContent = state.lastSuccess
|
||||||
|
? `Couldn't refresh (${err.message}). Showing data from ${formatTime(state.lastSuccess)}.`
|
||||||
|
: `Couldn't load backup status (${err.message}).`;
|
||||||
|
dom.errorBanner.hidden = false;
|
||||||
|
dom.lastUpdated.textContent = 'Update failed';
|
||||||
|
console.error(err);
|
||||||
|
} finally {
|
||||||
|
render();
|
||||||
|
dom.refreshBtn.disabled = false;
|
||||||
|
inFlight = null;
|
||||||
|
}
|
||||||
|
})();
|
||||||
|
return inFlight;
|
||||||
|
}
|
||||||
|
|
||||||
|
function schedulePoll() {
|
||||||
|
clearTimeout(pollTimer);
|
||||||
|
pollTimer = setTimeout(async () => {
|
||||||
|
await refresh();
|
||||||
|
schedulePoll();
|
||||||
|
}, CONFIG.refreshIntervalMs);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Don't poll from background tabs (e.g. a wall display that's been switched away); catch up on return
|
||||||
|
document.addEventListener('visibilitychange', () => {
|
||||||
|
if (document.hidden) clearTimeout(pollTimer);
|
||||||
|
else refresh().then(schedulePoll);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------- Detail modal ----------
|
||||||
|
// Every async result checks reqId first, so a slow response for job A can never paint over job B.
|
||||||
|
const modal = {
|
||||||
|
reqId: 0,
|
||||||
|
job: null,
|
||||||
|
detail: null,
|
||||||
|
runs: null, // null = loading, [] = none
|
||||||
|
runsError: null,
|
||||||
|
dayFilter: null, // local YYYY-MM-DD, or null for all days
|
||||||
|
selectedRunId: null, // null = the job's latest finished run (from the detail call)
|
||||||
|
runMessage: null, // { status, text, startedAt } for the selected run
|
||||||
|
runMessageError: null,
|
||||||
|
};
|
||||||
|
|
||||||
|
// Colour of a day cell: the worst thing that happened that day
|
||||||
|
const DAY_SEVERITY = ['failed', 'warning', 'unknown', 'running', 'success'];
|
||||||
|
|
||||||
|
const pad2 = (n) => String(n).padStart(2, '0');
|
||||||
|
const localDayKey = (d) => `${d.getFullYear()}-${pad2(d.getMonth() + 1)}-${pad2(d.getDate())}`;
|
||||||
|
const dayLabelFmt = new Intl.DateTimeFormat(undefined, { weekday: 'short', month: 'short', day: 'numeric' });
|
||||||
|
const dayTickFmt = new Intl.DateTimeFormat(undefined, { month: 'short', day: 'numeric' });
|
||||||
|
|
||||||
|
function normalizeRun(r) {
|
||||||
|
return {
|
||||||
|
id: String(r.id),
|
||||||
|
status: normalizeStatus(r.status),
|
||||||
|
startedAt: parseDate(r.started_at),
|
||||||
|
finishedAt: parseDate(r.finished_at),
|
||||||
|
duration: r.duration_seconds,
|
||||||
|
size: r.size_bytes,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function buildDays(runs, count) {
|
||||||
|
const today = new Date();
|
||||||
|
today.setHours(0, 0, 0, 0);
|
||||||
|
const days = [];
|
||||||
|
for (let i = count - 1; i >= 0; i--) {
|
||||||
|
const date = new Date(today);
|
||||||
|
date.setDate(date.getDate() - i);
|
||||||
|
days.push({ key: localDayKey(date), date, counts: {}, total: 0 });
|
||||||
|
}
|
||||||
|
const byKey = new Map(days.map((d) => [d.key, d]));
|
||||||
|
for (const run of runs) {
|
||||||
|
const day = byKey.get(localDayKey(run.startedAt));
|
||||||
|
if (!day) continue;
|
||||||
|
day.counts[run.status] = (day.counts[run.status] ?? 0) + 1;
|
||||||
|
day.total++;
|
||||||
|
}
|
||||||
|
for (const day of days) day.status = day.total ? DAY_SEVERITY.find((s) => day.counts[s]) : 'none';
|
||||||
|
return days;
|
||||||
|
}
|
||||||
|
|
||||||
|
function describeCounts(counts) {
|
||||||
|
return DAY_SEVERITY.filter((s) => counts[s])
|
||||||
|
.map((s) => `${counts[s]} ${STATUS[s].label.toLowerCase()}`)
|
||||||
|
.join(', ');
|
||||||
|
}
|
||||||
|
|
||||||
|
function detailRow(label, value) {
|
||||||
|
if (value == null || value === '') return [];
|
||||||
|
return [el('dt', {}, label), el('dd', {}, value)];
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderSummarySection() {
|
||||||
|
const { job, detail } = modal;
|
||||||
|
const status = detail ? normalizeStatus(detail.status) : job.status;
|
||||||
|
const runningSince = detail ? parseDate(detail.running_since) : job.runningSince;
|
||||||
|
const lastStatus = detail?.last_status ? normalizeStatus(detail.last_status) : null;
|
||||||
|
|
||||||
|
dom.detailTitle.textContent = detail?.job_name ?? job.name;
|
||||||
|
dom.detailSubtitle.textContent = detail?.company_name ?? job.company;
|
||||||
|
|
||||||
|
return el('section', {},
|
||||||
|
el('div', { class: 'detail-status' },
|
||||||
|
statusBadge(status),
|
||||||
|
runningSince ? el('span', { class: 'muted' }, `since ${formatDateTime(runningSince)}`) : null),
|
||||||
|
el('dl', { class: 'detail-grid' },
|
||||||
|
// While a job is running, its "last run" is the previous one — say how that went
|
||||||
|
status === 'running' && lastStatus ? detailRow('Previous result', STATUS[lastStatus].label) : [],
|
||||||
|
detailRow('Last finished', formatDateTime(detail ? parseDate(detail.last_run_at) : job.lastRun)),
|
||||||
|
detailRow('Next scheduled', formatDateTime(detail ? parseDate(detail.next_run_at) : job.nextRun, 'Not scheduled')),
|
||||||
|
detailRow('Duration', formatDuration(detail?.duration_seconds)),
|
||||||
|
detailRow('Size', formatBytes(detail?.size_bytes)),
|
||||||
|
detailRow('Source', detail?.source_system)));
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderHistorySection() {
|
||||||
|
const section = el('section', { id: 'detail-history' },
|
||||||
|
el('h3', { class: 'detail-section-title' }, `Last ${CONFIG.historyDays} days`));
|
||||||
|
|
||||||
|
if (modal.runsError) {
|
||||||
|
section.append(el('p', { class: 'banner banner--error' }, `Couldn't load run history: ${modal.runsError}`));
|
||||||
|
return section;
|
||||||
|
}
|
||||||
|
if (!modal.runs) {
|
||||||
|
section.append(el('p', { class: 'muted' }, 'Loading history…'));
|
||||||
|
return section;
|
||||||
|
}
|
||||||
|
|
||||||
|
const days = buildDays(modal.runs, CONFIG.historyDays);
|
||||||
|
const finished = modal.runs.filter((r) => r.finishedAt);
|
||||||
|
const ok = finished.filter((r) => r.status === 'success').length;
|
||||||
|
const totals = {};
|
||||||
|
for (const r of modal.runs) totals[r.status] = (totals[r.status] ?? 0) + 1;
|
||||||
|
|
||||||
|
section.append(
|
||||||
|
el('p', { class: 'history-stats' },
|
||||||
|
modal.runs.length
|
||||||
|
? `${modal.runs.length} runs · ${finished.length ? Math.round((ok / finished.length) * 100) : 0}% successful · ${describeCounts(totals)}`
|
||||||
|
: 'No runs recorded in this period.'),
|
||||||
|
el('div', { class: 'day-strip', role: 'group', 'aria-label': 'Daily results, oldest to newest. Select a day to filter runs.' },
|
||||||
|
days.map((d) => {
|
||||||
|
const label = `${dayLabelFmt.format(d.date)}: ${d.total ? describeCounts(d.counts) : 'no runs'}`;
|
||||||
|
return el('button', {
|
||||||
|
type: 'button',
|
||||||
|
class: `day day--${d.status}`,
|
||||||
|
title: label,
|
||||||
|
'aria-label': label,
|
||||||
|
'aria-pressed': String(modal.dayFilter === d.key),
|
||||||
|
disabled: !d.total,
|
||||||
|
dataset: { day: d.key },
|
||||||
|
});
|
||||||
|
})),
|
||||||
|
el('div', { class: 'day-strip__ticks muted', 'aria-hidden': 'true' },
|
||||||
|
el('span', {}, dayTickFmt.format(days[0].date)),
|
||||||
|
el('span', {}, 'Today')),
|
||||||
|
);
|
||||||
|
|
||||||
|
const rows = modal.dayFilter
|
||||||
|
? modal.runs.filter((r) => localDayKey(r.startedAt) === modal.dayFilter)
|
||||||
|
: modal.runs;
|
||||||
|
if (!rows.length) return section;
|
||||||
|
|
||||||
|
if (modal.dayFilter) {
|
||||||
|
const date = days.find((d) => d.key === modal.dayFilter)?.date;
|
||||||
|
section.append(el('p', { class: 'muted history-filter' },
|
||||||
|
`Showing ${dayLabelFmt.format(date)} · `,
|
||||||
|
el('button', { type: 'button', class: 'link', dataset: { day: modal.dayFilter } }, 'Show all days')));
|
||||||
|
}
|
||||||
|
|
||||||
|
section.append(el('div', { class: 'runs-wrap' },
|
||||||
|
el('table', { class: 'runs' },
|
||||||
|
el('thead', {}, el('tr', {},
|
||||||
|
['Status', 'Started', 'Finished', 'Duration', 'Size'].map((h) => el('th', { scope: 'col' }, h)))),
|
||||||
|
el('tbody', {}, rows.map((r) => el('tr', { class: r.id === modal.selectedRunId ? 'is-selected' : null },
|
||||||
|
el('td', {}, statusBadge(r.status, { asButton: true, dataset: { runId: r.id } })),
|
||||||
|
el('td', {}, formatDateTime(r.startedAt)),
|
||||||
|
el('td', {}, r.finishedAt ? formatDateTime(r.finishedAt) : el('span', { class: 'muted' }, 'In progress')),
|
||||||
|
el('td', {}, formatDuration(r.duration) ?? '—'),
|
||||||
|
el('td', {}, formatBytes(r.size) ?? '—')))))));
|
||||||
|
return section;
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderMessageSection() {
|
||||||
|
const { detail, job } = modal;
|
||||||
|
let status;
|
||||||
|
let text;
|
||||||
|
let heading = 'Message';
|
||||||
|
if (modal.selectedRunId) {
|
||||||
|
const run = modal.runs?.find((r) => r.id === modal.selectedRunId);
|
||||||
|
heading = `Message · run started ${formatDateTime(run?.startedAt)}`;
|
||||||
|
status = run?.status ?? 'unknown';
|
||||||
|
text = modal.runMessageError
|
||||||
|
? `Couldn't load this run: ${modal.runMessageError}`
|
||||||
|
: modal.runMessage?.text ?? 'Loading…';
|
||||||
|
} else {
|
||||||
|
status = detail?.last_status ? normalizeStatus(detail.last_status) : job.status;
|
||||||
|
text = detail?.message ?? job.message;
|
||||||
|
}
|
||||||
|
|
||||||
|
const extras = !modal.selectedRunId && detail?.details && typeof detail.details === 'object'
|
||||||
|
? Object.entries(detail.details).flatMap(([k, v]) =>
|
||||||
|
detailRow(k.replace(/_/g, ' '), typeof v === 'object' ? JSON.stringify(v) : String(v)))
|
||||||
|
: modal.runMessage?.extras ?? [];
|
||||||
|
|
||||||
|
return el('section', { id: 'detail-message' },
|
||||||
|
el('h3', { class: 'detail-section-title' }, heading),
|
||||||
|
el('pre', { class: `message message--${status}` }, text || 'No message reported.'),
|
||||||
|
extras.length ? el('h3', { class: 'detail-section-title' }, 'Additional details') : null,
|
||||||
|
extras.length ? el('dl', { class: 'detail-grid' }, extras) : null);
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderModal({ loading = false, error = null } = {}) {
|
||||||
|
dom.detailBody.replaceChildren(...[
|
||||||
|
renderSummarySection(),
|
||||||
|
loading ? el('p', { class: 'muted' }, 'Loading full details…') : null,
|
||||||
|
error ? el('p', { class: 'banner banner--error' }, `Couldn't load full details: ${error}`) : null,
|
||||||
|
renderMessageSection(),
|
||||||
|
renderHistorySection(),
|
||||||
|
].filter(Boolean));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Re-render just the history section, keeping the runs table's scroll position
|
||||||
|
function renderHistory() {
|
||||||
|
const wrap = dom.detailBody.querySelector('.runs-wrap');
|
||||||
|
const scrollTop = wrap?.scrollTop ?? 0;
|
||||||
|
dom.detailBody.querySelector('#detail-history')?.replaceWith(renderHistorySection());
|
||||||
|
const newWrap = dom.detailBody.querySelector('.runs-wrap');
|
||||||
|
if (newWrap) newWrap.scrollTop = scrollTop;
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderMessage() {
|
||||||
|
dom.detailBody.querySelector('#detail-message')?.replaceWith(renderMessageSection());
|
||||||
|
}
|
||||||
|
|
||||||
|
async function openDetail(jobId) {
|
||||||
|
const job = state.jobs.find((j) => j.id === jobId);
|
||||||
|
if (!job) return;
|
||||||
|
const reqId = ++modal.reqId;
|
||||||
|
Object.assign(modal, {
|
||||||
|
job, detail: null, runs: null, runsError: null, dayFilter: null,
|
||||||
|
selectedRunId: null, runMessage: null, runMessageError: null,
|
||||||
|
});
|
||||||
|
renderModal({ loading: true }); // show what we have immediately
|
||||||
|
if (!dom.dialog.open) dom.dialog.showModal();
|
||||||
|
|
||||||
|
const current = () => reqId === modal.reqId && dom.dialog.open;
|
||||||
|
const detailP = api.getJob(jobId).then(
|
||||||
|
(detail) => { if (current()) { modal.detail = detail; renderModal(); } },
|
||||||
|
(err) => { if (current()) renderModal({ error: err.message }); },
|
||||||
|
);
|
||||||
|
const runsP = api.listRuns(jobId, CONFIG.historyDays).then(
|
||||||
|
(data) => {
|
||||||
|
if (!current()) return;
|
||||||
|
modal.runs = (data.runs ?? []).map(normalizeRun).filter((r) => r.startedAt);
|
||||||
|
renderHistory();
|
||||||
|
},
|
||||||
|
(err) => { if (current()) { modal.runsError = err.message; renderHistory(); } },
|
||||||
|
);
|
||||||
|
await Promise.all([detailP, runsP]);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function selectRun(runId) {
|
||||||
|
const reqId = modal.reqId;
|
||||||
|
modal.selectedRunId = runId;
|
||||||
|
modal.runMessage = null;
|
||||||
|
modal.runMessageError = null;
|
||||||
|
renderHistory();
|
||||||
|
renderMessage();
|
||||||
|
try {
|
||||||
|
const run = await api.getRun(runId);
|
||||||
|
if (reqId !== modal.reqId || modal.selectedRunId !== runId) return;
|
||||||
|
modal.runMessage = {
|
||||||
|
text: run.message,
|
||||||
|
extras: Object.entries(run.details ?? {}).flatMap(([k, v]) =>
|
||||||
|
detailRow(k.replace(/_/g, ' '), typeof v === 'object' ? JSON.stringify(v) : String(v))),
|
||||||
|
};
|
||||||
|
} catch (err) {
|
||||||
|
if (reqId !== modal.reqId || modal.selectedRunId !== runId) return;
|
||||||
|
modal.runMessageError = err.message;
|
||||||
|
}
|
||||||
|
renderMessage();
|
||||||
|
dom.detailBody.querySelector('#detail-message')?.scrollIntoView({ block: 'nearest', behavior: 'smooth' });
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------- Events ----------
|
||||||
|
dom.tbody.addEventListener('click', (e) => {
|
||||||
|
const btn = e.target.closest('button[data-job-id]');
|
||||||
|
if (btn) openDetail(btn.dataset.jobId);
|
||||||
|
});
|
||||||
|
|
||||||
|
dom.dialog.addEventListener('click', (e) => {
|
||||||
|
// Click on the backdrop (the dialog element itself, outside its content) or the close button
|
||||||
|
if (e.target === dom.dialog || e.target.closest('[data-close]')) dom.dialog.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
dom.detailBody.addEventListener('click', (e) => {
|
||||||
|
const day = e.target.closest('button[data-day]');
|
||||||
|
if (day) {
|
||||||
|
modal.dayFilter = modal.dayFilter === day.dataset.day ? null : day.dataset.day;
|
||||||
|
renderHistory();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const run = e.target.closest('button[data-run-id]');
|
||||||
|
if (run) selectRun(run.dataset.runId);
|
||||||
|
});
|
||||||
|
|
||||||
|
document.querySelector('.summary').addEventListener('click', (e) => {
|
||||||
|
const card = e.target.closest('.summary-card');
|
||||||
|
if (!card) return;
|
||||||
|
state.filter = card.dataset.filter;
|
||||||
|
render();
|
||||||
|
});
|
||||||
|
|
||||||
|
document.querySelector('.jobs thead').addEventListener('click', (e) => {
|
||||||
|
const btn = e.target.closest('.sort');
|
||||||
|
if (!btn) return;
|
||||||
|
const key = btn.dataset.sort;
|
||||||
|
state.sort = state.sort.key === key ? { key, dir: -state.sort.dir } : { key, dir: 1 };
|
||||||
|
render();
|
||||||
|
});
|
||||||
|
|
||||||
|
dom.search.addEventListener('input', () => {
|
||||||
|
state.search = dom.search.value;
|
||||||
|
render();
|
||||||
|
});
|
||||||
|
|
||||||
|
dom.refreshBtn.addEventListener('click', () => refresh().then(schedulePoll));
|
||||||
|
|
||||||
|
// ---------- Boot ----------
|
||||||
|
refresh().then(schedulePoll);
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
// Central runtime config. Kept as a static file so ops can change it without a build.
|
||||||
|
const params = new URLSearchParams(window.location.search);
|
||||||
|
|
||||||
|
export const CONFIG = Object.freeze({
|
||||||
|
apiBaseUrl: '/api/v1',
|
||||||
|
refreshIntervalMs: 60_000,
|
||||||
|
requestTimeoutMs: 15_000,
|
||||||
|
// A job whose next scheduled run is this far in the past is flagged "Overdue".
|
||||||
|
// Catches the silent failure mode: a job stops reporting and stays green forever.
|
||||||
|
overdueGraceMinutes: 30,
|
||||||
|
// Run history shown in the job modal (the API caps this at 30)
|
||||||
|
historyDays: 30,
|
||||||
|
// ?mock=1 → use in-browser sample data, no backend needed
|
||||||
|
useMock: params.has('mock'),
|
||||||
|
});
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
const dateTimeFmt = new Intl.DateTimeFormat(undefined, { dateStyle: 'medium', timeStyle: 'short' });
|
||||||
|
const timeFmt = new Intl.DateTimeFormat(undefined, { timeStyle: 'medium' });
|
||||||
|
const relFmt = new Intl.RelativeTimeFormat(undefined, { numeric: 'auto' });
|
||||||
|
|
||||||
|
const REL_UNITS = [
|
||||||
|
['year', 31_536_000], ['month', 2_592_000], ['week', 604_800],
|
||||||
|
['day', 86_400], ['hour', 3_600], ['minute', 60],
|
||||||
|
];
|
||||||
|
|
||||||
|
export function parseDate(value) {
|
||||||
|
if (!value) return null;
|
||||||
|
const d = new Date(value);
|
||||||
|
return Number.isNaN(d.getTime()) ? null : d;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const formatDateTime = (d, emptyLabel = '—') => (d ? dateTimeFmt.format(d) : emptyLabel);
|
||||||
|
export const formatTime = (d) => timeFmt.format(d);
|
||||||
|
|
||||||
|
export function formatRelative(d, now = Date.now()) {
|
||||||
|
if (!d) return '';
|
||||||
|
const diffSec = Math.round((d.getTime() - now) / 1000);
|
||||||
|
for (const [unit, secs] of REL_UNITS) {
|
||||||
|
if (Math.abs(diffSec) >= secs) return relFmt.format(Math.round(diffSec / secs), unit);
|
||||||
|
}
|
||||||
|
return relFmt.format(diffSec, 'second');
|
||||||
|
}
|
||||||
|
|
||||||
|
export function formatDuration(totalSec) {
|
||||||
|
if (totalSec == null || Number.isNaN(Number(totalSec))) return null;
|
||||||
|
const s = Math.max(0, Math.round(totalSec));
|
||||||
|
const h = Math.floor(s / 3600);
|
||||||
|
const m = Math.floor((s % 3600) / 60);
|
||||||
|
const sec = s % 60;
|
||||||
|
return [h && `${h}h`, (h || m) && `${m}m`, `${sec}s`].filter(Boolean).join(' ');
|
||||||
|
}
|
||||||
|
|
||||||
|
export function formatBytes(bytes) {
|
||||||
|
if (bytes == null || Number.isNaN(Number(bytes))) return null;
|
||||||
|
const units = ['B', 'KB', 'MB', 'GB', 'TB', 'PB'];
|
||||||
|
let n = Number(bytes);
|
||||||
|
let i = 0;
|
||||||
|
while (n >= 1024 && i < units.length - 1) { n /= 1024; i++; }
|
||||||
|
return `${n.toFixed(i === 0 ? 0 : 1)} ${units[i]}`;
|
||||||
|
}
|
||||||
@@ -0,0 +1,129 @@
|
|||||||
|
// Sample data matching the API contract. Timestamps are relative to page load so it always looks "live".
|
||||||
|
const now = Date.now();
|
||||||
|
const MIN = 60_000;
|
||||||
|
const HR = 60 * MIN;
|
||||||
|
const DAY = 24 * HR;
|
||||||
|
const iso = (offsetMs) => new Date(now + offsetMs).toISOString();
|
||||||
|
|
||||||
|
// every_h drives the generated run history; it isn't part of the API.
|
||||||
|
const JOBS = [
|
||||||
|
{
|
||||||
|
id: 1, company_name: 'Acme Corp', job_name: 'File Server – Nightly', status: 'success', every_h: 24,
|
||||||
|
last_run_at: iso(-6 * HR), next_run_at: iso(18 * HR),
|
||||||
|
message: 'Job completed successfully. 1,284 files processed.',
|
||||||
|
source_system: 'Veeam B&R', duration_seconds: 2840, size_bytes: 128_849_018_880,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 2, company_name: 'Acme Corp', job_name: 'SQL01 – Transaction Logs', status: 'warning', every_h: 1,
|
||||||
|
last_run_at: iso(-25 * MIN), next_run_at: iso(35 * MIN),
|
||||||
|
message: 'Processing SQL01 Warning:\nFailed to truncate transaction logs for database "ERP_PROD".\nLog backup completed but truncation was skipped.',
|
||||||
|
source_system: 'Veeam B&R', duration_seconds: 312, size_bytes: 2_147_483_648,
|
||||||
|
details: { retry_count: 0, repository: 'REPO-ONSITE-01' },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// Running now; "last run" is still the previous completion
|
||||||
|
id: 7, company_name: 'Acme Corp', job_name: 'Hyper-V Hosts – Daily', status: 'running', every_h: 24,
|
||||||
|
running_since: iso(-12 * MIN), last_status: 'success',
|
||||||
|
last_run_at: iso(-DAY + 40 * MIN), next_run_at: iso(DAY - 12 * MIN),
|
||||||
|
message: 'Completed successfully. 6 VMs processed.',
|
||||||
|
source_system: 'Veeam B&R', duration_seconds: 3120, size_bytes: 412_316_860_416,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 3, company_name: 'Bluewater Dental', job_name: 'Workstations – Daily', status: 'failed', every_h: 24,
|
||||||
|
last_run_at: iso(-3 * HR), next_run_at: iso(21 * HR),
|
||||||
|
message: 'Error: Unable to connect to backup repository \\\\NAS02\\Backups.\nThe network path was not found. (0x80070035)\nJob terminated after 3 retries.',
|
||||||
|
source_system: 'Datto', duration_seconds: 95,
|
||||||
|
details: { retry_count: 3, target: '\\\\NAS02\\Backups' },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 4, company_name: 'Bluewater Dental', job_name: 'Imaging Server – Hourly', status: 'success', every_h: 1,
|
||||||
|
last_run_at: iso(-50 * MIN), next_run_at: iso(10 * MIN),
|
||||||
|
message: 'Completed.', source_system: 'Datto', duration_seconds: 420, size_bytes: 8_589_934_592,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// Green status but next run is in the past → should show as Overdue
|
||||||
|
id: 5, company_name: 'Northfield Logistics', job_name: 'Exchange – Nightly', status: 'success', every_h: 24,
|
||||||
|
last_run_at: iso(-30 * HR), next_run_at: iso(-6 * HR),
|
||||||
|
message: 'Completed successfully.', source_system: 'Acronis', duration_seconds: 5400,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// Never run / unrecognised status → rendered as Unknown
|
||||||
|
id: 6, company_name: 'Northfield Logistics', job_name: 'New NAS – Weekly', status: null,
|
||||||
|
last_run_at: null, next_run_at: iso(2 * DAY),
|
||||||
|
message: '', source_system: 'Synology Hyper Backup',
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
const MESSAGES = {
|
||||||
|
success: 'Job completed successfully.',
|
||||||
|
warning: 'Completed with warnings: 1 file skipped (file in use).',
|
||||||
|
failed: 'Error: Target repository unreachable. Job terminated.',
|
||||||
|
};
|
||||||
|
|
||||||
|
// Deterministic pseudo-random so history doesn't reshuffle on every modal open
|
||||||
|
function rng(seed) {
|
||||||
|
let x = seed;
|
||||||
|
return () => ((x = (x * 1_103_515_245 + 12_345) % 2_147_483_648) / 2_147_483_648);
|
||||||
|
}
|
||||||
|
|
||||||
|
const RUNS = new Map(); // job id → runs, newest first
|
||||||
|
for (const job of JOBS) {
|
||||||
|
const runs = [];
|
||||||
|
if (job.running_since) {
|
||||||
|
runs.push({ id: `${job.id}-r`, status: 'running', started_at: job.running_since, finished_at: null, message: 'In progress…' });
|
||||||
|
}
|
||||||
|
if (job.last_run_at && job.every_h) {
|
||||||
|
const rand = rng(job.id * 7919);
|
||||||
|
const dur = (job.duration_seconds ?? 600) * 1000;
|
||||||
|
let end = Date.parse(job.last_run_at);
|
||||||
|
for (let i = 0; end - dur > now - 30 * DAY; i++, end -= job.every_h * HR) {
|
||||||
|
const r = rand();
|
||||||
|
const status = i === 0 ? (job.last_status ?? job.status) : r < 0.03 ? 'failed' : r < 0.1 ? 'warning' : 'success';
|
||||||
|
const thisDur = dur * (0.8 + rand() * 0.4);
|
||||||
|
runs.push({
|
||||||
|
id: `${job.id}-${i}`,
|
||||||
|
status,
|
||||||
|
started_at: new Date(end - thisDur).toISOString(),
|
||||||
|
finished_at: new Date(end).toISOString(),
|
||||||
|
duration_seconds: Math.round(thisDur / 1000),
|
||||||
|
size_bytes: status === 'failed' ? null : job.size_bytes,
|
||||||
|
message: i === 0 ? job.message : MESSAGES[status],
|
||||||
|
details: i === 0 ? job.details ?? {} : {},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
RUNS.set(String(job.id), runs);
|
||||||
|
}
|
||||||
|
|
||||||
|
const delay = (ms) => new Promise((r) => setTimeout(r, ms));
|
||||||
|
const summary = ({ details, every_h, last_status, duration_seconds, size_bytes, ...rest }) => rest;
|
||||||
|
|
||||||
|
export const mockApi = {
|
||||||
|
async listJobs() {
|
||||||
|
await delay(250);
|
||||||
|
return { generated_at: new Date().toISOString(), jobs: JOBS.map(summary) };
|
||||||
|
},
|
||||||
|
async getJob(id) {
|
||||||
|
await delay(200);
|
||||||
|
const job = JOBS.find((j) => String(j.id) === String(id));
|
||||||
|
if (!job) throw new Error('Job not found');
|
||||||
|
const { every_h, ...rest } = job;
|
||||||
|
return { last_status: job.status === 'running' ? null : job.status, ...rest, details: job.details ?? {} };
|
||||||
|
},
|
||||||
|
async listRuns(id, days) {
|
||||||
|
await delay(300);
|
||||||
|
const since = now - days * DAY;
|
||||||
|
const runs = (RUNS.get(String(id)) ?? [])
|
||||||
|
.filter((r) => Date.parse(r.started_at) >= since)
|
||||||
|
.map(({ message, details, ...r }) => r);
|
||||||
|
return { job_id: id, since: new Date(since).toISOString(), runs };
|
||||||
|
},
|
||||||
|
async getRun(id) {
|
||||||
|
await delay(150);
|
||||||
|
for (const runs of RUNS.values()) {
|
||||||
|
const run = runs.find((r) => r.id === id);
|
||||||
|
if (run) return { details: {}, ...run };
|
||||||
|
}
|
||||||
|
throw new Error('Run not found');
|
||||||
|
},
|
||||||
|
};
|
||||||
Reference in New Issue
Block a user