Compare commits
20
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2fa9eea274 | ||
|
|
8d0e6dca71 | ||
|
|
118a7f978d | ||
|
|
3f407ba76e | ||
|
|
dd3e2fe885 | ||
|
|
51b97c4556 | ||
|
|
555e928e01 | ||
|
|
b625a8af65 | ||
|
|
a14b1b0dc9 | ||
|
|
5982fc9434 | ||
|
|
10181e014b | ||
|
|
b4b703ceee | ||
|
|
10bfc7a3aa | ||
|
|
960839ed76 | ||
|
|
f0b4d84c0b | ||
|
|
ba798284df | ||
|
|
9ae93cfbc2 | ||
|
|
728b5771cc | ||
|
|
f55a44f2e6 | ||
|
|
bf752a904f |
@@ -5,14 +5,15 @@ Proof of concept backup dashboard, intended to display backup jobs from multiple
|
||||
```
|
||||
collectors ──POST /api/v1/ingest (API key)──▶ ┌──────────────┐
|
||||
│ FastAPI + │
|
||||
browser ──nginx (basic auth, TLS)──▶ /api/* ─▶│ SQLite │
|
||||
└─ static files: public/ └──────────────┘
|
||||
browser ──nginx (TLS)──▶ /api/* ─────────────▶│ SQLite │
|
||||
│ └─ static files: public/ └──────────────┘
|
||||
└─ auth_request ─▶ oauth2-proxy ─▶ Microsoft Entra ID (Microsoft 365 sign-in)
|
||||
```
|
||||
|
||||
- `public/`: static frontend (no build step). Add `?mock=1` to the URL to use built-in sample data.
|
||||
- `backend/`: FastAPI API backed by a single SQLite file.
|
||||
- `deploy/`: nginx site and systemd unit.
|
||||
- `collectors/`: scripts that report results from backup products ([Veeam B&R](collectors/veeam/README.md)).
|
||||
- `collectors/`: PowerShell scripts that report results from backup products ([Veeam B&R](collectors/veeam/README.md), [Azure Backup / MARS](collectors/mars/README.md), [Windows Server Backup](collectors/wsb/README.md), [Veeam Backup for M365](collectors/vb365/README.md)). Code they share (config, logging, posting, state, installing) is in `collectors/common/BackupDashboard.Collector.psm1`.
|
||||
|
||||
## Data model
|
||||
|
||||
@@ -72,12 +73,58 @@ The OpenAPI schema is at `/api/v1/openapi.json`.
|
||||
|
||||
## Authentication
|
||||
|
||||
- **Dashboard users:** nginx basic auth over HTTPS (`deploy/nginx.conf`). Every user can see every tenant.
|
||||
*Planned:* Microsoft 365 sign-in through [oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/)
|
||||
with the Entra ID provider and nginx `auth_request`. That's an nginx-only change. The frontend
|
||||
already tells the user to reload when a session expires (HTTP 401).
|
||||
- **Dashboard users:** Microsoft 365 sign-in. nginx checks every request with
|
||||
[oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/), which signs users in with Microsoft
|
||||
Entra ID. Who may sign in is controlled in Entra (users or a group assigned to the app). Every
|
||||
signed-in user can see every tenant. Sessions last 12 hours; signing in again is usually silent.
|
||||
The dashboard shows who's signed in; **Sign out** also signs out of Microsoft in that browser,
|
||||
otherwise the next visit would sign straight back in.
|
||||
- **Collectors:** per-collector bearer keys, checked by the app. Only a SHA-256 hash is stored.
|
||||
To revoke one, run `python manage.py revoke-key NAME`.
|
||||
To revoke one, run `python manage.py revoke-key NAME`. `/api/v1/ingest` is the only path that
|
||||
doesn't need Microsoft sign-in.
|
||||
|
||||
### Microsoft 365 sign-in setup
|
||||
|
||||
**In the Microsoft Entra admin center** (entra.microsoft.com), as an admin of your tenant:
|
||||
|
||||
1. *App registrations → New registration*. Name: `Backup Dashboard`. Supported account types:
|
||||
**Accounts in this organizational directory only** (single tenant). Redirect URI: platform
|
||||
**Web**, `https://<your dashboard host>/oauth2/callback`. Register.
|
||||
2. On its *Overview*, note the **Application (client) ID** and **Directory (tenant) ID**.
|
||||
3. *Certificates & secrets → New client secret*. Copy the secret's **Value** now (it's only shown
|
||||
once). Note its expiry date: sign-in stops working when it expires, so set a reminder to
|
||||
create a new one before then.
|
||||
4. *Enterprise applications → Backup Dashboard → Properties*: set **Assignment required** to
|
||||
**Yes**. Then *Users and groups → Add user/group* and add the people (or a group) who should
|
||||
see the dashboard. Everyone else in the tenant is refused by Microsoft before reaching it.
|
||||
|
||||
No API permissions beyond the default `User.Read` are needed.
|
||||
|
||||
**On the dashboard server:**
|
||||
|
||||
1. Install oauth2-proxy 7.8 or newer from its
|
||||
[releases](https://github.com/oauth2-proxy/oauth2-proxy/releases) as `/usr/local/bin/oauth2-proxy`,
|
||||
and create its user: `sudo useradd --system --no-create-home oauth2-proxy`.
|
||||
2. Install the config and the client secret, readable only by root and oauth2-proxy:
|
||||
```bash
|
||||
sudo install -d -m 750 -g oauth2-proxy /etc/oauth2-proxy
|
||||
sudo install -m 640 -g oauth2-proxy deploy/oauth2-proxy.cfg /etc/oauth2-proxy/backup-dashboard.cfg
|
||||
sudo sh -c 'umask 027; printf %s "<client secret value>" > /etc/oauth2-proxy/client-secret' && sudo chgrp oauth2-proxy /etc/oauth2-proxy/client-secret
|
||||
```
|
||||
In `/etc/oauth2-proxy/backup-dashboard.cfg`, fill in `<TENANT_ID>`, `<APPLICATION_CLIENT_ID>`,
|
||||
`<DASHBOARD_HOST>`, and `<COOKIE_SECRET>`, generated with:
|
||||
`openssl rand -base64 32 | tr -- '+/' '-_'`
|
||||
3. Install `deploy/oauth2-proxy.service` into `/etc/systemd/system/`, then
|
||||
`sudo systemctl daemon-reload && sudo systemctl enable --now oauth2-proxy`.
|
||||
Check it with `journalctl -u oauth2-proxy -n 20`: it should say "OAuthProxy configured for
|
||||
Microsoft Entra ID".
|
||||
4. Install the new `deploy/nginx.conf` (keeping your server name and certificate paths), then
|
||||
`sudo nginx -t && sudo systemctl reload nginx`. Basic auth and its htpasswd file are no longer
|
||||
used.
|
||||
|
||||
**Troubleshooting:** sign-in problems show in `journalctl -u oauth2-proxy`. Microsoft's own
|
||||
errors (wrong redirect URI, user not assigned, expired secret) appear on its sign-in page with an
|
||||
`AADSTS…` code.
|
||||
|
||||
## Development
|
||||
|
||||
@@ -96,7 +143,7 @@ BACKUP_DASH_SERVE_STATIC=1 uvicorn --factory app.main:create_app --reload
|
||||
1. Copy the repo to `/opt/backup-dashboard`, create a venv at `/opt/backup-dashboard/venv`, and run `pip install -r backend/requirements.txt`.
|
||||
2. Copy `deploy/backup-dashboard.env` to `/etc/backup-dashboard/backup-dashboard.env`. Both the service and `manage.py` read their settings from it.
|
||||
3. Create a `backupdash` system user, then install `deploy/backup-dashboard-api.service` into `/etc/systemd/system/` and enable it. The database is created on first start.
|
||||
4. Copy `public/` to `/var/www/backup-dashboard/public`, install `deploy/nginx.conf`, get a certificate with certbot, and create users with `htpasswd -B`.
|
||||
4. Copy `public/` to `/var/www/backup-dashboard/public`, install `deploy/nginx.conf`, get a certificate with certbot, and set up Microsoft 365 sign-in (see [Microsoft 365 sign-in setup](#microsoft-365-sign-in-setup)).
|
||||
5. Run `manage.py` as the service user. It finds the database through the env file:
|
||||
```bash
|
||||
cd /opt/backup-dashboard/backend
|
||||
|
||||
Binary file not shown.
+2
-2
@@ -1,7 +1,7 @@
|
||||
"""Collector (machine) authentication.
|
||||
|
||||
Dashboard *users* are authenticated by nginx in front of the app (basic auth today,
|
||||
oauth2-proxy + Microsoft Entra ID later), so nothing here deals with people.
|
||||
Dashboard *users* are authenticated by nginx in front of the app (Microsoft 365 sign-in via
|
||||
oauth2-proxy), so nothing here deals with people.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
|
||||
@@ -25,6 +25,16 @@ def health(conn: sqlite3.Connection = Depends(db.get_db)):
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@router.get("/me")
|
||||
def me(request: Request):
|
||||
# Set by nginx from oauth2-proxy's sign-in check (nginx overwrites anything the browser sends).
|
||||
# Display only: access control happens in nginx. Empty in dev or without Microsoft sign-in.
|
||||
return {
|
||||
"email": request.headers.get("x-forwarded-email") or None,
|
||||
"name": request.headers.get("x-forwarded-preferred-username") or request.headers.get("x-forwarded-user") or None,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/jobs")
|
||||
def list_jobs(conn: sqlite3.Connection = Depends(db.get_db)):
|
||||
return {"generated_at": to_db(utcnow()), "jobs": repo.list_jobs(conn)}
|
||||
|
||||
+24
-3
@@ -5,6 +5,9 @@ from pathlib import Path
|
||||
# The modal shows this many days of run history; retention can never be shorter.
|
||||
HISTORY_DAYS = 30
|
||||
|
||||
# Shared by the systemd service (EnvironmentFile=) and manage.py, so both always use the same DB.
|
||||
DEFAULT_ENV_FILE = "/etc/backup-dashboard/backup-dashboard.env"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Settings:
|
||||
@@ -14,9 +17,27 @@ class Settings:
|
||||
serve_static: bool = False
|
||||
|
||||
|
||||
def read_env_file(path: Path) -> dict[str, str]:
|
||||
"""Parse simple KEY=VALUE lines (the subset systemd's EnvironmentFile accepts)."""
|
||||
values = {}
|
||||
try:
|
||||
lines = path.read_text().splitlines()
|
||||
except FileNotFoundError:
|
||||
return values
|
||||
for line in lines:
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, _, value = line.partition("=")
|
||||
values[key.strip()] = value.strip().strip("\"'")
|
||||
return values
|
||||
|
||||
|
||||
def load_settings() -> Settings:
|
||||
# Real environment variables win over the env file; the env file wins over dev defaults.
|
||||
env = read_env_file(Path(os.environ.get("BACKUP_DASH_ENV_FILE", DEFAULT_ENV_FILE))) | os.environ
|
||||
return Settings(
|
||||
db_path=Path(os.environ.get("BACKUP_DASH_DB", "backup-dashboard.db")),
|
||||
retention_days=max(HISTORY_DAYS, int(os.environ.get("BACKUP_DASH_RETENTION_DAYS", "90"))),
|
||||
serve_static=os.environ.get("BACKUP_DASH_SERVE_STATIC") == "1",
|
||||
db_path=Path(env.get("BACKUP_DASH_DB", "backup-dashboard.db")),
|
||||
retention_days=max(HISTORY_DAYS, int(env.get("BACKUP_DASH_RETENTION_DAYS", "90"))),
|
||||
serve_static=env.get("BACKUP_DASH_SERVE_STATIC") == "1",
|
||||
)
|
||||
|
||||
Binary file not shown.
@@ -122,3 +122,9 @@ def test_tenant_name_is_case_insensitive(client):
|
||||
body["tenant"] = "ACME CORP"
|
||||
client.post("/api/v1/ingest", json=body)
|
||||
assert len(client.get("/api/v1/jobs").json()["jobs"]) == 1
|
||||
|
||||
|
||||
def test_me_reports_signed_in_user_from_proxy_headers(client):
|
||||
assert client.get("/api/v1/me").json() == {"email": None, "name": None}
|
||||
r = client.get("/api/v1/me", headers={"X-Forwarded-Email": "[email protected]", "X-Forwarded-User": "abc-123"})
|
||||
assert r.json() == {"email": "[email protected]", "name": "abc-123"}
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from app.settings import load_settings
|
||||
|
||||
BACKEND = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def run_manage(*args, env, cwd):
|
||||
return subprocess.run([sys.executable, str(BACKEND / "manage.py"), *args],
|
||||
capture_output=True, text=True, env=env, cwd=cwd)
|
||||
|
||||
|
||||
def base_env(tmp_path, monkeypatch):
|
||||
env_file = tmp_path / "backup-dashboard.env"
|
||||
monkeypatch.setenv("BACKUP_DASH_ENV_FILE", str(env_file))
|
||||
monkeypatch.delenv("BACKUP_DASH_DB", raising=False)
|
||||
return env_file, {"BACKUP_DASH_ENV_FILE": str(env_file), "PYTHONPATH": str(BACKEND), "PATH": "/usr/bin:/bin"}
|
||||
|
||||
|
||||
def test_env_file_supplies_db_path_and_real_env_wins(tmp_path, monkeypatch):
|
||||
env_file, _ = base_env(tmp_path, monkeypatch)
|
||||
env_file.write_text('# comment\nBACKUP_DASH_DB="/srv/from-file.db"\nBACKUP_DASH_RETENTION_DAYS=120\n')
|
||||
assert load_settings().db_path == Path("/srv/from-file.db")
|
||||
assert load_settings().retention_days == 120
|
||||
monkeypatch.setenv("BACKUP_DASH_DB", "/srv/from-env.db")
|
||||
assert load_settings().db_path == Path("/srv/from-env.db")
|
||||
|
||||
|
||||
def test_manage_refuses_to_create_db_implicitly(tmp_path, monkeypatch):
|
||||
_, env = base_env(tmp_path, monkeypatch)
|
||||
workdir = tmp_path / "somewhere-else"
|
||||
workdir.mkdir()
|
||||
r = run_manage("create-key", "k", env=env, cwd=workdir)
|
||||
assert r.returncode != 0
|
||||
assert "Database not found" in r.stderr
|
||||
assert not (workdir / "backup-dashboard.db").exists()
|
||||
|
||||
|
||||
def test_manage_uses_db_from_env_file(tmp_path, monkeypatch):
|
||||
env_file, env = base_env(tmp_path, monkeypatch)
|
||||
service_db = tmp_path / "service.db"
|
||||
env_file.write_text(f"BACKUP_DASH_DB={service_db}\n")
|
||||
assert run_manage("--create", "list-keys", env=env, cwd=tmp_path).returncode == 0 # service's first start
|
||||
workdir = tmp_path / "somewhere-else"
|
||||
workdir.mkdir()
|
||||
r = run_manage("create-key", "veeam-acme", env=env, cwd=workdir)
|
||||
assert r.returncode == 0, r.stderr
|
||||
assert str(service_db) in r.stderr
|
||||
assert "veeam-acme" in run_manage("list-keys", env=env, cwd=tmp_path).stdout
|
||||
assert not (workdir / "backup-dashboard.db").exists()
|
||||
@@ -0,0 +1,270 @@
|
||||
<#
|
||||
Shared code for the Backup Dashboard collectors: config and state files, logging, posting to
|
||||
the ingest API, and installing a collector as a scheduled task.
|
||||
|
||||
Each collector imports this from next to itself (installed: C:\Program Files\BackupDashboard)
|
||||
or from ..\common (in the repo). Collectors on the same server share one installed copy, so
|
||||
keep these functions backward compatible.
|
||||
|
||||
Works on Windows PowerShell 5.1 and PowerShell 7.
|
||||
#>
|
||||
|
||||
$script:Config = $null
|
||||
$script:DryRun = $false
|
||||
$script:LogPath = $null
|
||||
$script:StatePath = $null
|
||||
$script:LookbackHours = 48
|
||||
$script:Sent = @{}
|
||||
$script:RunsSent = 0
|
||||
$script:MaxLogBytes = 5MB
|
||||
$script:Invariant = [Globalization.CultureInfo]::InvariantCulture
|
||||
|
||||
# ---------- Logging and values ----------
|
||||
|
||||
function Write-Log {
|
||||
param([string]$Message, [string]$Level = 'INFO')
|
||||
$line = '{0} [{1}] {2}' -f (Get-Date).ToString('yyyy-MM-dd HH:mm:ss'), $Level, $Message
|
||||
Write-Host $line
|
||||
if ($script:DryRun -or -not $script:LogPath) { return }
|
||||
if ((Test-Path $script:LogPath) -and (Get-Item $script:LogPath).Length -gt $script:MaxLogBytes) {
|
||||
Move-Item $script:LogPath "$($script:LogPath).1" -Force
|
||||
}
|
||||
Add-Content -Path $script:LogPath -Value $line -Encoding UTF8
|
||||
}
|
||||
|
||||
function Get-ErrorText($ErrorRecord) {
|
||||
# Windows PowerShell puts the HTTP response body (e.g. the API's 422 explanation) here
|
||||
if ($ErrorRecord.ErrorDetails -and $ErrorRecord.ErrorDetails.Message) {
|
||||
return "$($ErrorRecord.Exception.Message) $($ErrorRecord.ErrorDetails.Message)"
|
||||
}
|
||||
return $ErrorRecord.Exception.Message
|
||||
}
|
||||
|
||||
# The API needs an explicit UTC offset. DateTimeOffset keeps a Local/Unspecified time as local and a
|
||||
# Utc time as +00:00, so convert UTC-valued times to Kind Utc before calling this.
|
||||
function ConvertTo-IsoTime($Value) {
|
||||
if ($null -eq $Value) { return $null }
|
||||
$dt = [datetime]$Value
|
||||
if ($dt.Year -le 1900) { return $null } # "not set" (DateTime.MinValue, MARS's 1601-01-01, ...)
|
||||
return ([DateTimeOffset]$dt).ToString("yyyy-MM-dd'T'HH:mm:sszzz", $script:Invariant)
|
||||
}
|
||||
|
||||
# First property of $Object, out of $Names, that exists and isn't empty. Backup products' object
|
||||
# models aren't fully documented, so reads go through this rather than assuming one spelling.
|
||||
function Get-FirstValue($Object, [string[]]$Names) {
|
||||
if ($null -eq $Object) { return $null }
|
||||
foreach ($name in $Names) {
|
||||
try {
|
||||
$prop = $Object.PSObject.Properties[$name]
|
||||
if (-not $prop) { continue }
|
||||
$value = $prop.Value
|
||||
# Check emptiness by type: casting to [string] would make a one-item list of objects ''
|
||||
if ($null -eq $value) { continue }
|
||||
if ($value -is [string] -and $value -eq '') { continue }
|
||||
if ($value -is [Collections.ICollection] -and $value.Count -eq 0) { continue }
|
||||
return $value
|
||||
} catch { }
|
||||
}
|
||||
return $null
|
||||
}
|
||||
|
||||
# ---------- Collector run ----------
|
||||
|
||||
<#
|
||||
Loads the config ($ConfigPath, JSON) and the record of runs already sent, and sets up logging
|
||||
and TLS. Log and state files sit next to the config: <Name>-collector.log / .state.json.
|
||||
Returns the config object; its LookbackHours is resolved (the parameter, else the config, else
|
||||
48). Throws if the config can't be read or lacks one of $RequiredKeys.
|
||||
#>
|
||||
function Initialize-Collector {
|
||||
param(
|
||||
[Parameter(Mandatory)][string]$Name,
|
||||
[Parameter(Mandatory)][string]$ConfigPath,
|
||||
[int]$LookbackHours = 0,
|
||||
[switch]$DryRun,
|
||||
[string[]]$RequiredKeys = @('ApiUrl', 'ApiKey', 'Tenant')
|
||||
)
|
||||
# Logging first, so a bad config can still be logged
|
||||
$dataDir = Split-Path -Parent $ConfigPath
|
||||
$script:LogPath = Join-Path $dataDir "$Name-collector.log"
|
||||
$script:StatePath = Join-Path $dataDir "$Name-collector.state.json"
|
||||
$script:DryRun = [bool]$DryRun
|
||||
$script:RunsSent = 0
|
||||
$script:Sent = @{}
|
||||
|
||||
$config = Get-Content -Path $ConfigPath -Raw | ConvertFrom-Json
|
||||
foreach ($key in $RequiredKeys) {
|
||||
if (-not $config.$key) { throw "Config $ConfigPath is missing '$key'" }
|
||||
}
|
||||
if ($LookbackHours -le 0) {
|
||||
$LookbackHours = if ($config.LookbackHours) { [int]$config.LookbackHours } else { 48 }
|
||||
}
|
||||
$config | Add-Member -NotePropertyName LookbackHours -NotePropertyValue $LookbackHours -Force
|
||||
$script:LookbackHours = $LookbackHours
|
||||
$script:Config = $config
|
||||
|
||||
# Windows PowerShell 5.1 may default to TLS 1.0
|
||||
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
|
||||
|
||||
# Run key -> start time (Unix seconds), for runs already delivered in their final state.
|
||||
# Numbers rather than ISO strings: PowerShell 7's ConvertFrom-Json turns date strings into
|
||||
# DateTimes and loses the offset.
|
||||
if (Test-Path $script:StatePath) {
|
||||
$saved = Get-Content -Path $script:StatePath -Raw | ConvertFrom-Json
|
||||
if ($saved) { foreach ($p in $saved.PSObject.Properties) { $script:Sent[$p.Name] = [long]$p.Value } }
|
||||
}
|
||||
return $config
|
||||
}
|
||||
|
||||
function Test-RunSent([string]$Key) {
|
||||
return $script:Sent.ContainsKey($Key)
|
||||
}
|
||||
|
||||
<#
|
||||
Posts one report to the ingest API: the job alone (to refresh next_run_at) or the job with a
|
||||
run. A run that has finished is then recorded as sent under $SentKey, so later collector runs
|
||||
skip it; a running one isn't, so it's re-sent until it finishes. Throws on an HTTP error.
|
||||
#>
|
||||
function Send-JobReport {
|
||||
param(
|
||||
[Parameter(Mandatory)][string]$Tenant,
|
||||
[Parameter(Mandatory)][string]$SourceSystem,
|
||||
[Parameter(Mandatory)]$Job,
|
||||
$Run,
|
||||
[string]$SentKey,
|
||||
$StartedAt
|
||||
)
|
||||
$body = [ordered]@{ tenant = $Tenant; source_system = $SourceSystem; job = $Job }
|
||||
if ($Run) { $body.run = $Run }
|
||||
$json = $body | ConvertTo-Json -Depth 6 -Compress
|
||||
if ($script:DryRun) {
|
||||
Write-Host $json
|
||||
} else {
|
||||
# Send bytes: Windows PowerShell would otherwise encode a string body as Latin-1,
|
||||
# mangling non-ASCII job names (e.g. en dashes)
|
||||
$params = @{
|
||||
Uri = "$($script:Config.ApiUrl.TrimEnd('/'))/api/v1/ingest"
|
||||
Method = 'Post'
|
||||
Headers = @{ Authorization = "Bearer $($script:Config.ApiKey)" }
|
||||
ContentType = 'application/json; charset=utf-8'
|
||||
Body = [Text.Encoding]::UTF8.GetBytes($json)
|
||||
TimeoutSec = 30
|
||||
}
|
||||
Invoke-RestMethod @params | Out-Null
|
||||
}
|
||||
if ($Run) {
|
||||
$script:RunsSent++
|
||||
if ($Run.status -ne 'running' -and $SentKey) {
|
||||
$script:Sent[$SentKey] = ([DateTimeOffset][datetime]$StartedAt).ToUnixTimeSeconds()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# True when a failed Send-JobReport was the API rejecting the key: no point trying more jobs
|
||||
function Test-AuthRejected($ErrorRecord) {
|
||||
$status = 0
|
||||
try { $status = [int]$ErrorRecord.Exception.Response.StatusCode } catch { }
|
||||
return $status -eq 401
|
||||
}
|
||||
|
||||
<#
|
||||
Saves the sent-runs record (dropping runs older than the lookback window plus a day) and logs
|
||||
the summary line. Doesn't exit: the collector does that with the returned exit code.
|
||||
#>
|
||||
function Complete-Collector {
|
||||
param([int]$JobCount = -1, [int]$Failures = 0)
|
||||
if (-not $script:DryRun) {
|
||||
$cutoff = [DateTimeOffset]::Now.AddHours(-$script:LookbackHours - 24).ToUnixTimeSeconds()
|
||||
$keep = [ordered]@{}
|
||||
foreach ($key in $script:Sent.Keys) {
|
||||
if ($script:Sent[$key] -ge $cutoff) { $keep[$key] = $script:Sent[$key] }
|
||||
}
|
||||
try {
|
||||
$keep | ConvertTo-Json -Compress | Set-Content -Path $script:StatePath -Encoding UTF8
|
||||
} catch {
|
||||
Write-Log "Couldn't save state file: $(Get-ErrorText $_)" 'WARN'
|
||||
}
|
||||
}
|
||||
if ($JobCount -ge 0) {
|
||||
Write-Log ("Reported {0} jobs, {1} runs, {2} failures" -f $JobCount, $script:RunsSent, $Failures)
|
||||
} else {
|
||||
Write-Log ("Reported {0} runs, {1} failures" -f $script:RunsSent, $Failures)
|
||||
}
|
||||
if ($Failures) { return 1 }
|
||||
return 0
|
||||
}
|
||||
|
||||
# ---------- Installing ----------
|
||||
|
||||
<#
|
||||
The part of every collector installer after its own checks: prompt for the API key, write the
|
||||
config to ProgramData\BackupDashboard (locked to SYSTEM and Administrators, since it holds the
|
||||
key), copy the collector script and this module to $InstallDir (Program Files: only admins can
|
||||
modify what runs as SYSTEM), and register a scheduled task running it as SYSTEM.
|
||||
#>
|
||||
function Install-CollectorTask {
|
||||
param(
|
||||
[Parameter(Mandatory)][string]$Name, # 'veeam' -> veeam-collector.json / .log
|
||||
[Parameter(Mandatory)][string]$ScriptPath, # the collector script to install
|
||||
[Parameter(Mandatory)][string]$TaskName,
|
||||
[Parameter(Mandatory)][string]$TaskDescription,
|
||||
[Parameter(Mandatory)][string]$Shell, # powershell.exe or pwsh.exe, full path
|
||||
[Parameter(Mandatory)][string]$ApiUrl,
|
||||
[Parameter(Mandatory)][int]$IntervalMinutes,
|
||||
[Parameter(Mandatory)][string]$InstallDir,
|
||||
[System.Collections.IDictionary]$Settings = @{} # extra config: Tenant, LookbackHours, ...
|
||||
)
|
||||
$dataDir = Join-Path $env:ProgramData 'BackupDashboard'
|
||||
$configPath = Join-Path $dataDir "$Name-collector.json"
|
||||
|
||||
if ($ApiUrl -notmatch '^https://') {
|
||||
Write-Warning 'ApiUrl is not HTTPS: the API key would be sent in clear text.'
|
||||
}
|
||||
|
||||
# Read the key as a SecureString so it isn't echoed to the console or kept in shell history
|
||||
$secure = Read-Host -AsSecureString 'API key (from: python manage.py create-key <name>)'
|
||||
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
|
||||
try { $apiKey = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) }
|
||||
finally { [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr) }
|
||||
if (-not $apiKey.StartsWith('bdk_')) { throw "That doesn't look like a dashboard API key (expected 'bdk_...')." }
|
||||
|
||||
# Config folder: SYSTEM + Administrators only (SIDs so this works on non-English Windows)
|
||||
New-Item -ItemType Directory -Path $dataDir -Force | Out-Null
|
||||
& icacls.exe $dataDir /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' | Out-Null
|
||||
if ($LASTEXITCODE) { throw "icacls failed on $dataDir" }
|
||||
|
||||
$config = [ordered]@{ ApiUrl = $ApiUrl.TrimEnd('/'); ApiKey = $apiKey }
|
||||
foreach ($key in $Settings.Keys) { $config[$key] = $Settings[$key] }
|
||||
$config | ConvertTo-Json -Depth 4 | Set-Content -Path $configPath -Encoding UTF8
|
||||
|
||||
# The collector and this module, side by side (the collector looks for the module next to itself)
|
||||
New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null
|
||||
$installedScript = Join-Path $InstallDir (Split-Path -Leaf $ScriptPath)
|
||||
foreach ($file in $ScriptPath, $PSCommandPath) {
|
||||
$target = Join-Path $InstallDir (Split-Path -Leaf $file)
|
||||
Copy-Item -Path $file -Destination $target -Force
|
||||
Unblock-File -Path $target
|
||||
}
|
||||
|
||||
$action = New-ScheduledTaskAction -Execute $Shell `
|
||||
-Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$installedScript`""
|
||||
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date).AddMinutes(1) `
|
||||
-RepetitionInterval (New-TimeSpan -Minutes $IntervalMinutes)
|
||||
$principal = New-ScheduledTaskPrincipal -UserId 'S-1-5-18' -LogonType ServiceAccount -RunLevel Highest
|
||||
$taskSettings = New-ScheduledTaskSettingsSet -ExecutionTimeLimit (New-TimeSpan -Minutes 30) `
|
||||
-MultipleInstances IgnoreNew -StartWhenAvailable
|
||||
Register-ScheduledTask -TaskName $TaskName -Action $action -Trigger $trigger -Principal $principal `
|
||||
-Settings $taskSettings -Description $TaskDescription -Force | Out-Null
|
||||
|
||||
Write-Host "Installed. Config: $configPath"
|
||||
Write-Host "Task '$TaskName' runs every $IntervalMinutes minutes as SYSTEM, using $Shell."
|
||||
Write-Host ''
|
||||
Write-Host 'Next steps:'
|
||||
Write-Host " 1. Preview what will be sent: & '$Shell' -File '$installedScript' -DryRun"
|
||||
Write-Host " 2. Backfill 30 days of history: & '$Shell' -File '$installedScript' -LookbackHours 720"
|
||||
Write-Host " 3. Check the log: Get-Content '$dataDir\$Name-collector.log' -Tail 20"
|
||||
}
|
||||
|
||||
Export-ModuleMember -Function Write-Log, Get-ErrorText, ConvertTo-IsoTime, Get-FirstValue,
|
||||
Initialize-Collector, Test-RunSent, Send-JobReport, Test-AuthRejected, Complete-Collector,
|
||||
Install-CollectorTask
|
||||
@@ -0,0 +1,102 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Read-only dump of what the MARS agent's cmdlets return, for troubleshooting the collector.
|
||||
|
||||
.DESCRIPTION
|
||||
The collector reads properties of Get-OBJob / Get-OBPolicy / Get-OBSchedule results whose names
|
||||
aren't documented. This prints those objects' real structure (property names, types and values,
|
||||
a few levels deep) so mismatches can be fixed. Nothing is changed. Output is also saved to a
|
||||
text file.
|
||||
|
||||
.EXAMPLE
|
||||
.\Get-MarsCollectorDiagnostics.ps1
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[int]$Jobs = 3,
|
||||
[string]$OutFile = (Join-Path $env:TEMP 'mars-collector-diagnostics.txt')
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$MaxDepth = 6
|
||||
|
||||
if (-not (Get-Command Get-OBJob -ErrorAction SilentlyContinue)) {
|
||||
if (Get-Module -ListAvailable -Name MSOnlineBackup) {
|
||||
Import-Module MSOnlineBackup
|
||||
} else {
|
||||
Import-Module (Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\Modules\MSOnlineBackup')
|
||||
}
|
||||
}
|
||||
|
||||
function Test-Scalar($Value) {
|
||||
return $null -eq $Value -or $Value -is [string] -or $Value -is [ValueType] -or $Value -is [guid]
|
||||
}
|
||||
|
||||
# Recursive "name (type) = value" listing. Collections show their first 3 items.
|
||||
function Show-Object($Value, [string]$Indent = ' ', [int]$Depth = 0) {
|
||||
if (Test-Scalar $Value) { return }
|
||||
if ($Depth -ge $MaxDepth) { "$Indent..."; return }
|
||||
if ($Value -is [Collections.IEnumerable]) {
|
||||
$items = @($Value)
|
||||
"$Indent[$($items.Count) item(s)]"
|
||||
$i = 0
|
||||
foreach ($item in $items | Select-Object -First 3) {
|
||||
"$Indent[$i] ($($item.GetType().FullName))$(if (Test-Scalar $item) { " = $item" })"
|
||||
Show-Object $item "$Indent " ($Depth + 1)
|
||||
$i++
|
||||
}
|
||||
return
|
||||
}
|
||||
foreach ($p in $Value.PSObject.Properties) {
|
||||
try { $v = $p.Value } catch { "$Indent$($p.Name) = <error: $($_.Exception.Message)>"; continue }
|
||||
$type = if ($null -eq $v) { 'null' } else { $v.GetType().FullName }
|
||||
if (Test-Scalar $v) {
|
||||
"$Indent$($p.Name) ($type) = $v"
|
||||
} else {
|
||||
$text = [string]$v
|
||||
if ($text.Length -gt 100) { $text = $text.Substring(0, 97) + '...' }
|
||||
"$Indent$($p.Name) ($type) = $text"
|
||||
Show-Object $v "$Indent " ($Depth + 1)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function Show-Section([string]$Title, [scriptblock]$Get) {
|
||||
"== $Title"
|
||||
try {
|
||||
$result = @(& $Get)
|
||||
if ($result.Count -eq 0 -or ($result.Count -eq 1 -and $null -eq $result[0])) { ' (nothing returned)' }
|
||||
$n = 0
|
||||
foreach ($item in $result) {
|
||||
if ($null -eq $item) { continue }
|
||||
" [$n] ($($item.GetType().FullName))"
|
||||
Show-Object $item ' '
|
||||
$n++
|
||||
}
|
||||
} catch {
|
||||
" error: $($_.Exception.Message)"
|
||||
}
|
||||
''
|
||||
}
|
||||
|
||||
$report = & {
|
||||
$agent = try {
|
||||
(Get-Item (Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\cbengine.exe')).VersionInfo.ProductVersion
|
||||
} catch { 'unknown' }
|
||||
"Generated $(Get-Date -Format o) on $env:COMPUTERNAME, MARS agent $agent, PowerShell $($PSVersionTable.PSVersion)"
|
||||
''
|
||||
Show-Section 'Get-OBJob (in progress)' { Get-OBJob }
|
||||
Show-Section "Get-OBJob -Previous $Jobs" { Get-OBJob -Previous $Jobs }
|
||||
Show-Section 'Get-OBPolicy' { Get-OBPolicy }
|
||||
Show-Section 'Get-OBPolicy | Get-OBSchedule' { Get-OBSchedule -Policy (Get-OBPolicy) }
|
||||
if (Get-Command Get-OBSystemStatePolicy -ErrorAction SilentlyContinue) {
|
||||
Show-Section 'Get-OBSystemStatePolicy | Get-OBSchedule' {
|
||||
$ssPolicy = Get-OBSystemStatePolicy
|
||||
if ($ssPolicy) { Get-OBSchedule -Policy $ssPolicy } else { '(no System State policy configured)' }
|
||||
}
|
||||
}
|
||||
} | Out-String -Width 300
|
||||
|
||||
$report
|
||||
$report | Set-Content -Path $OutFile -Encoding UTF8
|
||||
Write-Host "Saved to $OutFile"
|
||||
@@ -0,0 +1,55 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Installs the MARS (Azure Backup agent) collector as a SYSTEM scheduled task.
|
||||
|
||||
.DESCRIPTION
|
||||
- Copies Send-MarsBackupStatus.ps1 to Program Files (only admins can modify it, which
|
||||
matters because it runs as SYSTEM).
|
||||
- Writes the config to ProgramData\BackupDashboard and restricts that folder to SYSTEM and
|
||||
Administrators, since it contains the API key.
|
||||
- Registers a scheduled task that runs every IntervalMinutes under Windows PowerShell 5.1
|
||||
(powershell.exe), which the agent's MSOnlineBackup module targets.
|
||||
Safe to re-run to change settings or update the script. Can sit alongside the Veeam collector.
|
||||
|
||||
.EXAMPLE
|
||||
.\Install-MarsCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
(prompts for the API key)
|
||||
#>
|
||||
#Requires -RunAsAdministrator
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory)][string]$ApiUrl,
|
||||
[Parameter(Mandatory)][string]$Tenant,
|
||||
# Name shown on the dashboard; defaults to "<server> - Azure Backup"
|
||||
[string]$JobName,
|
||||
[string]$SourceSystem = 'Azure Backup (MARS)',
|
||||
[ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
|
||||
[ValidateRange(1, 720)][int]$LookbackHours = 48,
|
||||
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard')
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$WindowsPowerShell = Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\powershell.exe'
|
||||
|
||||
# Check the agent is here before changing anything
|
||||
$agentModule = Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\Modules\MSOnlineBackup'
|
||||
if (-not (Get-Module -ListAvailable -Name MSOnlineBackup) -and -not (Test-Path $agentModule)) {
|
||||
throw 'The MARS agent (MSOnlineBackup PowerShell module) is not installed on this server.'
|
||||
}
|
||||
|
||||
# Shared installer code: in the repo it's in ..\common, so copy the whole collectors folder
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
throw "BackupDashboard.Collector.psm1 not found. Copy the whole collectors folder, including common."
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
$settings = [ordered]@{ Tenant = $Tenant; SourceSystem = $SourceSystem; LookbackHours = $LookbackHours }
|
||||
if ($JobName) { $settings.JobName = $JobName }
|
||||
Install-CollectorTask -Name 'mars' -ScriptPath (Join-Path $PSScriptRoot 'Send-MarsBackupStatus.ps1') `
|
||||
-TaskName 'Backup Dashboard - MARS collector' `
|
||||
-TaskDescription 'Sends Azure Backup (MARS) results to the Backup Dashboard' `
|
||||
-Shell $WindowsPowerShell -ApiUrl $ApiUrl -IntervalMinutes $IntervalMinutes -InstallDir $InstallDir -Settings $settings
|
||||
@@ -0,0 +1,68 @@
|
||||
# Azure Backup (MARS agent) collector
|
||||
|
||||
A PowerShell script that runs on each server with the Microsoft Azure Recovery Services (MARS)
|
||||
agent and reports its backups to the dashboard. Each server appears as one job,
|
||||
named `<SERVER> - Azure Backup` by default.
|
||||
|
||||
> **Checked against:** MARS agent 2.0.9274, a Files & Folders policy with completed backups only.
|
||||
> Not yet seen on a real agent: a failed backup, a backup in progress, or a System State policy.
|
||||
> Those paths are written to the same object model, but if one looks wrong, run
|
||||
> `Get-MarsCollectorDiagnostics.ps1` while it's in that state and send the output.
|
||||
|
||||
## Install (on the server, as an administrator)
|
||||
|
||||
Runs under Windows PowerShell 5.1 (`powershell.exe`), which the agent's `MSOnlineBackup` module
|
||||
targets and which every supported Windows Server has.
|
||||
|
||||
1. On the dashboard host, create a key for this server:
|
||||
```bash
|
||||
python manage.py create-key mars-acme-fs01
|
||||
```
|
||||
2. Copy the whole `collectors` folder to the server (the installer needs `common`
|
||||
as well as this folder), then from `collectors\mars` run:
|
||||
```powershell
|
||||
.\Install-MarsCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
```
|
||||
Add `-JobName "File Server"` to choose the name shown on the dashboard. It prompts for the
|
||||
key, then:
|
||||
- writes `C:\ProgramData\BackupDashboard\mars-collector.json` (readable only by SYSTEM and Administrators)
|
||||
- copies the script to `C:\Program Files\BackupDashboard\`
|
||||
- registers the scheduled task **Backup Dashboard - MARS collector** (SYSTEM, every 60 minutes)
|
||||
3. Preview the payloads, then backfill 30 days so the history view is populated straight away:
|
||||
```powershell
|
||||
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-MarsBackupStatus.ps1' -DryRun
|
||||
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-MarsBackupStatus.ps1' -LookbackHours 720
|
||||
```
|
||||
|
||||
It can be installed alongside the other collectors: it uses its own config, state, log and task.
|
||||
Re-run the installer to change settings or update the script. It also installs
|
||||
`BackupDashboard.Collector.psm1`, the code shared by all collectors, next to the script.
|
||||
|
||||
## What it sends
|
||||
|
||||
| Dashboard | MARS |
|
||||
|---|---|
|
||||
| Company | `Tenant` from the config |
|
||||
| Job | One per server, ID `mars:<computer name>`, so changing `JobName` renames rather than duplicates it |
|
||||
| Status | Backup job state: Failed → **Failed**. Completed with failed bytes or an error → **Warning**. Aborted/cancelled or unrecognised → **Warning**. In progress → **Running**. Restore jobs are ignored |
|
||||
| Started / finished | Job start / end time. MARS reports these in UTC without marking them as UTC; the collector handles that |
|
||||
| Next scheduled | Next slot from the backup policy schedule, or the System State policy's if that's sooner. Empty (so never Overdue) for schedules that run every 2+ weeks, since which week is next can't be told |
|
||||
| Message | Job and per-datasource errors (non-zero error codes), failed file counts, and the failed-file log path |
|
||||
| Size | Bytes transferred, summed across datasources |
|
||||
|
||||
Each run covers the last 48 hours (`LookbackHours`). Jobs already delivered are recorded in
|
||||
`mars-collector.state.json`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Log: `C:\ProgramData\BackupDashboard\mars-collector.log`. Exit codes: `0` OK, `1` some runs
|
||||
failed to send (retried next run), `2` setup failed (config or MARS module).
|
||||
- Wrong status, sizes or messages: run `.\Get-MarsCollectorDiagnostics.ps1` (read-only). It prints
|
||||
the real structure of the objects the agent's cmdlets return.
|
||||
|
||||
## Testing without MARS
|
||||
|
||||
```powershell
|
||||
. ./tests/MockMars.ps1
|
||||
./Send-MarsBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||
```
|
||||
@@ -0,0 +1,320 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Reports Microsoft Azure Recovery Services (MARS) agent backups to the Backup Dashboard.
|
||||
|
||||
.DESCRIPTION
|
||||
Runs on a server with the MARS agent, normally as a SYSTEM scheduled task every 60 minutes
|
||||
(see Install-MarsCollector.ps1). The server appears on the dashboard as one job. Each run sends:
|
||||
- the next scheduled backup, worked out from the backup (and System State) policy schedules,
|
||||
which drives the dashboard's Overdue flag, and
|
||||
- every backup job from the last LookbackHours that the dashboard hasn't already received in
|
||||
its final state. A backup in progress is sent as Running until it finishes.
|
||||
|
||||
Already-reported jobs are tracked in a state file, and the ingest API is idempotent, so a
|
||||
failed or repeated run is harmless: anything not confirmed is retried next time.
|
||||
|
||||
Written for Windows PowerShell 5.1, which the MARS agent's MSOnlineBackup module targets.
|
||||
|
||||
.PARAMETER ConfigPath
|
||||
JSON config with ApiUrl, ApiKey and Tenant (optional: JobName, SourceSystem, LookbackHours).
|
||||
|
||||
.PARAMETER LookbackHours
|
||||
Overrides the config value. Use 720 once to backfill 30 days of history.
|
||||
|
||||
.PARAMETER DryRun
|
||||
Print the payloads instead of sending them. Doesn't touch the state or log files.
|
||||
|
||||
.EXAMPLE
|
||||
.\Send-MarsBackupStatus.ps1 -DryRun
|
||||
.EXAMPLE
|
||||
.\Send-MarsBackupStatus.ps1 -LookbackHours 720
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$ConfigPath = (Join-Path $env:ProgramData 'BackupDashboard\mars-collector.json'),
|
||||
[int]$LookbackHours = 0,
|
||||
[switch]$DryRun
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# Shared collector code: installed next to this script; in the repo, in ..\common
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
Write-Host "BackupDashboard.Collector.psm1 not found next to $PSCommandPath; re-run the installer."
|
||||
exit 2
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
# MARS keeps a limited job history; this comfortably covers 30 days at 3 backups a day
|
||||
$MaxPreviousJobs = 200
|
||||
|
||||
# ---------- Helpers ----------
|
||||
|
||||
function Import-Mars {
|
||||
if (Get-Command Get-OBJob -ErrorAction SilentlyContinue) { return } # already loaded
|
||||
if (Get-Module -ListAvailable -Name MSOnlineBackup) {
|
||||
Import-Module MSOnlineBackup
|
||||
return
|
||||
}
|
||||
# The agent normally adds its module folder to PSModulePath; fall back to the default install path
|
||||
$agentModule = Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\Modules\MSOnlineBackup'
|
||||
Import-Module $agentModule
|
||||
}
|
||||
|
||||
# ---------- Schedule ----------
|
||||
|
||||
# Next slot strictly after $After for a MARS schedule (days of week + times of day).
|
||||
# Returns $null when there's no schedule, or when it runs every N>1 weeks: we can't tell which
|
||||
# week is next, and a wrong guess would raise a false Overdue alarm.
|
||||
function Get-NextSlot($Schedule, [datetime]$After) {
|
||||
if ($null -eq $Schedule) { return $null }
|
||||
$weeks = Get-FirstValue $Schedule @('ScheduleWeeklyFrequency', 'WeeksInterval')
|
||||
if ($weeks -and [int]$weeks -gt 1) { return $null }
|
||||
|
||||
$days = @(Get-FirstValue $Schedule @('ScheduleRunDays', 'DaysOfWeek') | ForEach-Object { [string]$_ })
|
||||
# TimeSpans (time of day, local) on real agents; accept DateTimes too
|
||||
$times = @(Get-FirstValue $Schedule @('ScheduleRunTimes', 'TimesOfDay') | ForEach-Object {
|
||||
if ($_ -is [timespan]) { $_ } else { ([datetime]$_).TimeOfDay }
|
||||
} | Sort-Object)
|
||||
if ($days.Count -eq 0 -or $times.Count -eq 0) { return $null }
|
||||
|
||||
for ($offset = 0; $offset -le 7; $offset++) {
|
||||
$date = $After.Date.AddDays($offset)
|
||||
if ($days -notcontains [string]$date.DayOfWeek) { continue }
|
||||
foreach ($time in $times) {
|
||||
$candidate = $date + $time
|
||||
if ($candidate -gt $After) { return $candidate }
|
||||
}
|
||||
}
|
||||
return $null
|
||||
}
|
||||
|
||||
function Get-NextRun {
|
||||
$next = $null
|
||||
$policies = @()
|
||||
try { $policies += @(Get-OBPolicy) } catch { Write-Log "Couldn't read the backup policy: $(Get-ErrorText $_)" 'WARN' }
|
||||
# Windows Server can have a separate System State policy with its own schedule
|
||||
if (Get-Command Get-OBSystemStatePolicy -ErrorAction SilentlyContinue) {
|
||||
try { $policies += @(Get-OBSystemStatePolicy) } catch { }
|
||||
}
|
||||
foreach ($policy in $policies) {
|
||||
if ($null -eq $policy) { continue }
|
||||
try {
|
||||
$slot = Get-NextSlot (Get-OBSchedule -Policy $policy) (Get-Date)
|
||||
if ($slot -and (-not $next -or $slot -lt $next)) { $next = $slot }
|
||||
} catch {
|
||||
Write-Log "Couldn't read a policy schedule: $(Get-ErrorText $_)" 'WARN'
|
||||
}
|
||||
}
|
||||
return ConvertTo-IsoTime $next
|
||||
}
|
||||
|
||||
# ---------- Jobs ----------
|
||||
|
||||
# A job's StartTime/EndTime are UTC values, though not necessarily marked as UTC: on a real agent
|
||||
# StartTime matched the "StartTimeUTC" in the status text, while DisplayTime held local time.
|
||||
# Returns $null for "not set" (1601-01-01), e.g. EndTime while a backup is running.
|
||||
function Get-JobTimeUtc($Job, [string]$Field) {
|
||||
$value = Get-FirstValue $Job.JobStatus @($Field)
|
||||
if ($null -eq $value) { return $null }
|
||||
$dt = [datetime]$value
|
||||
if ($dt.Year -le 1900) { return $null }
|
||||
return [datetime]::SpecifyKind($dt, [DateTimeKind]::Utc)
|
||||
}
|
||||
|
||||
# MARS always includes an ErrorInfo; ErrorCode 0 means no error
|
||||
function Test-HasError($ErrorInfo) {
|
||||
if (-not $ErrorInfo) { return $false }
|
||||
$code = Get-FirstValue $ErrorInfo @('ErrorCode')
|
||||
$detailed = Get-FirstValue $ErrorInfo @('DetailedErrorCode')
|
||||
return ($code -and [long]$code -ne 0) -or ($detailed -and [long]$detailed -ne 0)
|
||||
}
|
||||
|
||||
# DatasourceStatus.Datasource is an object whose DataSourceName is e.g. "E:\" or "System State"
|
||||
function Get-DatasourceName($DatasourceStatus) {
|
||||
$ds = Get-FirstValue $DatasourceStatus @('Datasource', 'DataSource')
|
||||
$name = Get-FirstValue $ds @('DataSourceName', 'FriendlyNameDSPS', 'Name')
|
||||
if ($name) { return [string]$name }
|
||||
if ($ds -is [string]) { return $ds }
|
||||
return 'Datasource'
|
||||
}
|
||||
|
||||
function Get-FailedFiles($Job) {
|
||||
$sum = [long]0
|
||||
foreach ($ds in Get-DatasourceStatuses $Job) {
|
||||
$value = Get-FirstValue (Get-FirstValue $ds @('FileProgress')) @('Failed')
|
||||
if ($value) { $sum += [long]$value }
|
||||
}
|
||||
return $sum
|
||||
}
|
||||
|
||||
function Get-JobStatus($Job) {
|
||||
$status = $Job.JobStatus
|
||||
$state = [string](Get-FirstValue $status @('JobState', 'State'))
|
||||
if (-not (Get-JobTimeUtc $Job 'EndTime') -and $state -notmatch 'Fail|Abort|Cancel|Complete|Success|Succeed|Warn') {
|
||||
return 'running'
|
||||
}
|
||||
if ($state -match 'Fail') { return 'failed' }
|
||||
if ($state -match 'Warn|Abort|Cancel|Stop') { return 'warning' }
|
||||
if ($state -match 'Complete|Success|Succeed') {
|
||||
# MARS can report Completed while individual files failed or an error was recorded
|
||||
if ((Get-FailedBytes $Job) -gt 0 -or (Get-FailedFiles $Job) -gt 0 -or
|
||||
(Test-HasError (Get-FirstValue $status @('ErrorInfo')))) { return 'warning' }
|
||||
return 'success'
|
||||
}
|
||||
return 'warning' # finished in a state we don't recognise: surface it rather than hide it
|
||||
}
|
||||
|
||||
function Get-DatasourceStatuses($Job) {
|
||||
return @(Get-FirstValue $Job.JobStatus @('DatasourceStatus', 'DataSourceStatus') | Where-Object { $_ })
|
||||
}
|
||||
|
||||
function Get-ByteTotal($Job, [string]$Field) {
|
||||
$sum = [long]0
|
||||
foreach ($ds in Get-DatasourceStatuses $Job) {
|
||||
$progress = Get-FirstValue $ds @('ByteProgress')
|
||||
$value = Get-FirstValue $progress @($Field)
|
||||
if ($value) { $sum += [long]$value }
|
||||
}
|
||||
return $sum
|
||||
}
|
||||
|
||||
function Get-FailedBytes($Job) { return Get-ByteTotal $Job 'Failed' }
|
||||
|
||||
function Format-ErrorInfo($ErrorInfo) {
|
||||
if (-not (Test-HasError $ErrorInfo)) { return $null }
|
||||
$text = Get-FirstValue $ErrorInfo @('ErrorMessage', 'Message', 'Description')
|
||||
if ($text) { return ([string]$text).Trim() }
|
||||
$code = Get-FirstValue $ErrorInfo @('ErrorCode', 'DetailedErrorCode')
|
||||
$params = @(Get-FirstValue $ErrorInfo @('ErrorParamList') | Where-Object { $_ } | ForEach-Object {
|
||||
$name = Get-FirstValue $_ @('Name')
|
||||
$value = Get-FirstValue $_ @('Value')
|
||||
if ($name) { "$name=$value" } else { [string]$_ }
|
||||
} | Where-Object { $_ })
|
||||
$line = if ($code) { "Error code $code" } else { [string]$ErrorInfo }
|
||||
if ($params.Count) { $line += " ($($params -join ', '))" }
|
||||
return $line
|
||||
}
|
||||
|
||||
function Get-RunMessage($Job, [string]$Status) {
|
||||
$lines = New-Object 'System.Collections.Generic.List[string]'
|
||||
$state = [string](Get-FirstValue $Job.JobStatus @('JobState', 'State'))
|
||||
$jobError = Format-ErrorInfo (Get-FirstValue $Job.JobStatus @('ErrorInfo'))
|
||||
if ($jobError) { $lines.Add($jobError) }
|
||||
|
||||
foreach ($ds in Get-DatasourceStatuses $Job) {
|
||||
$name = Get-DatasourceName $ds
|
||||
$dsError = Format-ErrorInfo (Get-FirstValue $ds @('ErrorInfo'))
|
||||
$failedFiles = Get-FirstValue (Get-FirstValue $ds @('FileProgress')) @('Failed')
|
||||
$failedBytes = Get-FirstValue (Get-FirstValue $ds @('ByteProgress')) @('Failed')
|
||||
if ($dsError) { $lines.Add("${name}: $dsError") }
|
||||
if ($failedFiles -and [long]$failedFiles -gt 0) { $lines.Add("${name}: $failedFiles file(s) failed to back up") }
|
||||
elseif ($failedBytes -and [long]$failedBytes -gt 0) { $lines.Add("${name}: $failedBytes bytes failed to back up") }
|
||||
}
|
||||
$failedLog = Get-FirstValue $Job.JobStatus @('FailedFileLog')
|
||||
if ($failedLog -and $lines.Count) { $lines.Add("Failed file log: $failedLog") }
|
||||
|
||||
if ($lines.Count -eq 0) {
|
||||
switch ($Status) {
|
||||
'success' { return 'Backup completed successfully.' }
|
||||
'running' { return '' }
|
||||
default { return "Backup finished with state '$state'." }
|
||||
}
|
||||
}
|
||||
if ($state -and $Status -ne 'success') { $lines.Insert(0, "Job state: $state") }
|
||||
return ($lines -join "`n")
|
||||
}
|
||||
|
||||
function ConvertTo-Run($Job) {
|
||||
$status = Get-JobStatus $Job
|
||||
$datasources = @(Get-DatasourceStatuses $Job | ForEach-Object { Get-DatasourceName $_ })
|
||||
|
||||
$details = [ordered]@{ server = $env:COMPUTERNAME }
|
||||
if ($datasources.Count) { $details.datasources = $datasources }
|
||||
$agentState = Get-FirstValue $Job.JobStatus @('JobState', 'State')
|
||||
if ($agentState) { $details.mars_state = [string]$agentState }
|
||||
|
||||
$run = [ordered]@{
|
||||
external_id = [string]$Job.JobId
|
||||
status = $status
|
||||
started_at = ConvertTo-IsoTime (Get-JobTimeUtc $Job 'StartTime')
|
||||
message = Get-RunMessage $Job $status
|
||||
details = $details
|
||||
}
|
||||
if ($status -ne 'running') {
|
||||
$finished = ConvertTo-IsoTime (Get-JobTimeUtc $Job 'EndTime')
|
||||
if ($finished) { $run.finished_at = $finished } # otherwise the API uses receipt time
|
||||
$transferred = Get-ByteTotal $Job 'Progress'
|
||||
if ($transferred -gt 0) { $run.size_bytes = $transferred }
|
||||
$failedBytes = Get-FailedBytes $Job
|
||||
if ($failedBytes -gt 0) { $details.failed_bytes = $failedBytes }
|
||||
$failedFiles = Get-FailedFiles $Job
|
||||
if ($failedFiles -gt 0) { $details.failed_files = $failedFiles }
|
||||
}
|
||||
return $run
|
||||
}
|
||||
|
||||
function Test-IsBackupJob($Job) {
|
||||
# Get-OBJob also lists restores (JobType Recovery); only backups belong on the dashboard
|
||||
$type = [string](Get-FirstValue $Job @('JobType', 'Type'))
|
||||
return (-not $type) -or $type -match 'Backup'
|
||||
}
|
||||
|
||||
# ---------- Main ----------
|
||||
|
||||
try {
|
||||
$Config = Initialize-Collector -Name 'mars' -ConfigPath $ConfigPath -LookbackHours $LookbackHours -DryRun:$DryRun
|
||||
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Azure Backup (MARS)' }
|
||||
$JobName = if ($Config.JobName) { [string]$Config.JobName } else { "$env:COMPUTERNAME - Azure Backup" }
|
||||
$LookbackHours = $Config.LookbackHours
|
||||
|
||||
Import-Mars
|
||||
|
||||
$sinceUtc = (Get-Date).ToUniversalTime().AddHours(-$LookbackHours)
|
||||
# Get-OBJob with no arguments lists jobs in progress; -Previous lists finished ones
|
||||
$jobs = @(@(Get-OBJob) + @(Get-OBJob -Previous $MaxPreviousJobs) | Where-Object {
|
||||
$_ -and (Test-IsBackupJob $_)
|
||||
})
|
||||
$byId = [ordered]@{}
|
||||
foreach ($job in $jobs) {
|
||||
$start = Get-JobTimeUtc $job 'StartTime'
|
||||
if (-not $start -or $start -lt $sinceUtc) { continue }
|
||||
$byId[[string]$job.JobId] = $job # a job in both lists: the finished entry (later) wins
|
||||
}
|
||||
$pending = @($byId.Values |
|
||||
Where-Object { -not (Test-RunSent ([string]$_.JobId)) } |
|
||||
Sort-Object { Get-JobTimeUtc $_ 'StartTime' })
|
||||
|
||||
$jobInfo = [ordered]@{
|
||||
# Stable per server, so renaming the dashboard job via JobName doesn't create a new one
|
||||
external_id = "mars:$($env:COMPUTERNAME.ToLowerInvariant())"
|
||||
name = $JobName
|
||||
# Always sent (null when unscheduled) so the dashboard's value tracks the agent's
|
||||
next_run_at = Get-NextRun
|
||||
}
|
||||
} catch {
|
||||
Write-Log "Collector setup failed: $(Get-ErrorText $_)" 'ERROR'
|
||||
exit 2
|
||||
}
|
||||
|
||||
$failures = 0
|
||||
try {
|
||||
if ($pending.Count -eq 0) {
|
||||
# Keep next_run_at fresh even when nothing new ran
|
||||
Send-JobReport -Tenant $Config.Tenant -SourceSystem $SourceSystem -Job $jobInfo
|
||||
}
|
||||
foreach ($job in $pending) {
|
||||
$run = ConvertTo-Run $job
|
||||
Send-JobReport -Tenant $Config.Tenant -SourceSystem $SourceSystem -Job $jobInfo -Run $run -SentKey $run.external_id -StartedAt ((Get-JobTimeUtc $job 'StartTime'))
|
||||
}
|
||||
} catch {
|
||||
$failures++
|
||||
Write-Log "Failed to report: $(Get-ErrorText $_)" 'ERROR'
|
||||
}
|
||||
|
||||
$exitCode = Complete-Collector -Failures $failures
|
||||
if ($exitCode) { exit $exitCode }
|
||||
@@ -0,0 +1,78 @@
|
||||
# Stand-ins for the MARS agent cmdlets. Object shapes follow Get-MarsCollectorDiagnostics.ps1
|
||||
# output from a real agent (2.0.9274): job times are UTC values with no UTC marking, "not set" is
|
||||
# 1601-01-01, ErrorInfo is always present (ErrorCode 0 = no error), Datasource is an object, and
|
||||
# schedule times are TimeSpans.
|
||||
#
|
||||
# . ./tests/MockMars.ps1
|
||||
# ./Send-MarsBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||
|
||||
# Fixed to the top of the hour, and IDs derived from times, so repeated runs see the same jobs
|
||||
$now = Get-Date -Minute 0 -Second 0 -Millisecond 0
|
||||
$notSet = [datetime]::new(1601, 1, 1)
|
||||
function New-StableGuid([string]$Seed) {
|
||||
[guid]::new([Security.Cryptography.MD5]::Create().ComputeHash([Text.Encoding]::UTF8.GetBytes($Seed)))
|
||||
}
|
||||
|
||||
# Local time -> the UTC value MARS reports, with Kind Unspecified like the real objects
|
||||
function ConvertTo-MarsTime([datetime]$Local) {
|
||||
[datetime]::SpecifyKind($Local.ToUniversalTime(), [DateTimeKind]::Unspecified)
|
||||
}
|
||||
|
||||
function New-MockErrorInfo([int]$Code = 0, $Params = @()) {
|
||||
[pscustomobject]@{ DetailedErrorCode = 0; DetailedErrorSource = $null; ErrorCode = $Code; ErrorParamList = $Params }
|
||||
}
|
||||
|
||||
function New-MockDatasource($Name, [uint64]$Done, [uint64]$FailedBytes = 0, [uint64]$FailedFiles = 0, $ErrorInfo = (New-MockErrorInfo)) {
|
||||
[pscustomobject]@{
|
||||
ByteProgress = [pscustomobject]@{ Changed = $Done; Failed = $FailedBytes; Progress = $Done; Total = 2TB }
|
||||
Datasource = [pscustomobject]@{ DataSourceName = $Name; DataSourceId = 6310158806455554328 }
|
||||
ErrorInfo = $ErrorInfo
|
||||
FileProgress = [pscustomobject]@{ Changed = 100; CurrentFile = ''; Failed = $FailedFiles; Progress = 100; Total = 100 }
|
||||
JobState = 'Completed'
|
||||
}
|
||||
}
|
||||
|
||||
function New-MockJob($Type, [datetime]$Start, $End, $State, $Datasources = @(), $ErrorInfo = (New-MockErrorInfo), $FailedFileLog = '') {
|
||||
$endValue = if ($End) { ConvertTo-MarsTime $End } else { $notSet }
|
||||
[pscustomobject]@{
|
||||
IsRetried = $false
|
||||
JobId = New-StableGuid "$Type|$($Start.Ticks)"
|
||||
JobType = $Type
|
||||
DisplayTime = $Start # the only local time on the object
|
||||
JobStatus = [pscustomobject]@{
|
||||
DatasourceStatus = $Datasources
|
||||
EndTime = $endValue
|
||||
ErrorInfo = $ErrorInfo
|
||||
FailedFileLog = $FailedFileLog
|
||||
JobState = $State
|
||||
StartTime = ConvertTo-MarsTime $Start
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$global:MockMarsRunning = @(
|
||||
(New-MockJob 'Backup' $now.AddMinutes(-10) $null 'InProgress' @((New-MockDatasource 'E:\' 2GB)))
|
||||
)
|
||||
$global:MockMarsPrevious = @(
|
||||
(New-MockJob 'Backup' $now.AddHours(-12) $now.AddHours(-11.5) 'Completed' @((New-MockDatasource 'E:\' 523407872))),
|
||||
# Completed, but some files failed
|
||||
(New-MockJob 'Backup' $now.AddHours(-36) $now.AddHours(-35) 'Completed' @(
|
||||
(New-MockDatasource 'E:\' 3GB 40MB 12)) -FailedFileLog 'C:\Program Files\Microsoft Azure Recovery Services Agent\Temp\Errors.log'),
|
||||
(New-MockJob 'Backup' $now.AddHours(-30) $now.AddHours(-29.5) 'Failed' @() `
|
||||
(New-MockErrorInfo 100050 @([pscustomobject]@{ Name = 'Volume'; Value = 'E:' }))),
|
||||
(New-MockJob 'Recovery' $now.AddHours(-5) $now.AddHours(-4) 'Completed'), # a restore: must be ignored
|
||||
(New-MockJob 'Backup' $now.AddDays(-5) $now.AddDays(-5).AddHours(1) 'Completed') # outside 48h lookback
|
||||
)
|
||||
|
||||
# Every day at 21:00 local, as on the real server
|
||||
$global:MockMarsSchedule = [pscustomobject]@{
|
||||
ScheduleRunDays = [DayOfWeek[]](0..6)
|
||||
ScheduleRunTimes = [timespan[]]@([timespan]::FromHours(21))
|
||||
ScheduleWeeklyFrequency = 1
|
||||
TypeOfBackupSchedule = 'Daily'
|
||||
}
|
||||
|
||||
function Get-OBJob { param([int]$Previous) if ($PSBoundParameters.ContainsKey('Previous')) { $global:MockMarsPrevious } else { $global:MockMarsRunning } }
|
||||
function Get-OBPolicy { [pscustomobject]@{ PolicyName = 'policy'; BackupSchedule = $global:MockMarsSchedule } }
|
||||
function Get-OBSchedule { param($Policy) $global:MockMarsSchedule }
|
||||
function Get-OBSystemStatePolicy { $null } # none configured, as on the real server
|
||||
@@ -0,0 +1,119 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Read-only dump of what the Veeam Backup for Microsoft 365 cmdlets return, for troubleshooting
|
||||
the collector.
|
||||
|
||||
.DESCRIPTION
|
||||
The collector reads properties of Get-VBOJob / Get-VBOJobSession results whose names aren't
|
||||
fully documented. This prints those objects' real structure (property names, types, values and
|
||||
each date's Kind, a few levels deep) for a few jobs and their latest sessions. Nothing is
|
||||
changed. Output is also saved to a text file.
|
||||
|
||||
.EXAMPLE
|
||||
.\Get-Vb365CollectorDiagnostics.ps1
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[int]$Jobs = 3,
|
||||
[int]$SessionsPerJob = 2,
|
||||
[string]$OutFile = (Join-Path $env:TEMP 'vb365-collector-diagnostics.txt')
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$MaxDepth = 4
|
||||
|
||||
if (-not (Get-Command Get-VBOJob -ErrorAction SilentlyContinue)) {
|
||||
if (Get-Module -ListAvailable -Name Veeam.Archiver.PowerShell) {
|
||||
Import-Module Veeam.Archiver.PowerShell
|
||||
} else {
|
||||
Import-Module (Join-Path $env:ProgramFiles 'Veeam\Backup365\Veeam.Archiver.PowerShell\Veeam.Archiver.PowerShell.psd1')
|
||||
}
|
||||
}
|
||||
|
||||
function Test-Scalar($Value) {
|
||||
return $null -eq $Value -or $Value -is [string] -or $Value -is [ValueType] -or $Value -is [guid]
|
||||
}
|
||||
|
||||
# Recursive "name (type) = value" listing. Collections show their first 3 items.
|
||||
function Show-Object($Value, [string]$Indent = ' ', [int]$Depth = 0) {
|
||||
if (Test-Scalar $Value) { return }
|
||||
if ($Depth -ge $MaxDepth) { "$Indent..."; return }
|
||||
if ($Value -is [Collections.IEnumerable]) {
|
||||
$items = @($Value)
|
||||
"$Indent[$($items.Count) item(s)]"
|
||||
$i = 0
|
||||
foreach ($item in $items | Select-Object -First 3) {
|
||||
"$Indent[$i] ($($item.GetType().FullName))$(if (Test-Scalar $item) { " = $item" })"
|
||||
Show-Object $item "$Indent " ($Depth + 1)
|
||||
$i++
|
||||
}
|
||||
return
|
||||
}
|
||||
foreach ($p in $Value.PSObject.Properties) {
|
||||
try { $v = $p.Value } catch { "$Indent$($p.Name) = <error: $($_.Exception.Message)>"; continue }
|
||||
$type = if ($null -eq $v) { 'null' } else { $v.GetType().FullName }
|
||||
if (Test-Scalar $v) {
|
||||
$kind = if ($v -is [datetime]) { " [Kind=$($v.Kind)]" } else { '' }
|
||||
"$Indent$($p.Name) ($type) = $v$kind"
|
||||
} else {
|
||||
$text = [string]$v
|
||||
if ($text.Length -gt 100) { $text = $text.Substring(0, 97) + '...' }
|
||||
"$Indent$($p.Name) ($type) = $text"
|
||||
Show-Object $v "$Indent " ($Depth + 1)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function Show-Section([string]$Title, [scriptblock]$Get) {
|
||||
"== $Title"
|
||||
try {
|
||||
$result = @(& $Get)
|
||||
if ($result.Count -eq 0 -or ($result.Count -eq 1 -and $null -eq $result[0])) { ' (nothing returned)' }
|
||||
$n = 0
|
||||
foreach ($item in $result) {
|
||||
if ($null -eq $item) { continue }
|
||||
" [$n] ($($item.GetType().FullName))"
|
||||
Show-Object $item ' '
|
||||
$n++
|
||||
}
|
||||
} catch {
|
||||
" error: $($_.Exception.Message)"
|
||||
}
|
||||
''
|
||||
}
|
||||
|
||||
$report = & {
|
||||
$module = Get-Module Veeam.Archiver.PowerShell
|
||||
"Generated $(Get-Date -Format o) on $env:COMPUTERNAME, PowerShell $($PSVersionTable.PSVersion), module $($module.Version)"
|
||||
"Local time zone: $([TimeZoneInfo]::Local.Id), UTC offset now $([TimeZoneInfo]::Local.GetUtcOffset((Get-Date))), culture $([Globalization.CultureInfo]::CurrentCulture.Name)"
|
||||
''
|
||||
"== Organizations"
|
||||
try {
|
||||
Get-VBOOrganization | ForEach-Object { " $($_.Name) [$($_.Type)]" }
|
||||
} catch { " error: $($_.Exception.Message)" }
|
||||
''
|
||||
$allJobs = @(Get-VBOJob)
|
||||
"== Jobs: $($allJobs.Count) in total"
|
||||
$allJobs | ForEach-Object { " $($_.Name) [$($_.Id)] enabled=$($_.IsEnabled) next=$($_.NextRun)" }
|
||||
''
|
||||
Show-Section "Get-VBOJob (first $Jobs, full detail)" { $allJobs | Select-Object -First $Jobs }
|
||||
foreach ($j in $allJobs | Select-Object -First $Jobs) {
|
||||
Show-Section "Get-VBOJobSession -Job '$($j.Name)' (latest $SessionsPerJob)" {
|
||||
Get-VBOJobSession -Job $j | Sort-Object CreationTime -Descending | Select-Object -First $SessionsPerJob
|
||||
}
|
||||
}
|
||||
if (Get-Command Get-VBOCopyJob -ErrorAction SilentlyContinue) {
|
||||
Show-Section 'Get-VBOCopyJob (first 2)' { Get-VBOCopyJob | Select-Object -First 2 }
|
||||
if (Get-Command Get-VBOCopyJobSession -ErrorAction SilentlyContinue) {
|
||||
Show-Section 'Get-VBOCopyJobSession (latest 2)' {
|
||||
Get-VBOCopyJobSession | Sort-Object CreationTime -Descending | Select-Object -First 2
|
||||
}
|
||||
}
|
||||
} else {
|
||||
'== Get-VBOCopyJob: not available'
|
||||
}
|
||||
} | Out-String -Width 300
|
||||
|
||||
$report
|
||||
$report | Set-Content -Path $OutFile -Encoding UTF8
|
||||
Write-Host "Saved to $OutFile"
|
||||
@@ -0,0 +1,74 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Installs the Veeam Backup for Microsoft 365 collector as a SYSTEM scheduled task.
|
||||
|
||||
.DESCRIPTION
|
||||
- Copies Send-Vb365BackupStatus.ps1 to Program Files (only admins can modify it, which
|
||||
matters because it runs as SYSTEM).
|
||||
- Writes the config to ProgramData\BackupDashboard and restricts that folder to SYSTEM and
|
||||
Administrators, since it contains the API key.
|
||||
- Registers a scheduled task that runs every IntervalMinutes, under Windows PowerShell 5.1
|
||||
(powershell.exe) unless -PwshPath points at PowerShell 7.
|
||||
Safe to re-run to change settings or update the script. Can sit alongside the other collectors.
|
||||
|
||||
Dashboard company per job: -OrganizationTenants maps Microsoft 365 organization names to
|
||||
companies; jobs of unmapped organizations use -Tenant, or the organization's name if no -Tenant.
|
||||
|
||||
.EXAMPLE
|
||||
.\Install-Vb365Collector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
One client per VB365 server. Prompts for the API key.
|
||||
.EXAMPLE
|
||||
.\Install-Vb365Collector.ps1 -ApiUrl https://backups.example.com -OrganizationTenants @{
|
||||
'acme.onmicrosoft.com' = 'Acme Corp'; 'bluewater.onmicrosoft.com' = 'Bluewater Dental' }
|
||||
A shared VB365 server protecting several organizations.
|
||||
#>
|
||||
#Requires -RunAsAdministrator
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory)][string]$ApiUrl,
|
||||
[string]$Tenant,
|
||||
[hashtable]$OrganizationTenants = @{},
|
||||
[string]$SourceSystem = 'Veeam Backup for M365',
|
||||
[ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
|
||||
[ValidateRange(1, 720)][int]$LookbackHours = 48,
|
||||
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard'),
|
||||
# Run under PowerShell 7 instead of Windows PowerShell, e.g. 'C:\Program Files\PowerShell\7\pwsh.exe'
|
||||
[string]$PwshPath
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$Shell = Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\powershell.exe'
|
||||
if ($PwshPath) {
|
||||
if (-not (Test-Path $PwshPath)) { throw "PowerShell 7 not found at '$PwshPath'." }
|
||||
$Shell = $PwshPath
|
||||
}
|
||||
|
||||
# Check VB365 is here before changing anything
|
||||
$moduleFile = Join-Path $env:ProgramFiles 'Veeam\Backup365\Veeam.Archiver.PowerShell\Veeam.Archiver.PowerShell.psd1'
|
||||
if (-not (Get-Module -ListAvailable -Name Veeam.Archiver.PowerShell) -and -not (Test-Path $moduleFile)) {
|
||||
throw 'The Veeam Backup for Microsoft 365 PowerShell module (Veeam.Archiver.PowerShell) is not installed on this server.'
|
||||
}
|
||||
if (-not $Tenant -and $OrganizationTenants.Count -eq 0) {
|
||||
Write-Warning 'No -Tenant or -OrganizationTenants: jobs will be listed under their Microsoft 365 organization name.'
|
||||
}
|
||||
|
||||
# Shared installer code: in the repo it's in ..\common, so copy the whole collectors folder
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
throw "BackupDashboard.Collector.psm1 not found. Copy the whole collectors folder, including common."
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
$settings = [ordered]@{
|
||||
SourceSystem = $SourceSystem
|
||||
LookbackHours = $LookbackHours
|
||||
OrganizationTenants = $OrganizationTenants
|
||||
}
|
||||
if ($Tenant) { $settings.Tenant = $Tenant }
|
||||
Install-CollectorTask -Name 'vb365' -ScriptPath (Join-Path $PSScriptRoot 'Send-Vb365BackupStatus.ps1') `
|
||||
-TaskName 'Backup Dashboard - VB365 collector' `
|
||||
-TaskDescription 'Sends Veeam Backup for Microsoft 365 results to the Backup Dashboard' `
|
||||
-Shell $Shell -ApiUrl $ApiUrl -IntervalMinutes $IntervalMinutes -InstallDir $InstallDir -Settings $settings
|
||||
@@ -0,0 +1,88 @@
|
||||
# Veeam Backup for Microsoft 365 collector
|
||||
|
||||
A PowerShell script that runs on the Veeam Backup for Microsoft 365 (VB365) server and reports its
|
||||
backup jobs (and backup copy jobs) to the dashboard, one dashboard job per VB365 job.
|
||||
|
||||
> **Checked against:** VB365 8.4 under Windows PowerShell 5.1, a shared server protecting 11
|
||||
> organizations: successful, warning and retried sessions. Not yet seen on a real server: a failed
|
||||
> session, a session in progress, or a backup copy job. If one looks wrong, run
|
||||
> `Get-Vb365CollectorDiagnostics.ps1` while it's in that state and send the output.
|
||||
|
||||
## Which company a job appears under
|
||||
|
||||
One VB365 server can protect several Microsoft 365 organizations, so the company is picked per job
|
||||
from the job's organization:
|
||||
|
||||
1. `OrganizationTenants` maps an organization name (as `Get-VBOOrganization` shows it, e.g.
|
||||
`acme.onmicrosoft.com`; not case-sensitive) to a dashboard company.
|
||||
2. Otherwise `Tenant`, for a VB365 server dedicated to one client.
|
||||
3. Otherwise the organization's own name.
|
||||
|
||||
Changing a job's company later creates a new dashboard job under the new company; delete the old
|
||||
one with `python manage.py delete-job <id>`.
|
||||
|
||||
## Install (on the VB365 server, as an administrator)
|
||||
|
||||
1. On the dashboard host, create a key for this server:
|
||||
```bash
|
||||
python manage.py create-key vb365-01
|
||||
```
|
||||
2. Copy the whole `collectors` folder to the server (the installer needs `common`
|
||||
as well as this folder), then from `collectors\vb365` run one of:
|
||||
```powershell
|
||||
# VB365 server for one client
|
||||
.\Install-Vb365Collector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
|
||||
# Shared VB365 server
|
||||
.\Install-Vb365Collector.ps1 -ApiUrl https://backups.example.com -OrganizationTenants @{
|
||||
'acme.onmicrosoft.com' = 'Acme Corp'
|
||||
'bluewater.onmicrosoft.com' = 'Bluewater Dental'
|
||||
}
|
||||
```
|
||||
It prompts for the key, then:
|
||||
- writes `C:\ProgramData\BackupDashboard\vb365-collector.json` (readable only by SYSTEM and Administrators)
|
||||
- copies the script to `C:\Program Files\BackupDashboard\`
|
||||
- registers the scheduled task **Backup Dashboard - VB365 collector** (SYSTEM, every 60 minutes)
|
||||
|
||||
It runs under Windows PowerShell 5.1. If your VB365 version's module needs PowerShell 7, add
|
||||
`-PwshPath 'C:\Program Files\PowerShell\7\pwsh.exe'`.
|
||||
3. Preview the payloads, then backfill 30 days so the history view is populated straight away:
|
||||
```powershell
|
||||
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-Vb365BackupStatus.ps1' -DryRun
|
||||
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-Vb365BackupStatus.ps1' -LookbackHours 720
|
||||
```
|
||||
|
||||
It can be installed alongside the other collectors: it uses its own config, state, log and task.
|
||||
Re-run the installer to change settings (including the organization mapping) or update the script. It also installs
|
||||
`BackupDashboard.Collector.psm1`, the code shared by all collectors, next to the script.
|
||||
|
||||
## What it sends
|
||||
|
||||
| Dashboard | VB365 |
|
||||
|---|---|
|
||||
| Company | See above |
|
||||
| Job | Each backup job and backup copy job. The job's GUID is the ID, so renaming it in VB365 doesn't duplicate it |
|
||||
| Status | Session status: Success / Warning / Failed. Running or queued → **Running**. Stopped (cancelled) or unrecognised → **Warning** |
|
||||
| Started / finished | Session `CreationTime` / `EndTime` |
|
||||
| Next scheduled | Job's `NextRun`. Empty for disabled jobs |
|
||||
| Message | Warning and error lines from the session log (items titled `[Warning] ...`, `[Failed] ...`), plus "Veeam will retry this job." when a retry is pending |
|
||||
| Retries | VB365 runs a retry as a separate session, so it appears as its own run, marked `details.retry_of` (the session it retried). The retried run is marked `details.will_retry` |
|
||||
| Size | Data transferred (`Statistics.TransferredData`) |
|
||||
| Details | Organization, objects processed, job kind (`backup` / `copy`) |
|
||||
|
||||
Each run covers the last 48 hours (`LookbackHours`). Sessions already delivered are recorded in
|
||||
`vb365-collector.state.json`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Log: `C:\ProgramData\BackupDashboard\vb365-collector.log`. Exit codes: `0` OK, `1` some jobs
|
||||
failed to send (retried next run), `2` setup failed (config or VB365 module).
|
||||
- Wrong status, times or messages: run `.\Get-Vb365CollectorDiagnostics.ps1` (read-only). It prints
|
||||
the real structure of jobs and sessions, including whether each date is local or UTC.
|
||||
|
||||
## Testing without VB365
|
||||
|
||||
```powershell
|
||||
. ./tests/MockVb365.ps1
|
||||
./Send-Vb365BackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||
```
|
||||
@@ -0,0 +1,278 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Reports Veeam Backup for Microsoft 365 (VB365) job results to the Backup Dashboard.
|
||||
|
||||
.DESCRIPTION
|
||||
Runs on the VB365 server, normally as a SYSTEM scheduled task every 60 minutes
|
||||
(see Install-Vb365Collector.ps1). For every backup job (and backup copy job) it sends:
|
||||
- the job's next scheduled run, which drives the dashboard's Overdue flag, and
|
||||
- every session from the last LookbackHours that the dashboard hasn't already received in
|
||||
its final state. Running sessions are re-sent each time until they finish.
|
||||
|
||||
One VB365 server can protect several Microsoft 365 organizations, so the dashboard company is
|
||||
chosen per job from its organization: OrganizationTenants in the config maps an organization
|
||||
name (e.g. "contoso.onmicrosoft.com") to a company; otherwise Tenant is used; otherwise the
|
||||
organization's own name.
|
||||
|
||||
Already-reported sessions are tracked in a state file, and the ingest API is idempotent, so a
|
||||
failed or repeated run is harmless: anything not confirmed is retried next time.
|
||||
|
||||
.PARAMETER ConfigPath
|
||||
JSON config with ApiUrl and ApiKey (optional: Tenant, OrganizationTenants, SourceSystem,
|
||||
LookbackHours).
|
||||
|
||||
.PARAMETER LookbackHours
|
||||
Overrides the config value. Use 720 once to backfill 30 days of history.
|
||||
|
||||
.PARAMETER DryRun
|
||||
Print the payloads instead of sending them. Doesn't touch the state or log files.
|
||||
|
||||
.EXAMPLE
|
||||
.\Send-Vb365BackupStatus.ps1 -DryRun
|
||||
.EXAMPLE
|
||||
.\Send-Vb365BackupStatus.ps1 -LookbackHours 720
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$ConfigPath = (Join-Path $env:ProgramData 'BackupDashboard\vb365-collector.json'),
|
||||
[int]$LookbackHours = 0,
|
||||
[switch]$DryRun
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# Shared collector code: installed next to this script; in the repo, in ..\common
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
Write-Host "BackupDashboard.Collector.psm1 not found next to $PSCommandPath; re-run the installer."
|
||||
exit 2
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
$Invariant = [Globalization.CultureInfo]::InvariantCulture
|
||||
|
||||
# ---------- Helpers ----------
|
||||
|
||||
function Import-Vb365 {
|
||||
if (Get-Command Get-VBOJob -ErrorAction SilentlyContinue) { return } # already loaded
|
||||
if (Get-Module -ListAvailable -Name Veeam.Archiver.PowerShell) {
|
||||
Import-Module Veeam.Archiver.PowerShell
|
||||
return
|
||||
}
|
||||
# Fall back to the default install path
|
||||
Import-Module (Join-Path $env:ProgramFiles 'Veeam\Backup365\Veeam.Archiver.PowerShell\Veeam.Archiver.PowerShell.psd1')
|
||||
}
|
||||
|
||||
function Get-Time($Object, [string]$Field) {
|
||||
$value = Get-FirstValue $Object @($Field)
|
||||
if ($null -eq $value) { return $null }
|
||||
$dt = [datetime]$value
|
||||
if ($dt.Year -le 1900) { return $null } # e.g. EndTime while a session is running
|
||||
return $dt
|
||||
}
|
||||
|
||||
# "1.5 GB", "512 MB", "123456" -> bytes; $null when it can't be read
|
||||
function ConvertTo-Bytes($Value) {
|
||||
if ($null -eq $Value) { return $null }
|
||||
if ($Value -is [ValueType]) { return [long]$Value }
|
||||
$match = [regex]::Match([string]$Value, '^\s*([\d.,]+)\s*([KMGTP]?B)?\s*$', 'IgnoreCase')
|
||||
if (-not $match.Success) { return $null }
|
||||
$number = 0.0
|
||||
if (-not [double]::TryParse(($match.Groups[1].Value -replace ',', '.'), [Globalization.NumberStyles]::Float, $Invariant, [ref]$number)) {
|
||||
return $null
|
||||
}
|
||||
$power = @{ '' = 0; 'B' = 0; 'KB' = 1; 'MB' = 2; 'GB' = 3; 'TB' = 4; 'PB' = 5 }[$match.Groups[2].Value.ToUpperInvariant()]
|
||||
return [long]($number * [math]::Pow(1024, $power))
|
||||
}
|
||||
|
||||
# ---------- Jobs and tenants ----------
|
||||
|
||||
function Get-OrganizationName($Job) {
|
||||
$org = Get-FirstValue $Job @('Organization')
|
||||
if ($null -eq $org) {
|
||||
# Copy jobs point at their source backup job
|
||||
$org = Get-FirstValue (Get-FirstValue $Job @('BackupJob')) @('Organization')
|
||||
}
|
||||
if ($org -is [string]) { return $org }
|
||||
$name = Get-FirstValue $org @('Name', 'OfficeName', 'DisplayName')
|
||||
if ($name) { return [string]$name }
|
||||
return $null
|
||||
}
|
||||
|
||||
function Get-TenantForJob($Job) {
|
||||
$org = Get-OrganizationName $Job
|
||||
if ($org -and $Config.OrganizationTenants) {
|
||||
$mapped = $Config.OrganizationTenants.PSObject.Properties[$org] # case-insensitive lookup
|
||||
if ($mapped -and $mapped.Value) { return [string]$mapped.Value }
|
||||
}
|
||||
if ($Config.Tenant) { return [string]$Config.Tenant }
|
||||
if ($org) { return $org }
|
||||
return $env:COMPUTERNAME
|
||||
}
|
||||
|
||||
function Get-NextRun($Job) {
|
||||
try {
|
||||
$enabled = Get-FirstValue $Job @('IsEnabled', 'Enabled')
|
||||
if ($enabled -eq $false) { return $null }
|
||||
return ConvertTo-IsoTime (Get-Time $Job 'NextRun')
|
||||
} catch {
|
||||
Write-Log "Couldn't read schedule for job '$($Job.Name)': $(Get-ErrorText $_)" 'WARN'
|
||||
return $null
|
||||
}
|
||||
}
|
||||
|
||||
# ---------- Sessions ----------
|
||||
|
||||
function Get-RunStatus($Session) {
|
||||
$status = [string](Get-FirstValue $Session @('Status', 'Result', 'State'))
|
||||
if ($status -match 'Fail|Error') { return 'failed' }
|
||||
if ($status -match 'Warn') { return 'warning' }
|
||||
if ($status -match 'Succe') { return 'success' }
|
||||
if ($status -match 'Run|Queue|Progress|Start|Wait') { return 'running' }
|
||||
if (-not (Get-Time $Session 'EndTime')) { return 'running' }
|
||||
# Stopped (cancelled), Disconnected, or a state we don't recognise: surface it rather than hide it
|
||||
return 'warning'
|
||||
}
|
||||
|
||||
function Test-EmptyGuid($Value) {
|
||||
return (-not $Value) -or [string]$Value -eq [string][guid]::Empty
|
||||
}
|
||||
|
||||
# Warning/error lines from the session log. Each log item's status is a prefix of its Title, e.g.
|
||||
# "[Warning] Item was skipped" (there's no separate status property), so match on that rather than
|
||||
# on words in the text, which would catch e.g. "[Success] Processing site: .../error-reports".
|
||||
function Get-RunMessage($Session, [string]$Status) {
|
||||
if ($Status -eq 'running') { return '' }
|
||||
$lines = New-Object 'System.Collections.Generic.List[string]'
|
||||
$sessionStatus = [string](Get-FirstValue $Session @('Status', 'Result', 'State'))
|
||||
if ($Status -ne 'success') {
|
||||
foreach ($entry in @(Get-FirstValue $Session @('Log', 'Logs') | Where-Object { $_ })) {
|
||||
$title = ([string](Get-FirstValue $entry @('Title', 'Message', 'Text'))).Trim()
|
||||
$prefix = [regex]::Match($title, '^\[(\w+)\]\s*')
|
||||
if ($prefix.Success) {
|
||||
if ($prefix.Groups[1].Value -notmatch 'Warn|Fail|Error') { continue }
|
||||
$title = $title.Substring($prefix.Length)
|
||||
} elseif ([string](Get-FirstValue $entry @('Status', 'Type', 'Severity')) -notmatch 'Warn|Fail|Error') {
|
||||
continue
|
||||
}
|
||||
if ($title -and -not $lines.Contains($title)) { $lines.Add($title) }
|
||||
}
|
||||
}
|
||||
if ($lines.Count -eq 0) {
|
||||
if ($Status -eq 'success') { $lines.Add('Job completed successfully.') }
|
||||
else { $lines.Add("Job finished with status '$sessionStatus'.") }
|
||||
} elseif ($sessionStatus -and $sessionStatus -notmatch 'Fail|Warn') {
|
||||
$lines.Insert(0, "Session status: $sessionStatus")
|
||||
}
|
||||
# A warning/failure VB365 is about to retry; the retry arrives as its own run
|
||||
if ($Status -ne 'success' -and (Get-FirstValue $Session @('JobWillBeRetried')) -eq $true) {
|
||||
$lines.Add('Veeam will retry this job.')
|
||||
}
|
||||
return ($lines -join "`n")
|
||||
}
|
||||
|
||||
function ConvertTo-Run($Session, $Job, [string]$JobKind) {
|
||||
$status = Get-RunStatus $Session
|
||||
$details = [ordered]@{ vb365_server = $env:COMPUTERNAME; job_kind = $JobKind }
|
||||
$org = Get-OrganizationName $Job
|
||||
if ($org) { $details.organization = $org }
|
||||
$sessionStatus = Get-FirstValue $Session @('Status', 'Result', 'State')
|
||||
if ($sessionStatus) { $details.vb365_status = [string]$sessionStatus }
|
||||
# Retries are separate sessions pointing at the session they retry
|
||||
$mainSession = Get-FirstValue $Session @('MainSessionId')
|
||||
if (-not (Test-EmptyGuid $mainSession)) { $details.retry_of = [string]$mainSession }
|
||||
if ((Get-FirstValue $Session @('JobWillBeRetried')) -eq $true) { $details.will_retry = $true }
|
||||
|
||||
$run = [ordered]@{
|
||||
external_id = [string](Get-FirstValue $Session @('Id', 'SessionId'))
|
||||
status = $status
|
||||
started_at = ConvertTo-IsoTime (Get-Time $Session 'CreationTime')
|
||||
message = Get-RunMessage $Session $status
|
||||
details = $details
|
||||
}
|
||||
if ($status -ne 'running') {
|
||||
$finished = ConvertTo-IsoTime (Get-Time $Session 'EndTime')
|
||||
if ($finished) { $run.finished_at = $finished } # otherwise the API uses receipt time
|
||||
$stats = Get-FirstValue $Session @('Statistics')
|
||||
$processed = Get-FirstValue $stats @('ProcessedObjects')
|
||||
if ($null -ne $processed) { $details.processed_objects = $processed }
|
||||
$transferred = ConvertTo-Bytes (Get-FirstValue $stats @('TransferredData', 'TransferredDataSize'))
|
||||
if ($transferred -gt 0) { $run.size_bytes = $transferred }
|
||||
}
|
||||
return $run
|
||||
}
|
||||
|
||||
# ---------- Main ----------
|
||||
|
||||
try {
|
||||
$Config = Initialize-Collector -Name 'vb365' -ConfigPath $ConfigPath -LookbackHours $LookbackHours -DryRun:$DryRun -RequiredKeys 'ApiUrl', 'ApiKey'
|
||||
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Veeam Backup for M365' }
|
||||
$LookbackHours = $Config.LookbackHours
|
||||
|
||||
Import-Vb365
|
||||
|
||||
$since = (Get-Date).AddHours(-$LookbackHours)
|
||||
|
||||
# (job, kind, sessions) for backup jobs, plus backup copy jobs where the cmdlets exist
|
||||
$work = New-Object 'System.Collections.Generic.List[object]'
|
||||
foreach ($job in @(Get-VBOJob)) {
|
||||
$sessions = @(Get-VBOJobSession -Job $job | Where-Object { (Get-Time $_ 'CreationTime') -ge $since })
|
||||
$work.Add([pscustomobject]@{ Job = $job; Kind = 'backup'; Sessions = $sessions })
|
||||
}
|
||||
if ((Get-Command Get-VBOCopyJob -ErrorAction SilentlyContinue) -and (Get-Command Get-VBOCopyJobSession -ErrorAction SilentlyContinue)) {
|
||||
$copyJobs = @(Get-VBOCopyJob)
|
||||
if ($copyJobs.Count) {
|
||||
# Fetch once and match by job ID, since the per-job parameter name isn't consistent
|
||||
$copySessions = @(Get-VBOCopyJobSession | Where-Object { (Get-Time $_ 'CreationTime') -ge $since })
|
||||
foreach ($job in $copyJobs) {
|
||||
$id = [string]$job.Id
|
||||
$work.Add([pscustomobject]@{
|
||||
Job = $job; Kind = 'copy'
|
||||
Sessions = @($copySessions | Where-Object { [string](Get-FirstValue $_ @('JobId')) -eq $id })
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
Write-Log "Collector setup failed: $(Get-ErrorText $_)" 'ERROR'
|
||||
exit 2
|
||||
}
|
||||
|
||||
$failures = 0
|
||||
foreach ($item in $work) {
|
||||
$job = $item.Job
|
||||
$tenant = Get-TenantForJob $job
|
||||
$jobInfo = [ordered]@{
|
||||
external_id = [string]$job.Id
|
||||
name = [string]$job.Name
|
||||
# Always sent (null when disabled/unscheduled) so the dashboard's value tracks VB365's
|
||||
next_run_at = Get-NextRun $job
|
||||
}
|
||||
$pending = @($item.Sessions |
|
||||
Where-Object { $_ -and -not (Test-RunSent ([string](Get-FirstValue $_ @('Id', 'SessionId')))) } |
|
||||
Sort-Object { Get-Time $_ 'CreationTime' })
|
||||
|
||||
try {
|
||||
if ($pending.Count -eq 0) {
|
||||
# Keep next_run_at fresh even when nothing new ran
|
||||
Send-JobReport -Tenant $tenant -SourceSystem $SourceSystem -Job $jobInfo
|
||||
}
|
||||
foreach ($session in $pending) {
|
||||
$run = ConvertTo-Run $session $job $item.Kind
|
||||
Send-JobReport -Tenant $tenant -SourceSystem $SourceSystem -Job $jobInfo -Run $run -SentKey $run.external_id -StartedAt ((Get-Time $session 'CreationTime'))
|
||||
}
|
||||
} catch {
|
||||
$failures++
|
||||
Write-Log "Failed to report job '$($job.Name)': $(Get-ErrorText $_)" 'ERROR'
|
||||
if (Test-AuthRejected $_) {
|
||||
Write-Log 'API key rejected; check ApiKey in the config. Stopping.' 'ERROR'
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$exitCode = Complete-Collector -JobCount $work.Count -Failures $failures
|
||||
if ($exitCode) { exit $exitCode }
|
||||
@@ -0,0 +1,78 @@
|
||||
# Stand-ins for the Veeam Backup for Microsoft 365 cmdlets. Object shapes follow
|
||||
# Get-Vb365CollectorDiagnostics.ps1 output from a real VB365 8.4 server: session and NextRun times
|
||||
# are local, Statistics.TransferredData is a string ("187.2 MB"), log items carry their status as a
|
||||
# Title prefix ("[Warning] ..."), and retries are separate sessions whose MainSessionId points at
|
||||
# the session they retry (which has JobWillBeRetried = True).
|
||||
#
|
||||
# . ./tests/MockVb365.ps1
|
||||
# ./Send-Vb365BackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||
|
||||
# Fixed to the top of the hour, and IDs derived from names/times, so repeated runs see the same data
|
||||
$now = Get-Date -Minute 0 -Second 0 -Millisecond 0
|
||||
function New-StableGuid([string]$Seed) {
|
||||
[guid]::new([Security.Cryptography.MD5]::Create().ComputeHash([Text.Encoding]::UTF8.GetBytes($Seed)))
|
||||
}
|
||||
|
||||
function New-MockOrg($Name) { [pscustomobject]@{ Name = $Name; OfficeName = $Name; Type = 'Office365'; IsBackedUp = $true } }
|
||||
$acme = New-MockOrg 'acme.onmicrosoft.com'
|
||||
$blue = New-MockOrg 'bluewater.onmicrosoft.com'
|
||||
|
||||
function New-MockJob($Name, $Org, $NextRun, [bool]$Enabled = $true) {
|
||||
[pscustomobject]@{
|
||||
Id = New-StableGuid $Name; Name = $Name; Organization = $Org; JobType = 'Backup'
|
||||
IsEnabled = $Enabled; NextRun = $NextRun; LastStatus = 'Success'
|
||||
LastRun = $(if ($NextRun) { $NextRun.AddDays(-1).ToUniversalTime() }) # UTC, as on the real server
|
||||
}
|
||||
}
|
||||
|
||||
function New-LogItem($Status, $Text, $At) {
|
||||
[pscustomobject]@{ Id = [guid]::NewGuid(); USN = 1; Title = "[$Status] $Text"; CreationTime = $At; EndTime = $At }
|
||||
}
|
||||
|
||||
function New-MockSession($Job, $Start, $End, $Status, $Transferred = '0 B', [long]$Objects = 0, $Log = @(),
|
||||
$MainSessionId = [guid]::Empty, [bool]$WillRetry = $false) {
|
||||
$logItems = @((New-LogItem 'Success' "Job started at: $($Start.ToString('G'))" $Start)) + @($Log)
|
||||
[pscustomobject]@{
|
||||
JobId = $Job.Id; JobName = $Job.Name; Status = $Status
|
||||
Statistics = [pscustomobject]@{ ProcessingRate = '1.6 MB/s (4 items/s)'; TransferredData = $Transferred; ProcessedObjects = $Objects; Bottleneck = 'Target' }
|
||||
JobSessionConfigType = 'Incremental'
|
||||
Id = New-StableGuid "$($Job.Name)|$($Start.Ticks)"
|
||||
CreationTime = $Start
|
||||
EndTime = $(if ($End) { $End } else { [datetime]::MinValue })
|
||||
Progress = 49; MainSessionId = $MainSessionId; JobWillBeRetried = $WillRetry
|
||||
Log = $logItems
|
||||
}
|
||||
}
|
||||
|
||||
$global:MockVboJobs = @(
|
||||
(New-MockJob 'Acme 365 Backup' $acme $now.Date.AddHours(17).AddDays($(if ($now.Hour -ge 17) { 1 } else { 0 }))),
|
||||
(New-MockJob 'Bluewater 365 Backup' $blue $now.Date.AddHours(17.75).AddDays($(if ($now.Hour -ge 17) { 1 } else { 0 }))),
|
||||
(New-MockJob 'Bluewater Old Mailboxes' $blue $now.AddDays(1) $false) # disabled
|
||||
)
|
||||
$acmeJob, $blueJob, $oldJob = $global:MockVboJobs
|
||||
|
||||
$y = $now.Date.AddDays(-1) # yesterday
|
||||
$blueWarn = New-MockSession $blueJob $y.AddHours(17.75) $y.AddHours(17.75).AddMinutes(3.5) 'Warning' '32.9 MB' 47 @(
|
||||
(New-LogItem 'Success' 'Found 94 objects' $y.AddHours(17.75)),
|
||||
# A success line mentioning "error" must not end up in the message
|
||||
(New-LogItem 'Success' 'Processing site: https://bluewater.sharepoint.com/sites/error-reports' $y.AddHours(17.76)),
|
||||
(New-LogItem 'Warning' 'Processing mailbox [email protected] finished with warning: Item "Q3.pst" exceeds the maximum size' $y.AddHours(17.78))
|
||||
) -WillRetry $true
|
||||
|
||||
$global:MockVboSessions = @(
|
||||
(New-MockSession $acmeJob $y.AddHours(17) $y.AddHours(17).AddMinutes(4.8) 'Success' '187.2 MB' 568 @(
|
||||
(New-LogItem 'Success' 'Found 49 objects' $y.AddHours(17)), (New-LogItem 'Success' 'Connected to organization' $y.AddHours(17)))),
|
||||
(New-MockSession $acmeJob $now.AddMinutes(-20) $null 'Running' '0 B' 12),
|
||||
$blueWarn,
|
||||
# Its retry 13 minutes later, which succeeded
|
||||
(New-MockSession $blueJob $y.AddHours(17.75).AddMinutes(13.5) $y.AddHours(17.75).AddMinutes(16.8) 'Success' '0 B' 0 -MainSessionId $blueWarn.Id),
|
||||
(New-MockSession $blueJob $now.AddHours(-30) $now.AddHours(-29.9) 'Failed' '0 B' 0 @(
|
||||
(New-LogItem 'Failed' 'Failed to connect to organization: The remote server returned an error: (401) Unauthorized.' $now.AddHours(-30)))),
|
||||
(New-MockSession $oldJob $now.AddDays(-5) $now.AddDays(-5).AddHours(1) 'Success') # outside 48h lookback
|
||||
)
|
||||
|
||||
function Get-VBOJob { $global:MockVboJobs }
|
||||
function Get-VBOJobSession { param($Job) $global:MockVboSessions | Where-Object { -not $Job -or $_.JobId -eq $Job.Id } }
|
||||
function Get-VBOOrganization { $acme, $blue }
|
||||
function Get-VBOCopyJob { } # none, as on the real server
|
||||
function Get-VBOCopyJobSession { }
|
||||
@@ -9,6 +9,7 @@
|
||||
2. sessions whose JobId matches no job (the runs that are going missing)
|
||||
3. for jobs with no matching sessions, their type/link-related properties and methods
|
||||
4. what other job cmdlets (e.g. Get-VBRBackupCopyJob) return
|
||||
5. each job's schedule properties, which the collector reads for the next run
|
||||
Nothing is changed in Veeam or the dashboard. Output is also saved to a text file.
|
||||
|
||||
.EXAMPLE
|
||||
@@ -42,14 +43,14 @@ function Format-Value($Value) {
|
||||
|
||||
# Property and method names matching $LinkPattern, with property values. Every read is guarded:
|
||||
# some Veeam properties throw when accessed.
|
||||
function Show-LinkMembers($Object) {
|
||||
foreach ($p in $Object.PSObject.Properties | Where-Object { $_.Name -match $LinkPattern }) {
|
||||
function Show-LinkMembers($Object, [string]$Pattern = $LinkPattern, [string]$Indent = ' ') {
|
||||
foreach ($p in $Object.PSObject.Properties | Where-Object { $_.Name -match $Pattern }) {
|
||||
$value = try { Format-Value $p.Value } catch { "<error: $($_.Exception.Message)>" }
|
||||
' {0} = {1}' -f $p.Name, $value
|
||||
'{0}{1} = {2}' -f $Indent, $p.Name, $value
|
||||
}
|
||||
$methods = @($Object | Get-Member -MemberType Methods -ErrorAction SilentlyContinue |
|
||||
Where-Object { $_.Name -match $LinkPattern } | ForEach-Object Name)
|
||||
if ($methods) { ' methods: ' + ($methods -join ', ') }
|
||||
Where-Object { $_.Name -match $Pattern } | ForEach-Object Name)
|
||||
if ($methods) { $Indent + 'methods: ' + ($methods -join ', ') }
|
||||
}
|
||||
|
||||
function Get-Prop($Object, [string]$Name) {
|
||||
@@ -147,6 +148,39 @@ $report = & {
|
||||
" ${cmd}: error: $($_.Exception.Message)"
|
||||
}
|
||||
}
|
||||
''
|
||||
|
||||
# The collector uses IsScheduleEnabled and ScheduleOptions.NextRun; show what else is there in
|
||||
# case this version keeps the next run somewhere different
|
||||
"== 5. Schedules (culture: $([Globalization.CultureInfo]::CurrentCulture.Name))"
|
||||
$SchedulePattern = 'Schedul|Next|Latest|Run|Enabled|Daily|Periodic|Monthly|Continuous|After|Option'
|
||||
foreach ($j in $jobs) {
|
||||
" $($j.Name) [$(Get-Prop $j 'JobType')]"
|
||||
Show-LinkMembers $j $SchedulePattern ' '
|
||||
foreach ($source in 'ScheduleOptions', 'GetScheduleOptions()', 'Get-VBRJobScheduleOptions') {
|
||||
try {
|
||||
$opts = switch ($source) {
|
||||
'ScheduleOptions' { $j.ScheduleOptions }
|
||||
'GetScheduleOptions()' { $j.GetScheduleOptions() }
|
||||
'Get-VBRJobScheduleOptions' { Get-VBRJobScheduleOptions -Job $j -WarningAction SilentlyContinue }
|
||||
}
|
||||
if ($null -eq $opts) { " ${source}: <null>"; continue }
|
||||
" ${source} ($($opts.GetType().FullName)):"
|
||||
foreach ($p in $opts.PSObject.Properties) {
|
||||
$value = try { $p.Value } catch { "<error: $($_.Exception.Message)>" }
|
||||
$type = if ($null -eq $value) { 'null' } else { $value.GetType().Name }
|
||||
if ($value -is [string] -or $value -is [ValueType] -or $null -eq $value) {
|
||||
' {0} ({1}) = {2}' -f $p.Name, $type, (Format-Value $value)
|
||||
} else {
|
||||
' {0} ({1})' -f $p.Name, $type
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
" ${source}: not available: $($_.Exception.Message)"
|
||||
}
|
||||
}
|
||||
''
|
||||
}
|
||||
} | Out-String -Width 300
|
||||
|
||||
$report
|
||||
|
||||
@@ -22,7 +22,7 @@ param(
|
||||
[Parameter(Mandatory)][string]$ApiUrl,
|
||||
[Parameter(Mandatory)][string]$Tenant,
|
||||
[string]$SourceSystem = 'Veeam B&R',
|
||||
[ValidateRange(5, 1440)][int]$IntervalMinutes = 15,
|
||||
[ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
|
||||
[ValidateRange(1, 720)][int]$LookbackHours = 48,
|
||||
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard'),
|
||||
# Defaults to the machine-wide PowerShell 7 install
|
||||
@@ -30,9 +30,6 @@ param(
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$TaskName = 'Backup Dashboard - Veeam collector'
|
||||
$DataDir = Join-Path $env:ProgramData 'BackupDashboard'
|
||||
$ConfigPath = Join-Path $DataDir 'veeam-collector.json'
|
||||
|
||||
# Find PowerShell 7 before changing anything, so a missing install fails cleanly
|
||||
if (-not $PwshPath) {
|
||||
@@ -54,49 +51,18 @@ if ($LASTEXITCODE -or ($pwshMajor -as [int]) -lt 7) {
|
||||
throw "'$PwshPath' isn't a working PowerShell 7 (reported '$pwshVersion')."
|
||||
}
|
||||
|
||||
if ($ApiUrl -notmatch '^https://') {
|
||||
Write-Warning 'ApiUrl is not HTTPS: the API key would be sent in clear text.'
|
||||
# Shared installer code: in the repo it's in ..\common, so copy the whole collectors folder
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
throw "BackupDashboard.Collector.psm1 not found. Copy the whole collectors folder, including common."
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
# Read the key as a SecureString so it isn't echoed to the console or kept in shell history
|
||||
$secure = Read-Host -AsSecureString 'API key (from: python manage.py create-key <name>)'
|
||||
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
|
||||
try { $apiKey = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) }
|
||||
finally { [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr) }
|
||||
if (-not $apiKey.StartsWith('bdk_')) { throw "That doesn't look like a dashboard API key (expected 'bdk_...')." }
|
||||
|
||||
# Config folder: SYSTEM + Administrators only (SIDs so this works on non-English Windows)
|
||||
New-Item -ItemType Directory -Path $DataDir -Force | Out-Null
|
||||
& icacls.exe $DataDir /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' | Out-Null
|
||||
if ($LASTEXITCODE) { throw "icacls failed on $DataDir" }
|
||||
|
||||
[ordered]@{
|
||||
ApiUrl = $ApiUrl.TrimEnd('/')
|
||||
ApiKey = $apiKey
|
||||
Tenant = $Tenant
|
||||
SourceSystem = $SourceSystem
|
||||
LookbackHours = $LookbackHours
|
||||
} | ConvertTo-Json | Set-Content -Path $ConfigPath -Encoding UTF8
|
||||
|
||||
New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null
|
||||
$scriptPath = Join-Path $InstallDir 'Send-VeeamBackupStatus.ps1'
|
||||
Copy-Item -Path (Join-Path $PSScriptRoot 'Send-VeeamBackupStatus.ps1') -Destination $scriptPath -Force
|
||||
Unblock-File -Path $scriptPath
|
||||
|
||||
$action = New-ScheduledTaskAction -Execute $PwshPath `
|
||||
-Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$scriptPath`""
|
||||
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date).AddMinutes(1) `
|
||||
-RepetitionInterval (New-TimeSpan -Minutes $IntervalMinutes)
|
||||
$principal = New-ScheduledTaskPrincipal -UserId 'S-1-5-18' -LogonType ServiceAccount -RunLevel Highest
|
||||
$settings = New-ScheduledTaskSettingsSet -ExecutionTimeLimit (New-TimeSpan -Minutes 30) `
|
||||
-MultipleInstances IgnoreNew -StartWhenAvailable
|
||||
Register-ScheduledTask -TaskName $TaskName -Action $action -Trigger $trigger -Principal $principal `
|
||||
-Settings $settings -Description 'Sends Veeam job results to the Backup Dashboard' -Force | Out-Null
|
||||
|
||||
Write-Host "Installed. Config: $ConfigPath"
|
||||
Write-Host "Task '$TaskName' runs every $IntervalMinutes minutes as SYSTEM, using PowerShell $pwshVersion ($PwshPath)."
|
||||
Write-Host ''
|
||||
Write-Host 'Next steps:'
|
||||
Write-Host " 1. Preview what will be sent: & '$PwshPath' -File '$scriptPath' -DryRun"
|
||||
Write-Host " 2. Backfill 30 days of history: & '$PwshPath' -File '$scriptPath' -LookbackHours 720"
|
||||
Write-Host " 3. Check the log: Get-Content '$DataDir\veeam-collector.log' -Tail 20"
|
||||
$settings = [ordered]@{ Tenant = $Tenant; SourceSystem = $SourceSystem; LookbackHours = $LookbackHours }
|
||||
Install-CollectorTask -Name 'veeam' -ScriptPath (Join-Path $PSScriptRoot 'Send-VeeamBackupStatus.ps1') `
|
||||
-TaskName 'Backup Dashboard - Veeam collector' `
|
||||
-TaskDescription 'Sends Veeam job results to the Backup Dashboard' `
|
||||
-Shell $PwshPath -ApiUrl $ApiUrl -IntervalMinutes $IntervalMinutes -InstallDir $InstallDir -Settings $settings
|
||||
|
||||
@@ -5,25 +5,33 @@ It covers jobs returned by `Get-VBRJob`: VM backup, replication and backup copy
|
||||
|
||||
## Install (on the VBR server, as an administrator)
|
||||
|
||||
Requires PowerShell 7 installed machine-wide (`winget install Microsoft.PowerShell` or the MSI),
|
||||
which VBR 13's Veeam module expects. The Microsoft Store version won't work, because the SYSTEM
|
||||
account can't run it. The installer checks for this before changing anything.
|
||||
|
||||
1. On the dashboard host, create a key for this server:
|
||||
```bash
|
||||
python manage.py create-key veeam-acme
|
||||
```
|
||||
2. Copy this folder to the VBR server and run:
|
||||
2. Copy the whole `collectors` folder to the VBR server (the installer needs `common`
|
||||
as well as this folder), then from `collectors\veeam` run, from PowerShell 7 (`pwsh`):
|
||||
```powershell
|
||||
.\Install-VeeamCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
```
|
||||
It prompts for the key, then:
|
||||
- writes `C:\ProgramData\BackupDashboard\veeam-collector.json` (readable only by SYSTEM and Administrators)
|
||||
- copies the script to `C:\Program Files\BackupDashboard\`
|
||||
- registers the scheduled task **Backup Dashboard - Veeam collector** (SYSTEM, every 15 minutes)
|
||||
3. Preview the payloads, then backfill 30 days so the history view is populated straight away:
|
||||
- registers the scheduled task **Backup Dashboard - Veeam collector** (SYSTEM, every 60 minutes,
|
||||
run by `C:\Program Files\PowerShell\7\pwsh.exe`; use `-PwshPath` if it's installed elsewhere)
|
||||
3. Preview the payloads, then backfill 30 days so the history view is populated straight away
|
||||
(from `pwsh`):
|
||||
```powershell
|
||||
& 'C:\Program Files\BackupDashboard\Send-VeeamBackupStatus.ps1' -DryRun
|
||||
& 'C:\Program Files\BackupDashboard\Send-VeeamBackupStatus.ps1' -LookbackHours 720
|
||||
```
|
||||
|
||||
Re-run the installer to change settings or update the script. To remove the collector,
|
||||
Re-run the installer to change settings or update the script. It also installs
|
||||
`BackupDashboard.Collector.psm1`, the code shared by all collectors, next to the script. To remove the collector,
|
||||
delete the scheduled task and both `BackupDashboard` folders.
|
||||
|
||||
## What it sends
|
||||
@@ -34,10 +42,14 @@ delete the scheduled task and both `BackupDashboard` folders.
|
||||
| Job | Job name. The job's GUID is the ID, so renaming a job in Veeam doesn't duplicate it |
|
||||
| Status | Session result (Success / Warning / Failed). Any state other than Stopped is **Running**. A session that stopped with no result (e.g. cancelled) is reported as **Warning** |
|
||||
| Started / finished | Session `CreationTime` / `EndTime` |
|
||||
| Next scheduled | Job's next run. Sent as empty for disabled jobs and chained ("run after") jobs, so those are never flagged Overdue |
|
||||
| Next scheduled | Job's next run, from the live schedule (`GetScheduleOptions()`). Sent as empty, so never flagged Overdue, for disabled jobs, chained ("run after") jobs, and immediate-mode backup copy jobs, which run whenever their source does |
|
||||
| Message | Session reason, each VM or object that didn't succeed, and warning/error lines from the session log |
|
||||
| Size | `BackupStats.BackupSize` |
|
||||
|
||||
**Backup copy jobs (VBR 12+):** Veeam records their sessions under hidden per-source "worker" jobs.
|
||||
The collector files each worker session under its copy job (via the worker's `PolicyJobId`), so a
|
||||
copy job gets one run per source job it copies, with the source in `copied_from`.
|
||||
|
||||
Each run covers the last 48 hours (`LookbackHours`), so a few missed runs (server rebooting, API
|
||||
down) fill in automatically. Sessions already delivered are recorded in
|
||||
`veeam-collector.state.json`. Running sessions are re-sent until they finish.
|
||||
@@ -48,6 +60,9 @@ down) fill in automatically. Sessions already delivered are recorded in
|
||||
- Exit codes: `0` OK, `1` some jobs failed to send (they're retried next run), `2` setup failed
|
||||
(config, Veeam module, or Veeam connection).
|
||||
- `API key rejected`: the key was revoked or mistyped. Re-run the installer with a new key.
|
||||
- A job shows **Unknown** on the dashboard: the collector sent the job but matched no sessions to it.
|
||||
Run `.\Get-VeeamCollectorDiagnostics.ps1` (read-only). It lists sessions whose job ID matches no
|
||||
job, plus the type and link properties of the jobs left without sessions.
|
||||
- `Get-VBRBackupSession` loads every session in the VBR database. On a server with years of history,
|
||||
this can make each run take a minute or more. The task's 30-minute limit and "don't start if
|
||||
already running" setting keep that safe.
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
Reports Veeam Backup & Replication job results to the Backup Dashboard.
|
||||
|
||||
.DESCRIPTION
|
||||
Runs on the VBR server, normally as a SYSTEM scheduled task every 15 minutes
|
||||
Runs on the VBR server, normally as a SYSTEM scheduled task every 60 minutes
|
||||
(see Install-VeeamCollector.ps1). For every backup, replica and backup copy job it sends:
|
||||
- the job's next scheduled run (drives the dashboard's Overdue flag), and
|
||||
- every session from the last LookbackHours that the dashboard hasn't already received
|
||||
@@ -37,42 +37,19 @@ param(
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
$DataDir = Split-Path -Parent $ConfigPath
|
||||
$LogPath = Join-Path $DataDir 'veeam-collector.log'
|
||||
$StatePath = Join-Path $DataDir 'veeam-collector.state.json'
|
||||
$MaxLogBytes = 5MB
|
||||
$Invariant = [Globalization.CultureInfo]::InvariantCulture
|
||||
# Shared collector code: installed next to this script; in the repo, in ..\common
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
Write-Host "BackupDashboard.Collector.psm1 not found next to $PSCommandPath; re-run the installer."
|
||||
exit 2
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
# ---------- Helpers ----------
|
||||
|
||||
function Write-Log {
|
||||
param([string]$Message, [string]$Level = 'INFO')
|
||||
$line = '{0} [{1}] {2}' -f (Get-Date).ToString('yyyy-MM-dd HH:mm:ss'), $Level, $Message
|
||||
Write-Host $line
|
||||
if ($DryRun) { return }
|
||||
if ((Test-Path $LogPath) -and (Get-Item $LogPath).Length -gt $MaxLogBytes) {
|
||||
Move-Item $LogPath "$LogPath.1" -Force
|
||||
}
|
||||
Add-Content -Path $LogPath -Value $line -Encoding UTF8
|
||||
}
|
||||
|
||||
function Get-ErrorText($ErrorRecord) {
|
||||
# Windows PowerShell puts the HTTP response body (e.g. the API's 422 explanation) here
|
||||
if ($ErrorRecord.ErrorDetails -and $ErrorRecord.ErrorDetails.Message) {
|
||||
return "$($ErrorRecord.Exception.Message) $($ErrorRecord.ErrorDetails.Message)"
|
||||
}
|
||||
return $ErrorRecord.Exception.Message
|
||||
}
|
||||
|
||||
# The API needs an explicit UTC offset. Veeam returns local times, often with Kind=Unspecified,
|
||||
# which DateTimeOffset treats as local, exactly what we want.
|
||||
function ConvertTo-IsoTime($Value) {
|
||||
if ($null -eq $Value) { return $null }
|
||||
$dt = [datetime]$Value
|
||||
if ($dt.Year -le 1900) { return $null } # Veeam's "not set" value (e.g. EndTime while running)
|
||||
return ([DateTimeOffset]$dt).ToString("yyyy-MM-dd'T'HH:mm:sszzz", $Invariant)
|
||||
}
|
||||
|
||||
function Import-Veeam {
|
||||
if (Get-Command Get-VBRJob -ErrorAction SilentlyContinue) { return } # already loaded
|
||||
if (Get-Module -ListAvailable -Name Veeam.Backup.PowerShell) {
|
||||
@@ -85,14 +62,25 @@ function Import-Veeam {
|
||||
function Get-NextRun($Job) {
|
||||
try {
|
||||
if (-not $Job.IsScheduleEnabled) { return $null }
|
||||
$raw = $Job.ScheduleOptions.NextRun
|
||||
# Immediate-mode backup copy jobs run whenever their source job does. Their schedule still
|
||||
# carries a NextRun, but it's a stale date (months old on VBR 13) that would show as Overdue.
|
||||
if ($Job.IsContinuous) { return $null }
|
||||
|
||||
# The ScheduleOptions property is a stale snapshot on VBR 13 (NextRun empty, last run years
|
||||
# old); GetScheduleOptions() reads the live schedule. Fall back for versions without it.
|
||||
$options = $null
|
||||
try { $options = $Job.GetScheduleOptions() } catch { }
|
||||
if (-not $options) { $options = $Job.ScheduleOptions }
|
||||
$raw = $options.NextRun
|
||||
if (-not $raw) { return $null } # e.g. chained jobs that run "after" another job
|
||||
if ($raw -is [datetime]) { return ConvertTo-IsoTime $raw }
|
||||
# Older versions return a string formatted with the server's culture, so parse with it
|
||||
$parsed = [datetime]::MinValue
|
||||
if ([datetime]::TryParse([string]$raw, [Globalization.CultureInfo]::CurrentCulture,
|
||||
[Globalization.DateTimeStyles]::None, [ref]$parsed)) {
|
||||
return ConvertTo-IsoTime $parsed
|
||||
# A string formatted by VBR ("2026-10-09 12:15:00 AM" on VBR 13, en-CA). Try the server's
|
||||
# culture first, then the invariant one
|
||||
foreach ($culture in [Globalization.CultureInfo]::CurrentCulture, [Globalization.CultureInfo]::InvariantCulture) {
|
||||
$parsed = [datetime]::MinValue
|
||||
if ([datetime]::TryParse([string]$raw, $culture, [Globalization.DateTimeStyles]::None, [ref]$parsed)) {
|
||||
return ConvertTo-IsoTime $parsed
|
||||
}
|
||||
}
|
||||
Write-Log "Couldn't parse next run '$raw' for job '$($Job.Name)'" 'WARN'
|
||||
} catch {
|
||||
@@ -206,51 +194,15 @@ function ConvertTo-Run($Session, $Job) {
|
||||
return $run
|
||||
}
|
||||
|
||||
function Send-Report($Body) {
|
||||
$json = $Body | ConvertTo-Json -Depth 6 -Compress
|
||||
if ($DryRun) {
|
||||
Write-Host $json
|
||||
return
|
||||
}
|
||||
# Send bytes: Windows PowerShell would otherwise encode a string body as Latin-1,
|
||||
# mangling non-ASCII job names (e.g. en dashes).
|
||||
$params = @{
|
||||
Uri = "$($Config.ApiUrl.TrimEnd('/'))/api/v1/ingest"
|
||||
Method = 'Post'
|
||||
Headers = @{ Authorization = "Bearer $($Config.ApiKey)" }
|
||||
ContentType = 'application/json; charset=utf-8'
|
||||
Body = [Text.Encoding]::UTF8.GetBytes($json)
|
||||
TimeoutSec = 30
|
||||
}
|
||||
Invoke-RestMethod @params | Out-Null
|
||||
}
|
||||
|
||||
# ---------- Main ----------
|
||||
|
||||
try {
|
||||
$Config = Get-Content -Path $ConfigPath -Raw | ConvertFrom-Json
|
||||
foreach ($key in 'ApiUrl', 'ApiKey', 'Tenant') {
|
||||
if (-not $Config.$key) { throw "Config $ConfigPath is missing '$key'" }
|
||||
}
|
||||
$Config = Initialize-Collector -Name 'veeam' -ConfigPath $ConfigPath -LookbackHours $LookbackHours -DryRun:$DryRun
|
||||
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Veeam B&R' }
|
||||
if ($LookbackHours -le 0) {
|
||||
$LookbackHours = if ($Config.LookbackHours) { [int]$Config.LookbackHours } else { 48 }
|
||||
}
|
||||
|
||||
# Windows PowerShell 5.1 may default to TLS 1.0
|
||||
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
|
||||
$LookbackHours = $Config.LookbackHours
|
||||
|
||||
Import-Veeam
|
||||
|
||||
# Session ID -> start time (Unix seconds), for sessions already delivered in their final state.
|
||||
# Numbers rather than ISO strings: PowerShell 7's ConvertFrom-Json turns date strings into
|
||||
# DateTimes and loses the offset.
|
||||
$sent = @{}
|
||||
if (Test-Path $StatePath) {
|
||||
$saved = Get-Content -Path $StatePath -Raw | ConvertFrom-Json
|
||||
if ($saved) { foreach ($p in $saved.PSObject.Properties) { $sent[$p.Name] = [long]$p.Value } }
|
||||
}
|
||||
|
||||
$since = (Get-Date).AddHours(-$LookbackHours)
|
||||
$jobs = @(Get-VBRJob -WarningAction SilentlyContinue)
|
||||
|
||||
@@ -277,7 +229,6 @@ try {
|
||||
}
|
||||
|
||||
$failures = 0
|
||||
$runsSent = 0
|
||||
foreach ($job in $jobs) {
|
||||
$jobInfo = [ordered]@{
|
||||
external_id = $job.Id.ToString()
|
||||
@@ -288,48 +239,28 @@ foreach ($job in $jobs) {
|
||||
$pending = @()
|
||||
if ($sessionsByJob.ContainsKey($jobInfo.external_id)) {
|
||||
$pending = @($sessionsByJob[$jobInfo.external_id] |
|
||||
Where-Object { -not $sent.ContainsKey($_.Id.ToString()) } |
|
||||
Where-Object { -not (Test-RunSent ($_.Id.ToString())) } |
|
||||
Sort-Object CreationTime)
|
||||
}
|
||||
|
||||
try {
|
||||
if ($pending.Count -eq 0) {
|
||||
# Keep next_run_at fresh even when nothing new ran
|
||||
Send-Report ([ordered]@{ tenant = $Config.Tenant; source_system = $SourceSystem; job = $jobInfo })
|
||||
Send-JobReport -Tenant $Config.Tenant -SourceSystem $SourceSystem -Job $jobInfo
|
||||
}
|
||||
foreach ($session in $pending) {
|
||||
$run = ConvertTo-Run $session $job
|
||||
Send-Report ([ordered]@{ tenant = $Config.Tenant; source_system = $SourceSystem; job = $jobInfo; run = $run })
|
||||
$runsSent++
|
||||
if ($run.status -ne 'running') {
|
||||
$sent[$run.external_id] = ([DateTimeOffset][datetime]$session.CreationTime).ToUnixTimeSeconds()
|
||||
}
|
||||
Send-JobReport -Tenant $Config.Tenant -SourceSystem $SourceSystem -Job $jobInfo -Run $run -SentKey $run.external_id -StartedAt ([datetime]$session.CreationTime)
|
||||
}
|
||||
} catch {
|
||||
$failures++
|
||||
Write-Log "Failed to report job '$($job.Name)': $(Get-ErrorText $_)" 'ERROR'
|
||||
$httpStatus = 0
|
||||
try { $httpStatus = [int]$_.Exception.Response.StatusCode } catch { }
|
||||
if ($httpStatus -eq 401) {
|
||||
if (Test-AuthRejected $_) {
|
||||
Write-Log 'API key rejected; check ApiKey in the config. Stopping.' 'ERROR'
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (-not $DryRun) {
|
||||
# Forget sessions that have aged out of the lookback window (plus a day of slack)
|
||||
$cutoff = [DateTimeOffset]::Now.AddHours(-$LookbackHours - 24).ToUnixTimeSeconds()
|
||||
$keep = [ordered]@{}
|
||||
foreach ($id in $sent.Keys) {
|
||||
if ($sent[$id] -ge $cutoff) { $keep[$id] = $sent[$id] }
|
||||
}
|
||||
try {
|
||||
$keep | ConvertTo-Json -Compress | Set-Content -Path $StatePath -Encoding UTF8
|
||||
} catch {
|
||||
Write-Log "Couldn't save state file: $(Get-ErrorText $_)" 'WARN'
|
||||
}
|
||||
}
|
||||
|
||||
Write-Log ("Reported {0} jobs, {1} runs, {2} failures" -f $jobs.Count, $runsSent, $failures)
|
||||
if ($failures) { exit 1 }
|
||||
$exitCode = Complete-Collector -JobCount $jobs.Count -Failures $failures
|
||||
if ($exitCode) { exit $exitCode }
|
||||
|
||||
@@ -12,12 +12,22 @@ function New-StableGuid([string]$Seed) {
|
||||
}
|
||||
$notSet = Get-Date '1900-01-01'
|
||||
|
||||
function New-MockJob($Name, $Type, $NextRun, [bool]$Enabled = $true) {
|
||||
[pscustomobject]@{
|
||||
# As on VBR 13: the ScheduleOptions property is a stale snapshot with an empty NextRun, and the live
|
||||
# schedule (NextRun as a "2026-10-09 12:15:00 AM" string) comes from GetScheduleOptions().
|
||||
# -Continuous: an immediate-mode copy job, whose live NextRun is a stale date in the past.
|
||||
function New-MockJob($Name, $Type, $NextRun, [bool]$Enabled = $true, [switch]$Continuous) {
|
||||
$job = [pscustomobject]@{
|
||||
Id = New-StableGuid $Name; Name = $Name; JobType = $Type
|
||||
IsScheduleEnabled = $Enabled
|
||||
ScheduleOptions = [pscustomobject]@{ NextRun = $NextRun }
|
||||
IsContinuous = [bool]$Continuous
|
||||
ScheduleOptions = [pscustomobject]@{ NextRun = '' }
|
||||
LiveNextRun = $NextRun
|
||||
}
|
||||
$job | Add-Member ScriptMethod GetScheduleOptions { [pscustomobject]@{ NextRun = $this.LiveNextRun } }
|
||||
$job
|
||||
}
|
||||
function Format-VbrTime([datetime]$Value) {
|
||||
$Value.ToString('yyyy-MM-dd h:mm:ss tt', [Globalization.CultureInfo]::InvariantCulture)
|
||||
}
|
||||
|
||||
function New-MockSession($Job, $Start, $End, $State, $Result, $Reason = '', $Tasks = @(), $LogRecords = @(), $Size = 0) {
|
||||
@@ -40,13 +50,13 @@ function New-MockTask($Name, $Status, $Reason = '') {
|
||||
}
|
||||
|
||||
$global:MockJobs = @(
|
||||
(New-MockJob 'File Server – Nightly' 'Backup' $now.Date.AddDays(1).AddHours(22)),
|
||||
# Older VBR versions return NextRun as a culture-formatted string
|
||||
(New-MockJob 'SQL01 – Logs' 'Backup' $now.AddMinutes(40).ToString()),
|
||||
(New-MockJob 'File Server – Nightly' 'Backup' (Format-VbrTime $now.Date.AddDays(1).AddMinutes(15))),
|
||||
# Some versions return NextRun as a DateTime
|
||||
(New-MockJob 'SQL01 – Logs' 'Backup' $now.AddMinutes(40)),
|
||||
(New-MockJob 'Offsite Copy' 'BackupSync' $null),
|
||||
(New-MockJob 'Old Exchange' 'Backup' $now.AddDays(1) $false),
|
||||
# VBR 12+ backup copy job (immediate mode, no schedule): the "policy" job Get-VBRJob returns
|
||||
(New-MockJob 'Cloud Copy' 'SimpleBackupCopyPolicy' $null)
|
||||
(New-MockJob 'Cloud Copy' 'SimpleBackupCopyPolicy' (Format-VbrTime $now.AddMonths(-6)) -Continuous)
|
||||
)
|
||||
$fs, $sql, $copy, $old, $cloud = $global:MockJobs
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Read-only dump of what the Windows Server Backup cmdlets return, for troubleshooting the collector.
|
||||
|
||||
.DESCRIPTION
|
||||
The collector reads properties of Get-WBJob / Get-WBSummary results whose names and time zones
|
||||
aren't fully documented. This prints those objects' real structure (property names, types and
|
||||
values, a few levels deep), plus recent events from the Microsoft-Windows-Backup log, whose
|
||||
timestamps are reliable local times to compare against. Nothing is changed. Output is also
|
||||
saved to a text file.
|
||||
|
||||
.EXAMPLE
|
||||
.\Get-WsbCollectorDiagnostics.ps1
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[int]$Jobs = 3,
|
||||
[int]$Events = 15,
|
||||
[string]$OutFile = (Join-Path $env:TEMP 'wsb-collector-diagnostics.txt')
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$MaxDepth = 6
|
||||
|
||||
if (-not (Get-Command Get-WBJob -ErrorAction SilentlyContinue)) {
|
||||
Import-Module WindowsServerBackup
|
||||
}
|
||||
|
||||
function Test-Scalar($Value) {
|
||||
return $null -eq $Value -or $Value -is [string] -or $Value -is [ValueType] -or $Value -is [guid]
|
||||
}
|
||||
|
||||
# Recursive "name (type) = value" listing. Collections show their first 3 items.
|
||||
function Show-Object($Value, [string]$Indent = ' ', [int]$Depth = 0) {
|
||||
if (Test-Scalar $Value) { return }
|
||||
if ($Depth -ge $MaxDepth) { "$Indent..."; return }
|
||||
if ($Value -is [Collections.IEnumerable]) {
|
||||
$items = @($Value)
|
||||
"$Indent[$($items.Count) item(s)]"
|
||||
$i = 0
|
||||
foreach ($item in $items | Select-Object -First 3) {
|
||||
"$Indent[$i] ($($item.GetType().FullName))$(if (Test-Scalar $item) { " = $item" })"
|
||||
Show-Object $item "$Indent " ($Depth + 1)
|
||||
$i++
|
||||
}
|
||||
return
|
||||
}
|
||||
foreach ($p in $Value.PSObject.Properties) {
|
||||
try { $v = $p.Value } catch { "$Indent$($p.Name) = <error: $($_.Exception.Message)>"; continue }
|
||||
$type = if ($null -eq $v) { 'null' } else { $v.GetType().FullName }
|
||||
if (Test-Scalar $v) {
|
||||
"$Indent$($p.Name) ($type) = $v"
|
||||
} else {
|
||||
$text = [string]$v
|
||||
if ($text.Length -gt 100) { $text = $text.Substring(0, 97) + '...' }
|
||||
"$Indent$($p.Name) ($type) = $text"
|
||||
Show-Object $v "$Indent " ($Depth + 1)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function Show-Section([string]$Title, [scriptblock]$Get) {
|
||||
"== $Title"
|
||||
try {
|
||||
$result = @(& $Get)
|
||||
if ($result.Count -eq 0 -or ($result.Count -eq 1 -and $null -eq $result[0])) { ' (nothing returned)' }
|
||||
$n = 0
|
||||
foreach ($item in $result) {
|
||||
if ($null -eq $item) { continue }
|
||||
" [$n] ($($item.GetType().FullName))"
|
||||
Show-Object $item ' '
|
||||
$n++
|
||||
}
|
||||
} catch {
|
||||
" error: $($_.Exception.Message)"
|
||||
}
|
||||
''
|
||||
}
|
||||
|
||||
$report = & {
|
||||
"Generated $(Get-Date -Format o) on $env:COMPUTERNAME, PowerShell $($PSVersionTable.PSVersion)"
|
||||
"Local time zone: $([TimeZoneInfo]::Local.Id), UTC offset now $([TimeZoneInfo]::Local.GetUtcOffset((Get-Date)))"
|
||||
''
|
||||
Show-Section 'Get-WBSummary' { Get-WBSummary }
|
||||
Show-Section 'Get-WBJob (in progress)' { Get-WBJob }
|
||||
Show-Section "Get-WBJob -Previous $Jobs" { Get-WBJob -Previous $Jobs }
|
||||
Show-Section 'Get-WBPolicy | Get-WBSchedule' {
|
||||
$policy = Get-WBPolicy
|
||||
if ($policy) { Get-WBSchedule -Policy $policy } else { '(no scheduled backup policy)' }
|
||||
}
|
||||
"== Microsoft-Windows-Backup event log (last $Events, local times)"
|
||||
try {
|
||||
Get-WinEvent -LogName 'Microsoft-Windows-Backup' -MaxEvents $Events |
|
||||
Select-Object TimeCreated, Id, LevelDisplayName, @{ n = 'Message'; e = { ($_.Message -split "`r?`n")[0] } } |
|
||||
Format-Table -AutoSize -Wrap | Out-String -Width 300
|
||||
} catch {
|
||||
" error: $($_.Exception.Message)"
|
||||
}
|
||||
} | Out-String -Width 300
|
||||
|
||||
$report
|
||||
$report | Set-Content -Path $OutFile -Encoding UTF8
|
||||
Write-Host "Saved to $OutFile"
|
||||
@@ -0,0 +1,54 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Installs the Windows Server Backup collector as a SYSTEM scheduled task.
|
||||
|
||||
.DESCRIPTION
|
||||
- Copies Send-WsbBackupStatus.ps1 to Program Files (only admins can modify it, which
|
||||
matters because it runs as SYSTEM).
|
||||
- Writes the config to ProgramData\BackupDashboard and restricts that folder to SYSTEM and
|
||||
Administrators, since it contains the API key.
|
||||
- Registers a scheduled task that runs every IntervalMinutes under Windows PowerShell 5.1
|
||||
(powershell.exe), which the WindowsServerBackup module targets.
|
||||
Safe to re-run to change settings or update the script. Can sit alongside the other collectors.
|
||||
|
||||
.EXAMPLE
|
||||
.\Install-WsbCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
(prompts for the API key)
|
||||
#>
|
||||
#Requires -RunAsAdministrator
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory)][string]$ApiUrl,
|
||||
[Parameter(Mandatory)][string]$Tenant,
|
||||
# Name shown on the dashboard; defaults to "<server> - Windows Server Backup"
|
||||
[string]$JobName,
|
||||
[string]$SourceSystem = 'Windows Server Backup',
|
||||
[ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
|
||||
[ValidateRange(1, 720)][int]$LookbackHours = 48,
|
||||
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard')
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$WindowsPowerShell = Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\powershell.exe'
|
||||
|
||||
# Check Windows Server Backup is here before changing anything
|
||||
if (-not (Get-Module -ListAvailable -Name WindowsServerBackup)) {
|
||||
throw 'The WindowsServerBackup PowerShell module is not available. Install the Windows Server Backup feature (Install-WindowsFeature Windows-Server-Backup).'
|
||||
}
|
||||
|
||||
# Shared installer code: in the repo it's in ..\common, so copy the whole collectors folder
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
throw "BackupDashboard.Collector.psm1 not found. Copy the whole collectors folder, including common."
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
$settings = [ordered]@{ Tenant = $Tenant; SourceSystem = $SourceSystem; LookbackHours = $LookbackHours }
|
||||
if ($JobName) { $settings.JobName = $JobName }
|
||||
Install-CollectorTask -Name 'wsb' -ScriptPath (Join-Path $PSScriptRoot 'Send-WsbBackupStatus.ps1') `
|
||||
-TaskName 'Backup Dashboard - WSB collector' `
|
||||
-TaskDescription 'Sends Windows Server Backup results to the Backup Dashboard' `
|
||||
-Shell $WindowsPowerShell -ApiUrl $ApiUrl -IntervalMinutes $IntervalMinutes -InstallDir $InstallDir -Settings $settings
|
||||
@@ -0,0 +1,71 @@
|
||||
# Windows Server Backup collector
|
||||
|
||||
A PowerShell script that runs on each server using Windows Server Backup (WSB) and reports its
|
||||
backups to the dashboard. Each server appears as one job, named `<SERVER> - Windows Server Backup`
|
||||
by default.
|
||||
|
||||
> **Checked against:** Server 2019 Hyper-V host backing up VMs, successful backups only.
|
||||
> Not yet seen on a real server: a failed backup, a failed VM or volume, or a backup in progress.
|
||||
> Those paths are written to the same object model, but if one looks wrong, run
|
||||
> `Get-WsbCollectorDiagnostics.ps1` while it's in that state and send the output.
|
||||
|
||||
## Install (on the server, as an administrator)
|
||||
|
||||
Requires the Windows Server Backup feature with its PowerShell module
|
||||
(`Install-WindowsFeature Windows-Server-Backup`). Runs under Windows PowerShell 5.1
|
||||
(`powershell.exe`), which that module targets.
|
||||
|
||||
1. On the dashboard host, create a key for this server:
|
||||
```bash
|
||||
python manage.py create-key wsb-acme-dc01
|
||||
```
|
||||
2. Copy the whole `collectors` folder to the server (the installer needs `common`
|
||||
as well as this folder), then from `collectors\wsb` run:
|
||||
```powershell
|
||||
.\Install-WsbCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
|
||||
```
|
||||
Add `-JobName "Domain Controller"` to choose the name shown on the dashboard. It prompts for the
|
||||
key, then:
|
||||
- writes `C:\ProgramData\BackupDashboard\wsb-collector.json` (readable only by SYSTEM and Administrators)
|
||||
- copies the script to `C:\Program Files\BackupDashboard\`
|
||||
- registers the scheduled task **Backup Dashboard - WSB collector** (SYSTEM, every 60 minutes)
|
||||
3. Preview the payloads, then backfill 30 days so the history view is populated straight away:
|
||||
```powershell
|
||||
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-WsbBackupStatus.ps1' -DryRun
|
||||
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-WsbBackupStatus.ps1' -LookbackHours 720
|
||||
```
|
||||
|
||||
It can be installed alongside the other collectors: it uses its own config, state, log and task.
|
||||
Re-run the installer to change settings or update the script. It also installs
|
||||
`BackupDashboard.Collector.psm1`, the code shared by all collectors, next to the script.
|
||||
|
||||
## What it sends
|
||||
|
||||
| Dashboard | Windows Server Backup |
|
||||
|---|---|
|
||||
| Company | `Tenant` from the config |
|
||||
| Job | One per server, ID `wsb:<computer name>`, so changing `JobName` renames rather than duplicates it |
|
||||
| Run | One per backup job from `Get-WBJob`, identified by its start time (WSB jobs have no ID). Restores are ignored. `details.items` lists what was backed up: the VMs for a Hyper-V backup, otherwise volumes |
|
||||
| Status | Error HRESULT → **Failed**. Success, but a detailed error or a failed item (e.g. one volume) → **Warning**. Cancelled or unrecognised → **Warning**. In progress → **Running** |
|
||||
| Started / finished | Job start / end time. WSB gives these as local-time text to the minute, in the server's date format; the collector parses them with that format |
|
||||
| Next scheduled | `Get-WBSummary`'s `NextBackupTime`. Empty when there's no scheduled backup |
|
||||
| Message | Error description, error codes (as `0x8XXXXXXX`), failed VMs/volumes, and the failure log path |
|
||||
| Size | Bytes processed across the backup's items |
|
||||
|
||||
Each run covers the last 48 hours (`LookbackHours`). Runs already delivered are recorded in
|
||||
`wsb-collector.state.json`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Log: `C:\ProgramData\BackupDashboard\wsb-collector.log`. Exit codes: `0` OK, `1` some runs
|
||||
failed to send (retried next run), `2` setup failed (config or WindowsServerBackup module).
|
||||
- Wrong status, times or messages: run `.\Get-WsbCollectorDiagnostics.ps1` (read-only). It prints
|
||||
the real structure of what the cmdlets return, plus recent `Microsoft-Windows-Backup` events to
|
||||
compare times against.
|
||||
|
||||
## Testing without Windows Server Backup
|
||||
|
||||
```powershell
|
||||
. ./tests/MockWsb.ps1 # or: . ./tests/MockWsb.ps1 -WithRunningJob
|
||||
./Send-WsbBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||
```
|
||||
@@ -0,0 +1,302 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Reports Windows Server Backup (WSB) results to the Backup Dashboard.
|
||||
|
||||
.DESCRIPTION
|
||||
Runs on a server with the Windows Server Backup feature, normally as a SYSTEM scheduled task
|
||||
every 60 minutes (see Install-WsbCollector.ps1). The server appears on the dashboard as one job.
|
||||
Each run sends:
|
||||
- the next scheduled backup (Get-WBSummary's NextBackupTime), which drives the dashboard's
|
||||
Overdue flag, and
|
||||
- every backup from the last LookbackHours that the dashboard hasn't already received in its
|
||||
final state. A backup in progress is sent as Running until it finishes.
|
||||
|
||||
WSB jobs have no ID, so a run is identified by its start time. Already-reported runs are
|
||||
tracked in a state file, and the ingest API is idempotent, so a failed or repeated run is
|
||||
harmless: anything not confirmed is retried next time.
|
||||
|
||||
Written for Windows PowerShell 5.1, which the WindowsServerBackup module targets.
|
||||
|
||||
.PARAMETER ConfigPath
|
||||
JSON config with ApiUrl, ApiKey and Tenant (optional: JobName, SourceSystem, LookbackHours).
|
||||
|
||||
.PARAMETER LookbackHours
|
||||
Overrides the config value. Use 720 once to backfill 30 days of history.
|
||||
|
||||
.PARAMETER DryRun
|
||||
Print the payloads instead of sending them. Doesn't touch the state or log files.
|
||||
|
||||
.EXAMPLE
|
||||
.\Send-WsbBackupStatus.ps1 -DryRun
|
||||
.EXAMPLE
|
||||
.\Send-WsbBackupStatus.ps1 -LookbackHours 720
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$ConfigPath = (Join-Path $env:ProgramData 'BackupDashboard\wsb-collector.json'),
|
||||
[int]$LookbackHours = 0,
|
||||
[switch]$DryRun
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# Shared collector code: installed next to this script; in the repo, in ..\common
|
||||
$sharedModule = @(
|
||||
(Join-Path $PSScriptRoot 'BackupDashboard.Collector.psm1'),
|
||||
(Join-Path (Join-Path (Join-Path $PSScriptRoot '..') 'common') 'BackupDashboard.Collector.psm1')
|
||||
) | Where-Object { Test-Path $_ } | Select-Object -First 1
|
||||
if (-not $sharedModule) {
|
||||
Write-Host "BackupDashboard.Collector.psm1 not found next to $PSCommandPath; re-run the installer."
|
||||
exit 2
|
||||
}
|
||||
Import-Module $sharedModule -Force
|
||||
|
||||
# Get-WBJob -Previous accepts at most 100. That covers a 30-day backfill at up to 3 backups a day;
|
||||
# servers backing up more often only get their latest 100 runs backfilled.
|
||||
$MaxPreviousJobs = 100
|
||||
|
||||
# ---------- Helpers ----------
|
||||
|
||||
function Import-Wsb {
|
||||
if (Get-Command Get-WBJob -ErrorAction SilentlyContinue) { return } # already loaded
|
||||
# Present only when the Windows Server Backup feature (with its command-line tools) is installed
|
||||
Import-Module WindowsServerBackup
|
||||
}
|
||||
|
||||
# ---------- Jobs ----------
|
||||
|
||||
# WBJob times are strings in local time, to the minute, formatted with the current culture
|
||||
# ("10/6/2026 9:00 PM" on en-US). Parse with that same culture: PowerShell's [datetime] cast would
|
||||
# use the invariant (US) format and misread dd/MM dates. Empty while not set (e.g. EndTime of a
|
||||
# running backup).
|
||||
function Get-JobTime($Job, [string]$Field) {
|
||||
$value = Get-FirstValue $Job @($Field)
|
||||
if ($null -eq $value) { return $null }
|
||||
if ($value -is [datetime]) {
|
||||
$dt = $value
|
||||
} else {
|
||||
$dt = [datetime]::MinValue
|
||||
if (-not [datetime]::TryParse([string]$value, [Globalization.CultureInfo]::CurrentCulture,
|
||||
[Globalization.DateTimeStyles]::AssumeLocal, [ref]$dt)) {
|
||||
Write-Log "Couldn't parse $Field '$value'" 'WARN'
|
||||
return $null
|
||||
}
|
||||
}
|
||||
if ($dt.Year -le 1900) { return $null }
|
||||
return $dt
|
||||
}
|
||||
|
||||
# HRESULTs as numbers; 0 = OK. $null when the object doesn't have the property. WSB returns them
|
||||
# as strings ("0"); PowerShell's conversion also accepts hex ("0x80042302") and negative decimals.
|
||||
function Get-HResult($Object, [string]$Field) {
|
||||
$value = Get-FirstValue $Object @($Field)
|
||||
if ($null -eq $value) { return $null }
|
||||
try { return [long]$value } catch { return $null }
|
||||
}
|
||||
|
||||
function Format-HResult([long]$Value) {
|
||||
if ($Value -lt 0) { $Value += 4294967296 } # show as the familiar unsigned 0x8XXXXXXX
|
||||
return '0x{0:X8}' -f $Value
|
||||
}
|
||||
|
||||
function Get-JobItems($Job) {
|
||||
return @(Get-FirstValue $Job @('JobItems', 'Items') | Where-Object { $_ })
|
||||
}
|
||||
|
||||
function Get-ItemName($Item) {
|
||||
$name = Get-FirstValue $Item @('Name', 'VolumeLabel', 'MountPath', 'Path', 'Type')
|
||||
if ($name) { return [string]$name }
|
||||
return 'Item'
|
||||
}
|
||||
|
||||
# The things actually backed up. Items can nest: a Hyper-V backup has one VirtualMachineList item
|
||||
# whose SubItemList holds each VM (and "Host Component").
|
||||
function Get-LeafItems($Items) {
|
||||
foreach ($item in $Items) {
|
||||
$subItems = @(Get-FirstValue $item @('SubItemList') | Where-Object { $_ })
|
||||
if ($subItems.Count) { Get-LeafItems $subItems } else { $item }
|
||||
}
|
||||
}
|
||||
|
||||
function Test-ItemFailed($Item) {
|
||||
$hr = Get-HResult $Item 'HResult'
|
||||
$state = [string](Get-FirstValue $Item @('State', 'JobState'))
|
||||
return ($hr -and $hr -ne 0) -or $state -match 'Fail'
|
||||
}
|
||||
|
||||
# Items that reported an error: failed leaves, or a parent that counts failed sub-items without
|
||||
# saying which
|
||||
function Get-FailedItems($Job) {
|
||||
$items = Get-JobItems $Job
|
||||
$failed = @(Get-LeafItems $items | Where-Object { Test-ItemFailed $_ })
|
||||
if ($failed.Count) { return $failed }
|
||||
return @($items | Where-Object {
|
||||
(Test-ItemFailed $_) -or ((Get-FirstValue $_ @('SubItemFailed')) -and [long](Get-FirstValue $_ @('SubItemFailed')) -gt 0)
|
||||
})
|
||||
}
|
||||
|
||||
# Bytes processed, from the top-level items (sub-items are already counted in their parent)
|
||||
function Get-BytesProcessed($Job) {
|
||||
$sum = [long]0
|
||||
foreach ($item in Get-JobItems $Job) {
|
||||
$value = Get-FirstValue $item @('BytesProcessed')
|
||||
if ($value) { $sum += [long]$value }
|
||||
}
|
||||
return $sum
|
||||
}
|
||||
|
||||
function Get-RunStatus($Job) {
|
||||
$state = [string](Get-FirstValue $Job @('JobState', 'State'))
|
||||
if (-not (Get-JobTime $Job 'EndTime') -and $state -notmatch 'Complet|Fail|Cancel|Abort|Stop|Succe') {
|
||||
return 'running'
|
||||
}
|
||||
if ($state -match 'Fail') { return 'failed' }
|
||||
if ($state -match 'Cancel|Abort|Stop') { return 'warning' }
|
||||
$hr = Get-HResult $Job 'HResult'
|
||||
if ($hr -and $hr -ne 0) { return 'failed' }
|
||||
# Overall success but something underneath went wrong (e.g. a volume or files skipped)
|
||||
$detailed = Get-HResult $Job 'DetailedHResult'
|
||||
if (($detailed -and $detailed -ne 0) -or (Get-FailedItems $Job).Count) { return 'warning' }
|
||||
if (-not $state -or $state -match 'Complet|Succe') { return 'success' }
|
||||
return 'warning' # finished in a state we don't recognise: surface it rather than hide it
|
||||
}
|
||||
|
||||
function Get-RunMessage($Job, [string]$Status) {
|
||||
if ($Status -eq 'running') { return '' }
|
||||
$lines = New-Object 'System.Collections.Generic.List[string]'
|
||||
$state = [string](Get-FirstValue $Job @('JobState', 'State'))
|
||||
$description = Get-FirstValue $Job @('ErrorDescription', 'DetailedMessage')
|
||||
if ($description) { $lines.Add(([string]$description).Trim()) }
|
||||
|
||||
$hr = Get-HResult $Job 'HResult'
|
||||
$detailed = Get-HResult $Job 'DetailedHResult'
|
||||
if (($hr -and $hr -ne 0) -or ($detailed -and $detailed -ne 0)) {
|
||||
$codes = @()
|
||||
if ($hr -and $hr -ne 0) { $codes += "error $(Format-HResult $hr)" }
|
||||
if ($detailed -and $detailed -ne 0) { $codes += "detailed error $(Format-HResult $detailed)" }
|
||||
$lines.Add('Windows Server Backup reported ' + ($codes -join ', ') + '.')
|
||||
}
|
||||
|
||||
foreach ($item in Get-FailedItems $Job) {
|
||||
$itemError = Get-FirstValue $item @('ErrorDescription', 'DetailedMessage')
|
||||
$itemHr = Get-HResult $item 'HResult'
|
||||
$subFailed = Get-FirstValue $item @('SubItemFailed')
|
||||
$line = "$(Get-ItemName $item) - " # names are often volumes like "D:", so no colon
|
||||
if ($itemError) { $line += ([string]$itemError).Trim() }
|
||||
elseif ($itemHr) { $line += "error $(Format-HResult $itemHr)" }
|
||||
elseif ($subFailed -and [long]$subFailed -gt 0) { $line += "$subFailed sub-item(s) failed" }
|
||||
else { $line += [string](Get-FirstValue $item @('State', 'JobState')) }
|
||||
$lines.Add($line)
|
||||
}
|
||||
|
||||
# WSB fills in FailureLogPath even for successful backups, so only point at it when relevant
|
||||
if ($Status -ne 'success') {
|
||||
$failureLog = Get-FirstValue $Job @('FailureLogPath')
|
||||
if ($failureLog) { $lines.Add("Failure log: $failureLog") }
|
||||
}
|
||||
|
||||
if ($lines.Count -eq 0) {
|
||||
if ($Status -eq 'success') { return 'Backup completed successfully.' }
|
||||
return "Backup finished with state '$state'."
|
||||
}
|
||||
# WSB reports "Completed" even for a backup that failed with an error code, so only mention
|
||||
# the state when it says something else (e.g. cancelled)
|
||||
if ($state -and $Status -ne 'success' -and $state -notmatch 'Complet') { $lines.Insert(0, "Job state: $state") }
|
||||
return ($lines -join "`n")
|
||||
}
|
||||
|
||||
function ConvertTo-Run($Job) {
|
||||
$status = Get-RunStatus $Job
|
||||
$details = [ordered]@{ server = $env:COMPUTERNAME }
|
||||
$items = @(Get-LeafItems (Get-JobItems $Job) | ForEach-Object { Get-ItemName $_ })
|
||||
if ($items.Count) { $details.items = $items }
|
||||
$state = Get-FirstValue $Job @('JobState', 'State')
|
||||
if ($state) { $details.wsb_state = [string]$state }
|
||||
$version = Get-FirstValue $Job @('VersionId')
|
||||
if ($version) { $details.version_id = [string]$version }
|
||||
|
||||
# No external_id: WSB jobs have no ID, so the API keys the run on started_at
|
||||
$run = [ordered]@{
|
||||
status = $status
|
||||
started_at = ConvertTo-IsoTime (Get-JobTime $Job 'StartTime')
|
||||
message = Get-RunMessage $Job $status
|
||||
details = $details
|
||||
}
|
||||
if ($status -ne 'running') {
|
||||
$finished = ConvertTo-IsoTime (Get-JobTime $Job 'EndTime')
|
||||
if ($finished) { $run.finished_at = $finished } # otherwise the API uses receipt time
|
||||
$bytes = Get-BytesProcessed $Job
|
||||
if ($bytes -gt 0) { $run.size_bytes = $bytes }
|
||||
}
|
||||
return $run
|
||||
}
|
||||
|
||||
function Test-IsBackupJob($Job) {
|
||||
# Get-WBJob also lists recoveries; only backups belong on the dashboard
|
||||
$type = [string](Get-FirstValue $Job @('JobType', 'Type'))
|
||||
return (-not $type) -or $type -match 'Backup'
|
||||
}
|
||||
|
||||
function Get-NextRun {
|
||||
try {
|
||||
$summary = Get-WBSummary
|
||||
return ConvertTo-IsoTime (Get-FirstValue $summary @('NextBackupTime'))
|
||||
} catch {
|
||||
Write-Log "Couldn't read the backup summary: $(Get-ErrorText $_)" 'WARN'
|
||||
return $null
|
||||
}
|
||||
}
|
||||
|
||||
# ---------- Main ----------
|
||||
|
||||
try {
|
||||
$Config = Initialize-Collector -Name 'wsb' -ConfigPath $ConfigPath -LookbackHours $LookbackHours -DryRun:$DryRun
|
||||
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Windows Server Backup' }
|
||||
$JobName = if ($Config.JobName) { [string]$Config.JobName } else { "$env:COMPUTERNAME - Windows Server Backup" }
|
||||
$LookbackHours = $Config.LookbackHours
|
||||
|
||||
Import-Wsb
|
||||
|
||||
$since = (Get-Date).AddHours(-$LookbackHours)
|
||||
# Get-WBJob with no arguments returns the job in progress (if any); -Previous lists finished ones
|
||||
$jobs = @(@(Get-WBJob) + @(Get-WBJob -Previous $MaxPreviousJobs) | Where-Object {
|
||||
$_ -and (Test-IsBackupJob $_) -and (Get-JobTime $_ 'StartTime')
|
||||
})
|
||||
$byStart = [ordered]@{}
|
||||
foreach ($job in $jobs) {
|
||||
$start = Get-JobTime $job 'StartTime'
|
||||
if ($start -lt $since) { continue }
|
||||
$byStart[(ConvertTo-IsoTime $start)] = $job # a job in both lists: the finished entry (later) wins
|
||||
}
|
||||
$pending = @($byStart.Keys | Sort-Object | Where-Object { -not (Test-RunSent ($_)) } |
|
||||
ForEach-Object { $byStart[$_] })
|
||||
|
||||
$jobInfo = [ordered]@{
|
||||
# Stable per server, so renaming the dashboard job via JobName doesn't create a new one
|
||||
external_id = "wsb:$($env:COMPUTERNAME.ToLowerInvariant())"
|
||||
name = $JobName
|
||||
# Always sent (null when unscheduled) so the dashboard's value tracks WSB's
|
||||
next_run_at = Get-NextRun
|
||||
}
|
||||
} catch {
|
||||
Write-Log "Collector setup failed: $(Get-ErrorText $_)" 'ERROR'
|
||||
exit 2
|
||||
}
|
||||
|
||||
$failures = 0
|
||||
try {
|
||||
if ($pending.Count -eq 0) {
|
||||
# Keep next_run_at fresh even when nothing new ran
|
||||
Send-JobReport -Tenant $Config.Tenant -SourceSystem $SourceSystem -Job $jobInfo
|
||||
}
|
||||
foreach ($job in $pending) {
|
||||
$run = ConvertTo-Run $job
|
||||
Send-JobReport -Tenant $Config.Tenant -SourceSystem $SourceSystem -Job $jobInfo -Run $run -SentKey $run.started_at -StartedAt ((Get-JobTime $job 'StartTime'))
|
||||
}
|
||||
} catch {
|
||||
$failures++
|
||||
Write-Log "Failed to report: $(Get-ErrorText $_)" 'ERROR'
|
||||
}
|
||||
|
||||
$exitCode = Complete-Collector -Failures $failures
|
||||
if ($exitCode) { exit $exitCode }
|
||||
@@ -0,0 +1,99 @@
|
||||
# Stand-ins for the WindowsServerBackup cmdlets. Object shapes follow Get-WsbCollectorDiagnostics.ps1
|
||||
# output from a real Hyper-V host (Server 2019): job times are culture-formatted local-time strings
|
||||
# to the minute ("10/6/2026 9:00 PM" on en-US), HRESULTs are strings, and a Hyper-V backup has one
|
||||
# VirtualMachineList item whose SubItemList holds the VMs. When nothing is running, Get-WBJob
|
||||
# returns an empty job (JobType None).
|
||||
#
|
||||
# . ./tests/MockWsb.ps1 # nothing running
|
||||
# . ./tests/MockWsb.ps1 -WithRunningJob # a backup in progress
|
||||
# ./Send-WsbBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
|
||||
param([switch]$WithRunningJob)
|
||||
|
||||
# Fixed to the top of the hour so repeated runs see the same jobs
|
||||
$now = Get-Date -Minute 0 -Second 0 -Millisecond 0
|
||||
|
||||
# Same format as the real cmdlet: short date + short time in the current culture
|
||||
function Format-WbTime($Value) {
|
||||
if ($null -eq $Value) { return '' }
|
||||
return ([datetime]$Value).ToString('g', [Globalization.CultureInfo]::CurrentCulture)
|
||||
}
|
||||
|
||||
function New-MockItem($Name, $Type = 'Volume', [long]$Bytes = 0, [string]$HResult = '0', $ErrorDescription = '', $SubItems = $null) {
|
||||
$failedSubs = @($SubItems | Where-Object { $_ -and $_.HResult -ne '0' }).Count
|
||||
[pscustomobject]@{
|
||||
Name = $Name
|
||||
Type = $Type
|
||||
State = $(if ($HResult -ne '0') { 'Failed' } else { 'Completed' })
|
||||
HResult = $HResult
|
||||
DetailedHResult = '0'
|
||||
ErrorDescription = $ErrorDescription
|
||||
BytesProcessed = $Bytes
|
||||
TotalBytes = $Bytes
|
||||
CurrentItem = $null
|
||||
SubItemProcessed = [long]@($SubItems | Where-Object { $_ }).Count
|
||||
SubItemFailed = [long]$failedSubs
|
||||
TotalSubItem = [long]@($SubItems | Where-Object { $_ }).Count
|
||||
SubItemList = $SubItems
|
||||
}
|
||||
}
|
||||
|
||||
function New-VmList([long]$VmBytes, [string]$VmHResult = '0', $VmError = '') {
|
||||
$vms = @(
|
||||
(New-MockItem 'AAOG-EDM-AD1' 'Component' $VmBytes $VmHResult $VmError),
|
||||
(New-MockItem 'Host Component' 'Component' 447729)
|
||||
)
|
||||
New-MockItem 'VirtualMachineList' 'VirtualmachineList' ($VmBytes + 447729) -SubItems $vms
|
||||
}
|
||||
|
||||
function New-MockJob($Type, $Start, $End, $State, [string]$HResult = '0', [string]$DetailedHResult = '0', $ErrorDescription = '', $Items = $null) {
|
||||
[pscustomobject]@{
|
||||
JobType = $Type
|
||||
StartTime = Format-WbTime $Start
|
||||
EndTime = Format-WbTime $End
|
||||
JobState = $State
|
||||
CurrentOperation = ''
|
||||
HResult = $HResult
|
||||
DetailedHResult = $DetailedHResult
|
||||
ErrorDescription = $ErrorDescription
|
||||
JobItems = $Items
|
||||
VersionId = $(if ($End) { $Start.ToUniversalTime().ToString('MM/dd/yyyy-HH:mm', [Globalization.CultureInfo]::InvariantCulture) } else { '' })
|
||||
SuccessLogPath = 'C:\Windows\Logs\WindowsServerBackup\Backup.log'
|
||||
# Populated even for successful backups, as on the real server
|
||||
FailureLogPath = 'C:\Windows\Logs\WindowsServerBackup\Backup_Error.log'
|
||||
}
|
||||
}
|
||||
|
||||
$global:MockWsbRunning = if ($WithRunningJob) {
|
||||
@(New-MockJob 'Backup' $now.AddMinutes(-10) $null 'Running' -Items @(New-VmList 10GB))
|
||||
} else {
|
||||
@(New-MockJob 'None' $null $null 'Unknown')
|
||||
}
|
||||
|
||||
$global:MockWsbPrevious = @(
|
||||
(New-MockJob 'Backup' $now.AddHours(-12) $now.AddHours(-11.6) 'Completed' -Items @(New-VmList 124756706036)),
|
||||
# Completed overall, but one VM failed
|
||||
(New-MockJob 'Backup' $now.AddHours(-36) $now.AddHours(-35.6) 'Completed' '0' '0' '' @(
|
||||
New-VmList 0 '-2147024891' 'Access is denied.')),
|
||||
# Failed outright: backup target not found (0x80042302)
|
||||
(New-MockJob 'Backup' $now.AddHours(-30) $now.AddHours(-29.9) 'Completed' '-2147212542' '0' `
|
||||
'The backup storage location could not be found.'),
|
||||
# A volume-based backup, for servers that aren't Hyper-V hosts
|
||||
(New-MockJob 'Backup' $now.AddHours(-20) $now.AddHours(-19.5) 'Completed' -Items @(
|
||||
(New-MockItem 'C:' 'Volume' 40GB), (New-MockItem 'D:' 'Volume' 200GB))),
|
||||
(New-MockJob 'Recovery' $now.AddHours(-5) $now.AddHours(-4) 'Completed'), # a restore: must be ignored
|
||||
(New-MockJob 'Backup' $now.AddDays(-5) $now.AddDays(-5).AddHours(1) 'Completed') # outside 48h lookback
|
||||
)
|
||||
|
||||
$global:MockWsbSummary = [pscustomobject]@{
|
||||
NextBackupTime = $now.Date.AddHours(21).AddDays($(if ($now.Hour -ge 21) { 1 } else { 0 }))
|
||||
NumberOfVersions = 1
|
||||
LastSuccessfulBackupTime = $now.AddHours(-12)
|
||||
LastBackupResultHR = [uint32]0
|
||||
CurrentOperationStatus = $(if ($WithRunningJob) { 'BackupInProgress' } else { 'NoOperationInProgress' })
|
||||
}
|
||||
|
||||
# Same limit as the real cmdlet
|
||||
function Get-WBJob { param([ValidateRange(1, 100)][int]$Previous) if ($PSBoundParameters.ContainsKey('Previous')) { $global:MockWsbPrevious } else { $global:MockWsbRunning } }
|
||||
function Get-WBSummary { $global:MockWsbSummary }
|
||||
function Get-WBPolicy { [pscustomobject]@{ PolicyName = 'Scheduled' } }
|
||||
function Get-WBSchedule { param($Policy) @($global:MockWsbSummary.NextBackupTime) }
|
||||
@@ -6,8 +6,8 @@ After=network.target
|
||||
User=backupdash
|
||||
Group=backupdash
|
||||
WorkingDirectory=/opt/backup-dashboard/backend
|
||||
Environment=BACKUP_DASH_DB=/var/lib/backup-dashboard/backup-dashboard.db
|
||||
Environment=BACKUP_DASH_RETENTION_DAYS=90
|
||||
# Settings live in this file so manage.py reads exactly the same ones (see deploy/backup-dashboard.env)
|
||||
EnvironmentFile=/etc/backup-dashboard/backup-dashboard.env
|
||||
# Bound to localhost: only nginx can reach it, so nginx's auth can't be bypassed from outside.
|
||||
# One worker is plenty at this scale and keeps SQLite writes simple.
|
||||
ExecStart=/opt/backup-dashboard/venv/bin/uvicorn --factory app.main:create_app --host 127.0.0.1 --port 8000 --proxy-headers
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
# Install as /etc/backup-dashboard/backup-dashboard.env
|
||||
# Read by both the API service and manage.py, so they always use the same database.
|
||||
BACKUP_DASH_DB=/var/lib/backup-dashboard/backup-dashboard.db
|
||||
BACKUP_DASH_RETENTION_DAYS=90
|
||||
+54
-8
@@ -1,7 +1,7 @@
|
||||
server {
|
||||
listen 80;
|
||||
server_name backups.example.com;
|
||||
# Basic auth sends the password with every request — never serve this over plain HTTP
|
||||
# Session cookies and API keys must never cross plain HTTP
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
|
||||
@@ -23,26 +23,72 @@ server {
|
||||
add_header Referrer-Policy "no-referrer" always;
|
||||
add_header Strict-Transport-Security "max-age=31536000" always;
|
||||
|
||||
# Dashboard users. Create with: htpasswd -B -c /etc/nginx/backup-dashboard.htpasswd <user>
|
||||
# To move to Microsoft 365 sign-in later, replace these two lines with an auth_request to
|
||||
# oauth2-proxy (provider=entra-id); the app itself doesn't need to change.
|
||||
auth_basic "Backup Monitor";
|
||||
auth_basic_user_file /etc/nginx/backup-dashboard.htpasswd;
|
||||
# ---------- Microsoft 365 sign-in ----------
|
||||
# oauth2-proxy (deploy/oauth2-proxy.cfg) runs on 127.0.0.1:4180 and does the sign-in with
|
||||
# Entra ID. Every request below asks it "is this browser signed in?" via auth_request.
|
||||
|
||||
# Its own pages: /oauth2/start (go to Microsoft), /oauth2/callback, /oauth2/sign_out
|
||||
location /oauth2/ {
|
||||
proxy_pass http://127.0.0.1:4180;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
# Where to return after signing in: the page that was asked for
|
||||
proxy_set_header X-Auth-Request-Redirect $request_uri;
|
||||
# Entra ID session cookies are large
|
||||
proxy_buffer_size 16k;
|
||||
proxy_buffers 8 16k;
|
||||
}
|
||||
|
||||
# The sign-in check itself, only reachable as an auth_request subrequest
|
||||
location = /oauth2/auth {
|
||||
internal;
|
||||
proxy_pass http://127.0.0.1:4180;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-Uri $request_uri;
|
||||
proxy_set_header Content-Length "";
|
||||
proxy_pass_request_body off;
|
||||
proxy_buffer_size 16k;
|
||||
proxy_buffers 8 16k;
|
||||
}
|
||||
|
||||
# ---------- Dashboard ----------
|
||||
|
||||
location / {
|
||||
auth_request /oauth2/auth;
|
||||
# Not signed in: straight to Microsoft's sign-in, then back to this page
|
||||
error_page 401 = /oauth2/start;
|
||||
# Make browsers check for a newer page/script on every load (a cheap 304 when unchanged),
|
||||
# so an update shows up at once. (expires, not add_header: an add_header here would drop
|
||||
# the security headers above.)
|
||||
expires -1;
|
||||
try_files $uri $uri/ /index.html;
|
||||
}
|
||||
|
||||
location /api/ {
|
||||
auth_request /oauth2/auth;
|
||||
# No redirect here: API calls from the page get a plain 401, and the page asks the user to
|
||||
# reload, which goes through sign-in above.
|
||||
|
||||
# Who's signed in, for the dashboard's "signed in as" (proxy_set_header replaces any value
|
||||
# the browser sent, so these can't be spoofed)
|
||||
auth_request_set $auth_email $upstream_http_x_auth_request_email;
|
||||
auth_request_set $auth_user $upstream_http_x_auth_request_user;
|
||||
auth_request_set $auth_name $upstream_http_x_auth_request_preferred_username;
|
||||
proxy_set_header X-Forwarded-Email $auth_email;
|
||||
proxy_set_header X-Forwarded-User $auth_user;
|
||||
proxy_set_header X-Forwarded-Preferred-Username $auth_name;
|
||||
|
||||
proxy_pass http://127.0.0.1:8000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
|
||||
# Collectors authenticate with their own API key (Authorization: Bearer …), checked by the app
|
||||
# Collectors authenticate with their own API key (Authorization: Bearer …), checked by the app,
|
||||
# so no Microsoft sign-in here
|
||||
location = /api/v1/ingest {
|
||||
auth_basic off;
|
||||
client_max_body_size 256k;
|
||||
proxy_pass http://127.0.0.1:8000;
|
||||
proxy_set_header Host $host;
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
# oauth2-proxy config for Backup Dashboard's Microsoft 365 (Entra ID) sign-in.
|
||||
# Install as /etc/oauth2-proxy/backup-dashboard.cfg, readable only by root and the oauth2-proxy
|
||||
# user (it holds the cookie secret). Needs oauth2-proxy 7.8 or newer for the entra-id provider.
|
||||
# Replace the four <...> values; see the README's "Microsoft 365 sign-in" section.
|
||||
|
||||
## Where it listens: only nginx talks to it
|
||||
http_address = "127.0.0.1:4180"
|
||||
reverse_proxy = true
|
||||
# Only nginx (on this machine) may supply X-Forwarded-* headers
|
||||
trusted_proxy_ips = ["127.0.0.1/32", "::1/128"]
|
||||
|
||||
## Entra ID app registration (single tenant)
|
||||
provider = "entra-id"
|
||||
oidc_issuer_url = "https://login.microsoftonline.com/<TENANT_ID>/v2.0"
|
||||
client_id = "<APPLICATION_CLIENT_ID>"
|
||||
# The client secret's *value*, in a file of its own, root:oauth2-proxy, mode 640
|
||||
client_secret_file = "/etc/oauth2-proxy/client-secret"
|
||||
redirect_url = "https://<DASHBOARD_HOST>/oauth2/callback"
|
||||
|
||||
## Who may sign in
|
||||
# Anyone in the tenant that Entra lets through. Limit it in Entra itself: Enterprise
|
||||
# applications -> this app -> Properties -> "Assignment required" = Yes, then assign users or a
|
||||
# group. Entra then refuses everyone else before they ever reach the dashboard.
|
||||
email_domains = ["*"]
|
||||
# The user's sign-in name (UPN), which every work account has. The "email" claim is only present
|
||||
# when the account has a mailbox address.
|
||||
oidc_email_claim = "preferred_username"
|
||||
|
||||
## Session cookie
|
||||
# Generate with: openssl rand -base64 32 | tr -- '+/' '-_'
|
||||
cookie_secret = "<COOKIE_SECRET>"
|
||||
cookie_name = "_backup_dashboard"
|
||||
cookie_secure = true
|
||||
cookie_samesite = "lax"
|
||||
# Sign in again (usually silently, via the Microsoft session) after this long
|
||||
cookie_expire = "12h"
|
||||
|
||||
## Behaviour with nginx's auth_request
|
||||
# Auth-only: nginx serves the dashboard; oauth2-proxy just answers "signed in or not"
|
||||
upstreams = ["static://202"]
|
||||
# Put the user in X-Auth-Request-* headers on /oauth2/auth responses, for nginx to pass on
|
||||
set_xauthrequest = true
|
||||
# Go straight to Microsoft instead of showing a "Sign in with..." page first
|
||||
skip_provider_button = true
|
||||
# Allow "Sign out" to continue to Microsoft's sign-out page
|
||||
whitelist_domains = ["login.microsoftonline.com"]
|
||||
@@ -0,0 +1,18 @@
|
||||
[Unit]
|
||||
Description=oauth2-proxy (Microsoft 365 sign-in for Backup Dashboard)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
User=oauth2-proxy
|
||||
Group=oauth2-proxy
|
||||
ExecStart=/usr/local/bin/oauth2-proxy --config=/etc/oauth2-proxy/backup-dashboard.cfg
|
||||
Restart=on-failure
|
||||
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -71,6 +71,12 @@ body {
|
||||
}
|
||||
.topbar__title { margin: 0; font-size: 1.25rem; }
|
||||
.topbar__meta { display: flex; align-items: center; gap: 0.75rem; font-size: 0.875rem; }
|
||||
/* Only when shown: display:flex on the bare class would override the hidden attribute */
|
||||
.topbar__user:not([hidden]) {
|
||||
display: flex; align-items: center; gap: 0.75rem;
|
||||
padding-left: 0.75rem; border-left: 1px solid var(--border);
|
||||
}
|
||||
a.btn { text-decoration: none; }
|
||||
|
||||
.container {
|
||||
max-width: 1400px;
|
||||
|
||||
@@ -14,6 +14,11 @@
|
||||
<div class="topbar__meta">
|
||||
<span id="last-updated" class="muted" aria-live="polite">Loading…</span>
|
||||
<button id="refresh-btn" class="btn" type="button">Refresh</button>
|
||||
<!-- Filled in from /api/v1/me when signed in with Microsoft 365 -->
|
||||
<span id="signed-in" class="topbar__user" hidden>
|
||||
<span id="signed-in-email" class="muted"></span>
|
||||
<a id="sign-out" class="btn" href="#">Sign out</a>
|
||||
</span>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
|
||||
@@ -49,6 +49,10 @@ const httpApi = {
|
||||
getRun(id) {
|
||||
return request(`/runs/${encodeURIComponent(id)}`);
|
||||
},
|
||||
/** GET /me → { email, name } of the Microsoft 365 user, both null without sign-in */
|
||||
getMe() {
|
||||
return request('/me');
|
||||
},
|
||||
};
|
||||
|
||||
// Mock is loaded lazily so it never ships in the production request path.
|
||||
|
||||
+23
-4
@@ -56,6 +56,9 @@ const dom = {
|
||||
empty: $('#empty-state'),
|
||||
search: $('#search'),
|
||||
refreshBtn: $('#refresh-btn'),
|
||||
signedIn: $('#signed-in'),
|
||||
signedInEmail: $('#signed-in-email'),
|
||||
signOut: $('#sign-out'),
|
||||
lastUpdated: $('#last-updated'),
|
||||
errorBanner: $('#error-banner'),
|
||||
dialog: $('#detail-dialog'),
|
||||
@@ -95,8 +98,8 @@ function statusBadge(status, { asButton = false, opensDialog = false, dataset =
|
||||
|
||||
const runningTitle = (since) => (since ? `Running since ${formatDateTime(since)}` : null);
|
||||
|
||||
function timeCell(date, extra = null) {
|
||||
if (!date) return el('td', {}, el('span', { class: 'muted' }, 'Never'));
|
||||
function timeCell(date, { extra = null, emptyLabel = 'Never' } = {}) {
|
||||
if (!date) return el('td', {}, el('span', { class: 'muted' }, emptyLabel));
|
||||
return el('td', {},
|
||||
el('time', { class: 'time', datetime: date.toISOString(), title: date.toString() },
|
||||
el('span', {}, formatDateTime(date), extra),
|
||||
@@ -112,7 +115,8 @@ function renderRow(job) {
|
||||
asButton: true, opensDialog: true, dataset: { jobId: job.id }, title: runningTitle(job.runningSince),
|
||||
})),
|
||||
timeCell(job.lastRun),
|
||||
timeCell(job.nextRun, overdueTag));
|
||||
// No next run: unscheduled, or e.g. a Veeam copy job in immediate mode
|
||||
timeCell(job.nextRun, { extra: overdueTag, emptyLabel: 'Not scheduled' }));
|
||||
}
|
||||
|
||||
const collator = new Intl.Collator(undefined, { sensitivity: 'base', numeric: true });
|
||||
@@ -304,7 +308,7 @@ function renderSummarySection() {
|
||||
// While a job is running, its "last run" is the previous one — say how that went
|
||||
status === 'running' && lastStatus ? detailRow('Previous result', STATUS[lastStatus].label) : [],
|
||||
detailRow('Last finished', formatDateTime(detail ? parseDate(detail.last_run_at) : job.lastRun)),
|
||||
detailRow('Next scheduled', formatDateTime(detail ? parseDate(detail.next_run_at) : job.nextRun)),
|
||||
detailRow('Next scheduled', formatDateTime(detail ? parseDate(detail.next_run_at) : job.nextRun, 'Not scheduled')),
|
||||
detailRow('Duration', formatDuration(detail?.duration_seconds)),
|
||||
detailRow('Size', formatBytes(detail?.size_bytes)),
|
||||
detailRow('Source', detail?.source_system)));
|
||||
@@ -523,5 +527,20 @@ dom.search.addEventListener('input', () => {
|
||||
|
||||
dom.refreshBtn.addEventListener('click', () => refresh().then(schedulePoll));
|
||||
|
||||
// ---------- Signed-in user ----------
|
||||
async function showSignedInUser() {
|
||||
try {
|
||||
const me = await api.getMe();
|
||||
const who = me?.email || me?.name;
|
||||
if (!who) return; // no Microsoft 365 sign-in in front of the dashboard (dev, mock)
|
||||
dom.signedInEmail.textContent = who;
|
||||
dom.signOut.href = CONFIG.signOutUrl;
|
||||
dom.signedIn.hidden = false;
|
||||
} catch (err) {
|
||||
console.error(err); // not worth a banner: the jobs request will surface real problems
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- Boot ----------
|
||||
showSignedInUser();
|
||||
refresh().then(schedulePoll);
|
||||
@@ -12,4 +12,8 @@ export const CONFIG = Object.freeze({
|
||||
historyDays: 30,
|
||||
// ?mock=1 → use in-browser sample data, no backend needed
|
||||
useMock: params.has('mock'),
|
||||
// oauth2-proxy clears the dashboard session, then hands over to Microsoft's sign-out page.
|
||||
// Signing out of the dashboard alone would be undone at once: the browser's Microsoft session
|
||||
// would sign it straight back in.
|
||||
signOutUrl: '/oauth2/sign_out?rd=' + encodeURIComponent('https://login.microsoftonline.com/common/oauth2/v2.0/logout'),
|
||||
});
|
||||
+1
-1
@@ -13,7 +13,7 @@ export function parseDate(value) {
|
||||
return Number.isNaN(d.getTime()) ? null : d;
|
||||
}
|
||||
|
||||
export const formatDateTime = (d) => (d ? dateTimeFmt.format(d) : '—');
|
||||
export const formatDateTime = (d, emptyLabel = '—') => (d ? dateTimeFmt.format(d) : emptyLabel);
|
||||
export const formatTime = (d) => timeFmt.format(d);
|
||||
|
||||
export function formatRelative(d, now = Date.now()) {
|
||||
|
||||
@@ -126,4 +126,7 @@ export const mockApi = {
|
||||
}
|
||||
throw new Error('Run not found');
|
||||
},
|
||||
async getMe() {
|
||||
return { email: null, name: null }; // no sign-in in mock mode
|
||||
},
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user