Author SHA1 Message Date
kyleandClaude Opus 5.5 118a7f978d Fix VB365 collector against a real server's object model
Checked with Get-Vb365CollectorDiagnostics.ps1 output from a VB365 8.4
server (GDC-VEEAM-04, 11 organizations, Windows PowerShell 5.1). Times
(session CreationTime/EndTime, job NextRun) are local, and property
names matched. Two fixes:

- Session log items carry their status as a Title prefix ("[Warning]
  ...") with no separate status property. Match on the prefix and strip
  it, instead of matching words in the text, which would have pulled in
  lines like "[Success] Processing site: .../error-reports"
- Retries are separate sessions (MainSessionId -> the retried session,
  which has JobWillBeRetried = True). Mark them in details (retry_of,
  will_retry) and say "Veeam will retry this job." on a pending retry

Mock rebuilt on the real shapes, including a warning + retry pair.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-08 10:47:40 -06:00
kyleandClaude Opus 5.5 3f407ba76e Add Veeam Backup for Microsoft 365 collector
PowerShell script for the VB365 server, run hourly as a SYSTEM scheduled
task (Windows PowerShell 5.1 by default, -PwshPath for PowerShell 7).
One dashboard job per VB365 backup job and backup copy job.

- Sessions from Get-VBOJobSession (and Get-VBOCopyJobSession where
  available) within the lookback window; status from session Status,
  message from the session log's warning/error lines, size from
  Statistics.TransferredData ("1.5 GB" -> bytes)
- One VB365 server can protect several Microsoft 365 organizations, so
  the dashboard company is chosen per job: OrganizationTenants mapping,
  else Tenant, else the organization's name
- Next run from the job's NextRun; empty for disabled jobs
- Get-Vb365CollectorDiagnostics.ps1 dumps jobs and sessions, showing
  each DateTime's Kind (local vs UTC); installer adapted from MARS

Not yet run against a real server.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-08 10:31:43 -06:00
kyleandClaude Opus 5.5 dd3e2fe885 Read Veeam next run from the live schedule
On VBR 13 a job's ScheduleOptions property is a stale snapshot: NextRun
is empty (and its LatestRunLocal years old), so every Veeam job showed
no next run and could never be flagged Overdue. GetScheduleOptions() /
Get-VBRJobScheduleOptions return the live schedule ("2026-10-09
12:15:00 AM"). Use GetScheduleOptions(), falling back to the property.

Immediate-mode backup copy jobs (IsContinuous) have a live NextRun too,
but it's a stale date months in the past, which would show as Overdue;
keep reporting no next run for them.

Parsing tries the current culture, then invariant; VBR 13's format
parsed correctly under en-CA, en-US, fr-CA and en-GB. Mock updated to
VBR 13's schedule shape.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-08 09:53:55 -06:00
kyleandClaude Opus 5.5 51b97c4556 Diagnose Veeam schedules; show "Not scheduled" for no next run
Veeam jobs on a VBR 13 server show no next run on the dashboard. The
diagnostics script gains a section dumping each job's schedule
properties (ScheduleOptions, GetScheduleOptions(),
Get-VBRJobScheduleOptions) and the current culture, to find where VBR 13
keeps the next run.

The dashboard showed "Never" for a missing next run, which reads as
"this job will never run"; it now says "Not scheduled" (e.g. immediate
mode copy jobs).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-08 09:42:01 -06:00
kyleandClaude Opus 5.5 555e928e01 Cap WSB job history request at 100
Get-WBJob -Previous only accepts 1-100, so asking for 200 failed
collector setup on every run ("The 200 argument is greater than the
maximum allowed range of 100"). 100 still covers a 30-day backfill at
up to 3 backups a day. The mock now enforces the same range.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-08 09:33:07 -06:00
kyleandClaude Opus 5.5 b625a8af65 Fix WSB collector against a real server's object model
Checked with Get-WsbCollectorDiagnostics.ps1 output from a Server 2019
Hyper-V host (AAOG-EDM-HVR1):

- WBJob StartTime/EndTime are culture-formatted local-time strings
  ("10/6/2026 9:00 PM"). Parse them with the current culture: the
  [datetime] cast uses the invariant (US) format and would misread
  dd/MM dates, e.g. on en-CA/en-GB servers. Confirmed local, not UTC,
  against the Microsoft-Windows-Backup event log
- Hyper-V backups nest VMs under a VirtualMachineList item's
  SubItemList; failures, names and the items list now come from the
  leaf items, falling back to a parent's SubItemFailed count
- Report size from the top-level items' BytesProcessed
- FailureLogPath is set even on success: running backups get no
  message, and the log is only cited for warnings/failures
- HRESULTs are strings; conversion tolerates hex and bad values
- Mock rebuilt on the real shapes; tested under en-US and en-GB

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-07 16:14:03 -06:00
kyleandClaude Opus 5.5 a14b1b0dc9 Add Windows Server Backup collector
PowerShell script for servers using Windows Server Backup, run hourly as
a SYSTEM scheduled task under Windows PowerShell 5.1. Each server is one
dashboard job.

- Backup jobs from Get-WBJob (in progress and -Previous) within the
  lookback window; restores are ignored. WSB jobs have no ID, so runs
  are keyed on start time, which also merges a Running report with the
  finished one
- Error HRESULT -> failed; detailed error or a failed item -> warning;
  messages include the error description, codes as 0x8XXXXXXX, failed
  items and the failure log path
- Next run from Get-WBSummary's NextBackupTime
- Get-WsbCollectorDiagnostics.ps1 dumps the cmdlets' real objects and
  recent Microsoft-Windows-Backup events (to check whether job times
  are local or UTC); installer adapted from the MARS one

Not yet run against a real server.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-07 15:56:04 -06:00
kyleandClaude Opus 5.5 5982fc9434 Fix MARS collector against a real agent's object model
Checked with Get-MarsCollectorDiagnostics.ps1 output from MARS agent
2.0.9274 (VA-FS-01):

- JobStatus.StartTime/EndTime are UTC values not marked as UTC
  (DisplayTime is the local one); they were read as local, putting
  runs 6 hours off. Lookback filtering now compares in UTC too
- ErrorInfo is always present with ErrorCode 0 on success; every
  completed backup would have been a warning. Only non-zero error
  codes count
- Schedule times are TimeSpans; casting them to DateTime failed, so
  next_run_at was always empty
- DatasourceStatus.Datasource is an object; use its DataSourceName
- Also report FileProgress.Failed file counts and FailedFileLog
- Mock rebuilt on the real shapes; diagnostics handle no System State
  policy

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-07 15:40:08 -06:00
kyleandClaude Opus 5.5 10181e014b Add Azure Backup (MARS agent) collector
PowerShell script for servers running the MARS agent, run as a SYSTEM
scheduled task (hourly) under Windows PowerShell 5.1, which the
agent's MSOnlineBackup module targets. Each server is one dashboard job.

- Backup jobs from Get-OBJob (in progress and -Previous) within the
  lookback window; restores are ignored. Completed with failed bytes
  or an error reports as warning
- Next run computed from the backup and System State policy schedules;
  left empty for every-N-weeks schedules to avoid false Overdue alarms
- Property reads tolerate unknown names, since the agent's object
  model isn't documented; Get-MarsCollectorDiagnostics.ps1 dumps the
  real structure for checking
- Install-MarsCollector.ps1 mirrors the Veeam installer; tests/MockMars.ps1
  provides stand-in cmdlets

Not yet run against a real MARS agent.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-07 15:23:46 -06:00
kyleandClaude Opus 5.5 b4b703ceee Repair files broken by the web-UI merge
The merge of add-backend-api into main committed unresolved conflicts
and corrupted uploads; the API could not import on main.

- backend/app/settings.py, deploy/backup-dashboard-api.service,
  collectors/veeam/README.md: remove conflict markers, restoring the
  add-backend-api versions
- backend/app/__init__.py, backend/tests/__init__.py: were 1024 null
  bytes ("source code string cannot contain null bytes"); now empty
- Remove the accidental Backup-Dashboard gitlink at the repo root and a
  duplicate collectors/veeam/MockVeeam.ps1 (identical to tests/)
- Restore the 60-minute collector interval from 17ee303, which the
  merge reverted to 15, and update the docs to match

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-07 15:13:49 -06:00
kyle 10bfc7a3aa Update collectors/veeam/README.md 2026-10-07 21:02:46 +00:00
kyle 960839ed76 Update collectors/veeam/README.md 2026-10-07 21:01:23 +00:00
kyle f0b4d84c0b Merging 2026-10-07 15:01:02 -06:00
kyle 067b714678 Upload files to "/" 2026-10-07 20:10:11 +00:00
kyle 3355a7d4bc Update collectors/veeam/Send-VeeamBackupStatus.ps1 2026-10-07 20:09:47 +00:00
kyle e916afaae1 Update collectors/veeam/MockVeeam.ps1 2026-10-07 20:09:36 +00:00
kyle 1f96d11a27 Update collectors/veeam/Install-VeeamCollector.ps1 2026-10-07 20:09:26 +00:00
kyle 41c4fa2700 Delete collectors/veeam/Send-VeeamBackupStatus.ps1 2026-10-07 20:09:13 +00:00
kyle a84a35463b Delete collectors/veeam/MockVeeam.ps1 2026-10-07 20:09:09 +00:00
kyle 96b729bec8 Delete collectors/veeam/Install-VeeamCollector.ps1 2026-10-07 20:09:04 +00:00
kyle ae574fa2b6 Delete README.md 2026-10-07 20:08:58 +00:00
kyle e50711b93b Delete Get-VeeamCollectorDiagnostics.ps1 2026-10-07 20:07:30 +00:00
kyle 8fd17e3817 Upload files to "/" 2026-10-07 20:05:28 +00:00
kyle 986cf3be88 Upload files to "collectors/veeam" 2026-10-07 19:42:01 +00:00
kyle 17ee303670 Update scheduled task time to one hour from 15 minutes 2026-10-07 15:57:49 +00:00
kyle effafc5fa0 Delete directory 'test' 2026-10-07 15:56:47 +00:00
kyle 2318579baa Update test/testfolder 2026-10-07 15:56:26 +00:00
kyle 1f02c1c84b Add testfolder 2026-10-07 15:55:31 +00:00
kyle 3b1aee7016 Update scheduled task to use Powershell 7
Todo: add logic to check for and install Powershell 7
2026-10-07 15:48:27 +00:00
kyle d3ab05416d Update manage.py 2026-10-07 01:36:10 +00:00
kyle b8452e1fbb Upload files to "/" 2026-10-06 00:55:03 +00:00
kyle d12e00f69e Update collectors/veeam/README.md 2026-10-06 00:54:51 +00:00
kyle 33c1613ef9 Update collectors/vveam/README.md 2026-10-06 00:54:38 +00:00
kyle 8e29b9cbc4 Update collectors/veeam/Install-VeeamCollector.ps1 2026-10-06 00:54:26 +00:00
kyle 49bf9ef094 Update collectors/vveam/Install-VeeamCollector.ps1 2026-10-06 00:54:13 +00:00
kyle 4d183da7bd Update collectors/veeam/Send-VeeamBackupStatus.ps1 2026-10-06 00:53:47 +00:00
kyle 25f7717ace Update collectors/veeam/MockVeeam.ps1 2026-10-06 00:53:34 +00:00
kyle be23e03c5f Upload files to "/" 2026-10-06 00:53:21 +00:00
kyle 65c587da0d Delete format.js 2026-10-05 18:05:01 +00:00
kyle 29f961f80e Delete config.js 2026-10-05 18:04:49 +00:00
kyle 02789b1910 Update deploy/nginx.conf 2026-10-05 18:03:49 +00:00
kyle 54a6c0b3d6 Update deploy/backup-dashboard-api.service 2026-10-05 18:02:38 +00:00
kyle f11345bb0f Upload files to "backend" 2026-10-05 04:03:26 +00:00
kyle 7217b3e9a8 Upload files to "backend/app" 2026-10-05 04:03:05 +00:00
kyle 47ec7dc9a7 Upload files to "backend/app" 2026-10-05 04:02:57 +00:00
kyle 15ea9e0fec Update backend/app/__init__.py 2026-10-05 04:02:44 +00:00
kyle fe4bd6827c Upload files to "backend" 2026-10-05 04:02:31 +00:00
kyle 4e57eb1a46 Update backend/tests/test_api.py 2026-10-05 04:02:13 +00:00
kyle 670994b937 Update backend/tests/__init__.py 2026-10-05 04:02:03 +00:00
kyle a7c3169606 Upload files to "backend" 2026-10-05 04:01:49 +00:00
kyle 938a0a2af0 Upload files to "backend" 2026-10-05 04:01:38 +00:00
kyle f7e029db8a Upload files to "public/js" 2026-10-05 04:01:12 +00:00
kyle 2df1a75fd5 Update public/css/styles.css 2026-10-05 04:00:56 +00:00
kyle 4d8ee209a5 Update public/index.html 2026-10-05 04:00:38 +00:00
kyle 6c486fbaf3 Update public/js/api.js 2026-10-05 04:00:29 +00:00
kyle fdc7deff16 Upload files to "/" 2026-10-05 04:00:12 +00:00
kyle 0391807027 Upload files to "/" 2026-10-05 03:59:41 +00:00
kyle b0c560377b Update backend/requirements.txt 2026-10-05 03:58:42 +00:00
kyle 17346cc058 Update backend/requirements-dev.txt 2026-10-05 03:58:28 +00:00
kyle 65eb3c88f3 Update backend/manage.py 2026-10-05 03:58:08 +00:00
kyle 996fe33a40 initial upload 2026-10-05 03:57:48 +00:00
24 changed files with 2356 additions and 28 deletions
+1 -1
View File
@@ -12,7 +12,7 @@ browser ──nginx (basic auth, TLS)──▶ /api/* ─▶│ SQLite │
- `public/`: static frontend (no build step). Add `?mock=1` to the URL to use built-in sample data. - `public/`: static frontend (no build step). Add `?mock=1` to the URL to use built-in sample data.
- `backend/`: FastAPI API backed by a single SQLite file. - `backend/`: FastAPI API backed by a single SQLite file.
- `deploy/`: nginx site and systemd unit. - `deploy/`: nginx site and systemd unit.
- `collectors/`: scripts that report results from backup products ([Veeam B&R](collectors/veeam/README.md)). - `collectors/`: scripts that report results from backup products ([Veeam B&R](collectors/veeam/README.md), [Azure Backup / MARS](collectors/mars/README.md), [Windows Server Backup](collectors/wsb/README.md), [Veeam Backup for M365](collectors/vb365/README.md)).
## Data model ## Data model
+8
View File
@@ -0,0 +1,8 @@
__pycache__/
*.pyc
.venv/
venv/
.pytest_cache/
*.db
*.db-wal
*.db-shm
@@ -0,0 +1,102 @@
<#
.SYNOPSIS
Read-only dump of what the MARS agent's cmdlets return, for troubleshooting the collector.
.DESCRIPTION
The collector reads properties of Get-OBJob / Get-OBPolicy / Get-OBSchedule results whose names
aren't documented. This prints those objects' real structure (property names, types and values,
a few levels deep) so mismatches can be fixed. Nothing is changed. Output is also saved to a
text file.
.EXAMPLE
.\Get-MarsCollectorDiagnostics.ps1
#>
[CmdletBinding()]
param(
[int]$Jobs = 3,
[string]$OutFile = (Join-Path $env:TEMP 'mars-collector-diagnostics.txt')
)
$ErrorActionPreference = 'Stop'
$MaxDepth = 6
if (-not (Get-Command Get-OBJob -ErrorAction SilentlyContinue)) {
if (Get-Module -ListAvailable -Name MSOnlineBackup) {
Import-Module MSOnlineBackup
} else {
Import-Module (Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\Modules\MSOnlineBackup')
}
}
function Test-Scalar($Value) {
return $null -eq $Value -or $Value -is [string] -or $Value -is [ValueType] -or $Value -is [guid]
}
# Recursive "name (type) = value" listing. Collections show their first 3 items.
function Show-Object($Value, [string]$Indent = ' ', [int]$Depth = 0) {
if (Test-Scalar $Value) { return }
if ($Depth -ge $MaxDepth) { "$Indent..."; return }
if ($Value -is [Collections.IEnumerable]) {
$items = @($Value)
"$Indent[$($items.Count) item(s)]"
$i = 0
foreach ($item in $items | Select-Object -First 3) {
"$Indent[$i] ($($item.GetType().FullName))$(if (Test-Scalar $item) { " = $item" })"
Show-Object $item "$Indent " ($Depth + 1)
$i++
}
return
}
foreach ($p in $Value.PSObject.Properties) {
try { $v = $p.Value } catch { "$Indent$($p.Name) = <error: $($_.Exception.Message)>"; continue }
$type = if ($null -eq $v) { 'null' } else { $v.GetType().FullName }
if (Test-Scalar $v) {
"$Indent$($p.Name) ($type) = $v"
} else {
$text = [string]$v
if ($text.Length -gt 100) { $text = $text.Substring(0, 97) + '...' }
"$Indent$($p.Name) ($type) = $text"
Show-Object $v "$Indent " ($Depth + 1)
}
}
}
function Show-Section([string]$Title, [scriptblock]$Get) {
"== $Title"
try {
$result = @(& $Get)
if ($result.Count -eq 0 -or ($result.Count -eq 1 -and $null -eq $result[0])) { ' (nothing returned)' }
$n = 0
foreach ($item in $result) {
if ($null -eq $item) { continue }
" [$n] ($($item.GetType().FullName))"
Show-Object $item ' '
$n++
}
} catch {
" error: $($_.Exception.Message)"
}
''
}
$report = & {
$agent = try {
(Get-Item (Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\cbengine.exe')).VersionInfo.ProductVersion
} catch { 'unknown' }
"Generated $(Get-Date -Format o) on $env:COMPUTERNAME, MARS agent $agent, PowerShell $($PSVersionTable.PSVersion)"
''
Show-Section 'Get-OBJob (in progress)' { Get-OBJob }
Show-Section "Get-OBJob -Previous $Jobs" { Get-OBJob -Previous $Jobs }
Show-Section 'Get-OBPolicy' { Get-OBPolicy }
Show-Section 'Get-OBPolicy | Get-OBSchedule' { Get-OBSchedule -Policy (Get-OBPolicy) }
if (Get-Command Get-OBSystemStatePolicy -ErrorAction SilentlyContinue) {
Show-Section 'Get-OBSystemStatePolicy | Get-OBSchedule' {
$ssPolicy = Get-OBSystemStatePolicy
if ($ssPolicy) { Get-OBSchedule -Policy $ssPolicy } else { '(no System State policy configured)' }
}
}
} | Out-String -Width 300
$report
$report | Set-Content -Path $OutFile -Encoding UTF8
Write-Host "Saved to $OutFile"
+90
View File
@@ -0,0 +1,90 @@
<#
.SYNOPSIS
Installs the MARS (Azure Backup agent) collector as a SYSTEM scheduled task.
.DESCRIPTION
- Copies Send-MarsBackupStatus.ps1 to Program Files (only admins can modify it, which
matters because it runs as SYSTEM).
- Writes the config to ProgramData\BackupDashboard and restricts that folder to SYSTEM and
Administrators, since it contains the API key.
- Registers a scheduled task that runs every IntervalMinutes under Windows PowerShell 5.1
(powershell.exe), which the agent's MSOnlineBackup module targets.
Safe to re-run to change settings or update the script. Can sit alongside the Veeam collector.
.EXAMPLE
.\Install-MarsCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
(prompts for the API key)
#>
#Requires -RunAsAdministrator
[CmdletBinding()]
param(
[Parameter(Mandatory)][string]$ApiUrl,
[Parameter(Mandatory)][string]$Tenant,
# Name shown on the dashboard; defaults to "<server> - Azure Backup"
[string]$JobName,
[string]$SourceSystem = 'Azure Backup (MARS)',
[ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
[ValidateRange(1, 720)][int]$LookbackHours = 48,
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard')
)
$ErrorActionPreference = 'Stop'
$TaskName = 'Backup Dashboard - MARS collector'
$DataDir = Join-Path $env:ProgramData 'BackupDashboard'
$ConfigPath = Join-Path $DataDir 'mars-collector.json'
$WindowsPowerShell = Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\powershell.exe'
# Check the agent is here before changing anything
$agentModule = Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\Modules\MSOnlineBackup'
if (-not (Get-Module -ListAvailable -Name MSOnlineBackup) -and -not (Test-Path $agentModule)) {
throw 'The MARS agent (MSOnlineBackup PowerShell module) is not installed on this server.'
}
if ($ApiUrl -notmatch '^https://') {
Write-Warning 'ApiUrl is not HTTPS: the API key would be sent in clear text.'
}
# Read the key as a SecureString so it isn't echoed to the console or kept in shell history
$secure = Read-Host -AsSecureString 'API key (from: python manage.py create-key <name>)'
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
try { $apiKey = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) }
finally { [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr) }
if (-not $apiKey.StartsWith('bdk_')) { throw "That doesn't look like a dashboard API key (expected 'bdk_...')." }
# Config folder: SYSTEM + Administrators only (SIDs so this works on non-English Windows)
New-Item -ItemType Directory -Path $DataDir -Force | Out-Null
& icacls.exe $DataDir /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' | Out-Null
if ($LASTEXITCODE) { throw "icacls failed on $DataDir" }
$config = [ordered]@{
ApiUrl = $ApiUrl.TrimEnd('/')
ApiKey = $apiKey
Tenant = $Tenant
SourceSystem = $SourceSystem
LookbackHours = $LookbackHours
}
if ($JobName) { $config.JobName = $JobName }
$config | ConvertTo-Json | Set-Content -Path $ConfigPath -Encoding UTF8
New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null
$scriptPath = Join-Path $InstallDir 'Send-MarsBackupStatus.ps1'
Copy-Item -Path (Join-Path $PSScriptRoot 'Send-MarsBackupStatus.ps1') -Destination $scriptPath -Force
Unblock-File -Path $scriptPath
$action = New-ScheduledTaskAction -Execute $WindowsPowerShell `
-Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$scriptPath`""
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date).AddMinutes(1) `
-RepetitionInterval (New-TimeSpan -Minutes $IntervalMinutes)
$principal = New-ScheduledTaskPrincipal -UserId 'S-1-5-18' -LogonType ServiceAccount -RunLevel Highest
$settings = New-ScheduledTaskSettingsSet -ExecutionTimeLimit (New-TimeSpan -Minutes 30) `
-MultipleInstances IgnoreNew -StartWhenAvailable
Register-ScheduledTask -TaskName $TaskName -Action $action -Trigger $trigger -Principal $principal `
-Settings $settings -Description 'Sends Azure Backup (MARS) results to the Backup Dashboard' -Force | Out-Null
Write-Host "Installed. Config: $ConfigPath"
Write-Host "Task '$TaskName' runs every $IntervalMinutes minutes as SYSTEM, using Windows PowerShell."
Write-Host ''
Write-Host 'Next steps:'
Write-Host " 1. Preview what will be sent: & '$WindowsPowerShell' -File '$scriptPath' -DryRun"
Write-Host " 2. Backfill 30 days of history: & '$WindowsPowerShell' -File '$scriptPath' -LookbackHours 720"
Write-Host " 3. Check the log: Get-Content '$DataDir\mars-collector.log' -Tail 20"
+66
View File
@@ -0,0 +1,66 @@
# Azure Backup (MARS agent) collector
A PowerShell script that runs on each server with the Microsoft Azure Recovery Services (MARS)
agent and reports its backups to the dashboard. Each server appears as one job,
named `<SERVER> - Azure Backup` by default.
> **Checked against:** MARS agent 2.0.9274, a Files & Folders policy with completed backups only.
> Not yet seen on a real agent: a failed backup, a backup in progress, or a System State policy.
> Those paths are written to the same object model, but if one looks wrong, run
> `Get-MarsCollectorDiagnostics.ps1` while it's in that state and send the output.
## Install (on the server, as an administrator)
Runs under Windows PowerShell 5.1 (`powershell.exe`), which the agent's `MSOnlineBackup` module
targets and which every supported Windows Server has.
1. On the dashboard host, create a key for this server:
```bash
python manage.py create-key mars-acme-fs01
```
2. Copy this folder to the server and run:
```powershell
.\Install-MarsCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
```
Add `-JobName "File Server"` to choose the name shown on the dashboard. It prompts for the
key, then:
- writes `C:\ProgramData\BackupDashboard\mars-collector.json` (readable only by SYSTEM and Administrators)
- copies the script to `C:\Program Files\BackupDashboard\`
- registers the scheduled task **Backup Dashboard - MARS collector** (SYSTEM, every 60 minutes)
3. Preview the payloads, then backfill 30 days so the history view is populated straight away:
```powershell
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-MarsBackupStatus.ps1' -DryRun
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-MarsBackupStatus.ps1' -LookbackHours 720
```
It can be installed alongside the Veeam collector: it uses its own config, state, log and task.
Re-run the installer to change settings or update the script.
## What it sends
| Dashboard | MARS |
|---|---|
| Company | `Tenant` from the config |
| Job | One per server, ID `mars:<computer name>`, so changing `JobName` renames rather than duplicates it |
| Status | Backup job state: Failed → **Failed**. Completed with failed bytes or an error → **Warning**. Aborted/cancelled or unrecognised → **Warning**. In progress → **Running**. Restore jobs are ignored |
| Started / finished | Job start / end time. MARS reports these in UTC without marking them as UTC; the collector handles that |
| Next scheduled | Next slot from the backup policy schedule, or the System State policy's if that's sooner. Empty (so never Overdue) for schedules that run every 2+ weeks, since which week is next can't be told |
| Message | Job and per-datasource errors (non-zero error codes), failed file counts, and the failed-file log path |
| Size | Bytes transferred, summed across datasources |
Each run covers the last 48 hours (`LookbackHours`). Jobs already delivered are recorded in
`mars-collector.state.json`.
## Troubleshooting
- Log: `C:\ProgramData\BackupDashboard\mars-collector.log`. Exit codes: `0` OK, `1` some runs
failed to send (retried next run), `2` setup failed (config or MARS module).
- Wrong status, sizes or messages: run `.\Get-MarsCollectorDiagnostics.ps1` (read-only). It prints
the real structure of the objects the agent's cmdlets return.
## Testing without MARS
```powershell
. ./tests/MockMars.ps1
./Send-MarsBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
```
+413
View File
@@ -0,0 +1,413 @@
<#
.SYNOPSIS
Reports Microsoft Azure Recovery Services (MARS) agent backups to the Backup Dashboard.
.DESCRIPTION
Runs on a server with the MARS agent, normally as a SYSTEM scheduled task every 60 minutes
(see Install-MarsCollector.ps1). The server appears on the dashboard as one job. Each run sends:
- the next scheduled backup, worked out from the backup (and System State) policy schedules,
which drives the dashboard's Overdue flag, and
- every backup job from the last LookbackHours that the dashboard hasn't already received in
its final state. A backup in progress is sent as Running until it finishes.
Already-reported jobs are tracked in a state file, and the ingest API is idempotent, so a
failed or repeated run is harmless: anything not confirmed is retried next time.
Written for Windows PowerShell 5.1, which the MARS agent's MSOnlineBackup module targets.
.PARAMETER ConfigPath
JSON config with ApiUrl, ApiKey and Tenant (optional: JobName, SourceSystem, LookbackHours).
.PARAMETER LookbackHours
Overrides the config value. Use 720 once to backfill 30 days of history.
.PARAMETER DryRun
Print the payloads instead of sending them. Doesn't touch the state or log files.
.EXAMPLE
.\Send-MarsBackupStatus.ps1 -DryRun
.EXAMPLE
.\Send-MarsBackupStatus.ps1 -LookbackHours 720
#>
[CmdletBinding()]
param(
[string]$ConfigPath = (Join-Path $env:ProgramData 'BackupDashboard\mars-collector.json'),
[int]$LookbackHours = 0,
[switch]$DryRun
)
$ErrorActionPreference = 'Stop'
$DataDir = Split-Path -Parent $ConfigPath
$LogPath = Join-Path $DataDir 'mars-collector.log'
$StatePath = Join-Path $DataDir 'mars-collector.state.json'
$MaxLogBytes = 5MB
# MARS keeps a limited job history; this comfortably covers 30 days at 3 backups a day
$MaxPreviousJobs = 200
$Invariant = [Globalization.CultureInfo]::InvariantCulture
# ---------- Helpers ----------
function Write-Log {
param([string]$Message, [string]$Level = 'INFO')
$line = '{0} [{1}] {2}' -f (Get-Date).ToString('yyyy-MM-dd HH:mm:ss'), $Level, $Message
Write-Host $line
if ($DryRun) { return }
if ((Test-Path $LogPath) -and (Get-Item $LogPath).Length -gt $MaxLogBytes) {
Move-Item $LogPath "$LogPath.1" -Force
}
Add-Content -Path $LogPath -Value $line -Encoding UTF8
}
function Get-ErrorText($ErrorRecord) {
# Windows PowerShell puts the HTTP response body (e.g. the API's 422 explanation) here
if ($ErrorRecord.ErrorDetails -and $ErrorRecord.ErrorDetails.Message) {
return "$($ErrorRecord.Exception.Message) $($ErrorRecord.ErrorDetails.Message)"
}
return $ErrorRecord.Exception.Message
}
# The API needs an explicit UTC offset. DateTimeOffset keeps a Local/Unspecified time as local and a
# Utc time as +00:00.
function ConvertTo-IsoTime($Value) {
if ($null -eq $Value) { return $null }
$dt = [datetime]$Value
if ($dt.Year -le 1900) { return $null } # "not set" (MARS uses 1601-01-01)
return ([DateTimeOffset]$dt).ToString("yyyy-MM-dd'T'HH:mm:sszzz", $Invariant)
}
# First property of $Object, out of $Names, that exists and isn't empty. MARS object property names
# aren't documented, so reads go through this rather than assuming one spelling.
function Get-FirstValue($Object, [string[]]$Names) {
if ($null -eq $Object) { return $null }
foreach ($name in $Names) {
try {
$prop = $Object.PSObject.Properties[$name]
if (-not $prop) { continue }
$value = $prop.Value
# Check emptiness by type: casting to [string] would make a one-item list of objects ''
if ($null -eq $value) { continue }
if ($value -is [string] -and $value -eq '') { continue }
if ($value -is [Collections.ICollection] -and $value.Count -eq 0) { continue }
return $value
} catch { }
}
return $null
}
function Import-Mars {
if (Get-Command Get-OBJob -ErrorAction SilentlyContinue) { return } # already loaded
if (Get-Module -ListAvailable -Name MSOnlineBackup) {
Import-Module MSOnlineBackup
return
}
# The agent normally adds its module folder to PSModulePath; fall back to the default install path
$agentModule = Join-Path $env:ProgramFiles 'Microsoft Azure Recovery Services Agent\bin\Modules\MSOnlineBackup'
Import-Module $agentModule
}
# ---------- Schedule ----------
# Next slot strictly after $After for a MARS schedule (days of week + times of day).
# Returns $null when there's no schedule, or when it runs every N>1 weeks: we can't tell which
# week is next, and a wrong guess would raise a false Overdue alarm.
function Get-NextSlot($Schedule, [datetime]$After) {
if ($null -eq $Schedule) { return $null }
$weeks = Get-FirstValue $Schedule @('ScheduleWeeklyFrequency', 'WeeksInterval')
if ($weeks -and [int]$weeks -gt 1) { return $null }
$days = @(Get-FirstValue $Schedule @('ScheduleRunDays', 'DaysOfWeek') | ForEach-Object { [string]$_ })
# TimeSpans (time of day, local) on real agents; accept DateTimes too
$times = @(Get-FirstValue $Schedule @('ScheduleRunTimes', 'TimesOfDay') | ForEach-Object {
if ($_ -is [timespan]) { $_ } else { ([datetime]$_).TimeOfDay }
} | Sort-Object)
if ($days.Count -eq 0 -or $times.Count -eq 0) { return $null }
for ($offset = 0; $offset -le 7; $offset++) {
$date = $After.Date.AddDays($offset)
if ($days -notcontains [string]$date.DayOfWeek) { continue }
foreach ($time in $times) {
$candidate = $date + $time
if ($candidate -gt $After) { return $candidate }
}
}
return $null
}
function Get-NextRun {
$next = $null
$policies = @()
try { $policies += @(Get-OBPolicy) } catch { Write-Log "Couldn't read the backup policy: $(Get-ErrorText $_)" 'WARN' }
# Windows Server can have a separate System State policy with its own schedule
if (Get-Command Get-OBSystemStatePolicy -ErrorAction SilentlyContinue) {
try { $policies += @(Get-OBSystemStatePolicy) } catch { }
}
foreach ($policy in $policies) {
if ($null -eq $policy) { continue }
try {
$slot = Get-NextSlot (Get-OBSchedule -Policy $policy) (Get-Date)
if ($slot -and (-not $next -or $slot -lt $next)) { $next = $slot }
} catch {
Write-Log "Couldn't read a policy schedule: $(Get-ErrorText $_)" 'WARN'
}
}
return ConvertTo-IsoTime $next
}
# ---------- Jobs ----------
# A job's StartTime/EndTime are UTC values, though not necessarily marked as UTC: on a real agent
# StartTime matched the "StartTimeUTC" in the status text, while DisplayTime held local time.
# Returns $null for "not set" (1601-01-01), e.g. EndTime while a backup is running.
function Get-JobTimeUtc($Job, [string]$Field) {
$value = Get-FirstValue $Job.JobStatus @($Field)
if ($null -eq $value) { return $null }
$dt = [datetime]$value
if ($dt.Year -le 1900) { return $null }
return [datetime]::SpecifyKind($dt, [DateTimeKind]::Utc)
}
# MARS always includes an ErrorInfo; ErrorCode 0 means no error
function Test-HasError($ErrorInfo) {
if (-not $ErrorInfo) { return $false }
$code = Get-FirstValue $ErrorInfo @('ErrorCode')
$detailed = Get-FirstValue $ErrorInfo @('DetailedErrorCode')
return ($code -and [long]$code -ne 0) -or ($detailed -and [long]$detailed -ne 0)
}
# DatasourceStatus.Datasource is an object whose DataSourceName is e.g. "E:\" or "System State"
function Get-DatasourceName($DatasourceStatus) {
$ds = Get-FirstValue $DatasourceStatus @('Datasource', 'DataSource')
$name = Get-FirstValue $ds @('DataSourceName', 'FriendlyNameDSPS', 'Name')
if ($name) { return [string]$name }
if ($ds -is [string]) { return $ds }
return 'Datasource'
}
function Get-FailedFiles($Job) {
$sum = [long]0
foreach ($ds in Get-DatasourceStatuses $Job) {
$value = Get-FirstValue (Get-FirstValue $ds @('FileProgress')) @('Failed')
if ($value) { $sum += [long]$value }
}
return $sum
}
function Get-JobStatus($Job) {
$status = $Job.JobStatus
$state = [string](Get-FirstValue $status @('JobState', 'State'))
if (-not (Get-JobTimeUtc $Job 'EndTime') -and $state -notmatch 'Fail|Abort|Cancel|Complete|Success|Succeed|Warn') {
return 'running'
}
if ($state -match 'Fail') { return 'failed' }
if ($state -match 'Warn|Abort|Cancel|Stop') { return 'warning' }
if ($state -match 'Complete|Success|Succeed') {
# MARS can report Completed while individual files failed or an error was recorded
if ((Get-FailedBytes $Job) -gt 0 -or (Get-FailedFiles $Job) -gt 0 -or
(Test-HasError (Get-FirstValue $status @('ErrorInfo')))) { return 'warning' }
return 'success'
}
return 'warning' # finished in a state we don't recognise: surface it rather than hide it
}
function Get-DatasourceStatuses($Job) {
return @(Get-FirstValue $Job.JobStatus @('DatasourceStatus', 'DataSourceStatus') | Where-Object { $_ })
}
function Get-ByteTotal($Job, [string]$Field) {
$sum = [long]0
foreach ($ds in Get-DatasourceStatuses $Job) {
$progress = Get-FirstValue $ds @('ByteProgress')
$value = Get-FirstValue $progress @($Field)
if ($value) { $sum += [long]$value }
}
return $sum
}
function Get-FailedBytes($Job) { return Get-ByteTotal $Job 'Failed' }
function Format-ErrorInfo($ErrorInfo) {
if (-not (Test-HasError $ErrorInfo)) { return $null }
$text = Get-FirstValue $ErrorInfo @('ErrorMessage', 'Message', 'Description')
if ($text) { return ([string]$text).Trim() }
$code = Get-FirstValue $ErrorInfo @('ErrorCode', 'DetailedErrorCode')
$params = @(Get-FirstValue $ErrorInfo @('ErrorParamList') | Where-Object { $_ } | ForEach-Object {
$name = Get-FirstValue $_ @('Name')
$value = Get-FirstValue $_ @('Value')
if ($name) { "$name=$value" } else { [string]$_ }
} | Where-Object { $_ })
$line = if ($code) { "Error code $code" } else { [string]$ErrorInfo }
if ($params.Count) { $line += " ($($params -join ', '))" }
return $line
}
function Get-RunMessage($Job, [string]$Status) {
$lines = New-Object 'System.Collections.Generic.List[string]'
$state = [string](Get-FirstValue $Job.JobStatus @('JobState', 'State'))
$jobError = Format-ErrorInfo (Get-FirstValue $Job.JobStatus @('ErrorInfo'))
if ($jobError) { $lines.Add($jobError) }
foreach ($ds in Get-DatasourceStatuses $Job) {
$name = Get-DatasourceName $ds
$dsError = Format-ErrorInfo (Get-FirstValue $ds @('ErrorInfo'))
$failedFiles = Get-FirstValue (Get-FirstValue $ds @('FileProgress')) @('Failed')
$failedBytes = Get-FirstValue (Get-FirstValue $ds @('ByteProgress')) @('Failed')
if ($dsError) { $lines.Add("${name}: $dsError") }
if ($failedFiles -and [long]$failedFiles -gt 0) { $lines.Add("${name}: $failedFiles file(s) failed to back up") }
elseif ($failedBytes -and [long]$failedBytes -gt 0) { $lines.Add("${name}: $failedBytes bytes failed to back up") }
}
$failedLog = Get-FirstValue $Job.JobStatus @('FailedFileLog')
if ($failedLog -and $lines.Count) { $lines.Add("Failed file log: $failedLog") }
if ($lines.Count -eq 0) {
switch ($Status) {
'success' { return 'Backup completed successfully.' }
'running' { return '' }
default { return "Backup finished with state '$state'." }
}
}
if ($state -and $Status -ne 'success') { $lines.Insert(0, "Job state: $state") }
return ($lines -join "`n")
}
function ConvertTo-Run($Job) {
$status = Get-JobStatus $Job
$datasources = @(Get-DatasourceStatuses $Job | ForEach-Object { Get-DatasourceName $_ })
$details = [ordered]@{ server = $env:COMPUTERNAME }
if ($datasources.Count) { $details.datasources = $datasources }
$agentState = Get-FirstValue $Job.JobStatus @('JobState', 'State')
if ($agentState) { $details.mars_state = [string]$agentState }
$run = [ordered]@{
external_id = [string]$Job.JobId
status = $status
started_at = ConvertTo-IsoTime (Get-JobTimeUtc $Job 'StartTime')
message = Get-RunMessage $Job $status
details = $details
}
if ($status -ne 'running') {
$finished = ConvertTo-IsoTime (Get-JobTimeUtc $Job 'EndTime')
if ($finished) { $run.finished_at = $finished } # otherwise the API uses receipt time
$transferred = Get-ByteTotal $Job 'Progress'
if ($transferred -gt 0) { $run.size_bytes = $transferred }
$failedBytes = Get-FailedBytes $Job
if ($failedBytes -gt 0) { $details.failed_bytes = $failedBytes }
$failedFiles = Get-FailedFiles $Job
if ($failedFiles -gt 0) { $details.failed_files = $failedFiles }
}
return $run
}
function Test-IsBackupJob($Job) {
# Get-OBJob also lists restores (JobType Recovery); only backups belong on the dashboard
$type = [string](Get-FirstValue $Job @('JobType', 'Type'))
return (-not $type) -or $type -match 'Backup'
}
function Send-Report($Body) {
$json = $Body | ConvertTo-Json -Depth 6 -Compress
if ($DryRun) {
Write-Host $json
return
}
# Send bytes: Windows PowerShell would otherwise encode a string body as Latin-1
$params = @{
Uri = "$($Config.ApiUrl.TrimEnd('/'))/api/v1/ingest"
Method = 'Post'
Headers = @{ Authorization = "Bearer $($Config.ApiKey)" }
ContentType = 'application/json; charset=utf-8'
Body = [Text.Encoding]::UTF8.GetBytes($json)
TimeoutSec = 30
}
Invoke-RestMethod @params | Out-Null
}
# ---------- Main ----------
try {
$Config = Get-Content -Path $ConfigPath -Raw | ConvertFrom-Json
foreach ($key in 'ApiUrl', 'ApiKey', 'Tenant') {
if (-not $Config.$key) { throw "Config $ConfigPath is missing '$key'" }
}
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Azure Backup (MARS)' }
$JobName = if ($Config.JobName) { [string]$Config.JobName } else { "$env:COMPUTERNAME - Azure Backup" }
if ($LookbackHours -le 0) {
$LookbackHours = if ($Config.LookbackHours) { [int]$Config.LookbackHours } else { 48 }
}
# Windows PowerShell 5.1 may default to TLS 1.0
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
Import-Mars
# Job ID -> start time (Unix seconds), for jobs already delivered in their final state
$sent = @{}
if (Test-Path $StatePath) {
$saved = Get-Content -Path $StatePath -Raw | ConvertFrom-Json
if ($saved) { foreach ($p in $saved.PSObject.Properties) { $sent[$p.Name] = [long]$p.Value } }
}
$sinceUtc = (Get-Date).ToUniversalTime().AddHours(-$LookbackHours)
# Get-OBJob with no arguments lists jobs in progress; -Previous lists finished ones
$jobs = @(@(Get-OBJob) + @(Get-OBJob -Previous $MaxPreviousJobs) | Where-Object {
$_ -and (Test-IsBackupJob $_)
})
$byId = [ordered]@{}
foreach ($job in $jobs) {
$start = Get-JobTimeUtc $job 'StartTime'
if (-not $start -or $start -lt $sinceUtc) { continue }
$byId[[string]$job.JobId] = $job # a job in both lists: the finished entry (later) wins
}
$pending = @($byId.Values |
Where-Object { -not $sent.ContainsKey([string]$_.JobId) } |
Sort-Object { Get-JobTimeUtc $_ 'StartTime' })
$jobInfo = [ordered]@{
# Stable per server, so renaming the dashboard job via JobName doesn't create a new one
external_id = "mars:$($env:COMPUTERNAME.ToLowerInvariant())"
name = $JobName
# Always sent (null when unscheduled) so the dashboard's value tracks the agent's
next_run_at = Get-NextRun
}
} catch {
Write-Log "Collector setup failed: $(Get-ErrorText $_)" 'ERROR'
exit 2
}
$failures = 0
$runsSent = 0
try {
if ($pending.Count -eq 0) {
# Keep next_run_at fresh even when nothing new ran
Send-Report ([ordered]@{ tenant = $Config.Tenant; source_system = $SourceSystem; job = $jobInfo })
}
foreach ($job in $pending) {
$run = ConvertTo-Run $job
Send-Report ([ordered]@{ tenant = $Config.Tenant; source_system = $SourceSystem; job = $jobInfo; run = $run })
$runsSent++
if ($run.status -ne 'running') {
$sent[$run.external_id] = ([DateTimeOffset](Get-JobTimeUtc $job 'StartTime')).ToUnixTimeSeconds()
}
}
} catch {
$failures++
Write-Log "Failed to report: $(Get-ErrorText $_)" 'ERROR'
}
if (-not $DryRun) {
# Forget jobs that have aged out of the lookback window (plus a day of slack)
$cutoff = [DateTimeOffset]::Now.AddHours(-$LookbackHours - 24).ToUnixTimeSeconds()
$keep = [ordered]@{}
foreach ($id in $sent.Keys) {
if ($sent[$id] -ge $cutoff) { $keep[$id] = $sent[$id] }
}
try {
$keep | ConvertTo-Json -Compress | Set-Content -Path $StatePath -Encoding UTF8
} catch {
Write-Log "Couldn't save state file: $(Get-ErrorText $_)" 'WARN'
}
}
Write-Log ("Reported {0} runs, {1} failures" -f $runsSent, $failures)
if ($failures) { exit 1 }
+78
View File
@@ -0,0 +1,78 @@
# Stand-ins for the MARS agent cmdlets. Object shapes follow Get-MarsCollectorDiagnostics.ps1
# output from a real agent (2.0.9274): job times are UTC values with no UTC marking, "not set" is
# 1601-01-01, ErrorInfo is always present (ErrorCode 0 = no error), Datasource is an object, and
# schedule times are TimeSpans.
#
# . ./tests/MockMars.ps1
# ./Send-MarsBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
# Fixed to the top of the hour, and IDs derived from times, so repeated runs see the same jobs
$now = Get-Date -Minute 0 -Second 0 -Millisecond 0
$notSet = [datetime]::new(1601, 1, 1)
function New-StableGuid([string]$Seed) {
[guid]::new([Security.Cryptography.MD5]::Create().ComputeHash([Text.Encoding]::UTF8.GetBytes($Seed)))
}
# Local time -> the UTC value MARS reports, with Kind Unspecified like the real objects
function ConvertTo-MarsTime([datetime]$Local) {
[datetime]::SpecifyKind($Local.ToUniversalTime(), [DateTimeKind]::Unspecified)
}
function New-MockErrorInfo([int]$Code = 0, $Params = @()) {
[pscustomobject]@{ DetailedErrorCode = 0; DetailedErrorSource = $null; ErrorCode = $Code; ErrorParamList = $Params }
}
function New-MockDatasource($Name, [uint64]$Done, [uint64]$FailedBytes = 0, [uint64]$FailedFiles = 0, $ErrorInfo = (New-MockErrorInfo)) {
[pscustomobject]@{
ByteProgress = [pscustomobject]@{ Changed = $Done; Failed = $FailedBytes; Progress = $Done; Total = 2TB }
Datasource = [pscustomobject]@{ DataSourceName = $Name; DataSourceId = 6310158806455554328 }
ErrorInfo = $ErrorInfo
FileProgress = [pscustomobject]@{ Changed = 100; CurrentFile = ''; Failed = $FailedFiles; Progress = 100; Total = 100 }
JobState = 'Completed'
}
}
function New-MockJob($Type, [datetime]$Start, $End, $State, $Datasources = @(), $ErrorInfo = (New-MockErrorInfo), $FailedFileLog = '') {
$endValue = if ($End) { ConvertTo-MarsTime $End } else { $notSet }
[pscustomobject]@{
IsRetried = $false
JobId = New-StableGuid "$Type|$($Start.Ticks)"
JobType = $Type
DisplayTime = $Start # the only local time on the object
JobStatus = [pscustomobject]@{
DatasourceStatus = $Datasources
EndTime = $endValue
ErrorInfo = $ErrorInfo
FailedFileLog = $FailedFileLog
JobState = $State
StartTime = ConvertTo-MarsTime $Start
}
}
}
$global:MockMarsRunning = @(
(New-MockJob 'Backup' $now.AddMinutes(-10) $null 'InProgress' @((New-MockDatasource 'E:\' 2GB)))
)
$global:MockMarsPrevious = @(
(New-MockJob 'Backup' $now.AddHours(-12) $now.AddHours(-11.5) 'Completed' @((New-MockDatasource 'E:\' 523407872))),
# Completed, but some files failed
(New-MockJob 'Backup' $now.AddHours(-36) $now.AddHours(-35) 'Completed' @(
(New-MockDatasource 'E:\' 3GB 40MB 12)) -FailedFileLog 'C:\Program Files\Microsoft Azure Recovery Services Agent\Temp\Errors.log'),
(New-MockJob 'Backup' $now.AddHours(-30) $now.AddHours(-29.5) 'Failed' @() `
(New-MockErrorInfo 100050 @([pscustomobject]@{ Name = 'Volume'; Value = 'E:' }))),
(New-MockJob 'Recovery' $now.AddHours(-5) $now.AddHours(-4) 'Completed'), # a restore: must be ignored
(New-MockJob 'Backup' $now.AddDays(-5) $now.AddDays(-5).AddHours(1) 'Completed') # outside 48h lookback
)
# Every day at 21:00 local, as on the real server
$global:MockMarsSchedule = [pscustomobject]@{
ScheduleRunDays = [DayOfWeek[]](0..6)
ScheduleRunTimes = [timespan[]]@([timespan]::FromHours(21))
ScheduleWeeklyFrequency = 1
TypeOfBackupSchedule = 'Daily'
}
function Get-OBJob { param([int]$Previous) if ($PSBoundParameters.ContainsKey('Previous')) { $global:MockMarsPrevious } else { $global:MockMarsRunning } }
function Get-OBPolicy { [pscustomobject]@{ PolicyName = 'policy'; BackupSchedule = $global:MockMarsSchedule } }
function Get-OBSchedule { param($Policy) $global:MockMarsSchedule }
function Get-OBSystemStatePolicy { $null } # none configured, as on the real server
@@ -0,0 +1,119 @@
<#
.SYNOPSIS
Read-only dump of what the Veeam Backup for Microsoft 365 cmdlets return, for troubleshooting
the collector.
.DESCRIPTION
The collector reads properties of Get-VBOJob / Get-VBOJobSession results whose names aren't
fully documented. This prints those objects' real structure (property names, types, values and
each date's Kind, a few levels deep) for a few jobs and their latest sessions. Nothing is
changed. Output is also saved to a text file.
.EXAMPLE
.\Get-Vb365CollectorDiagnostics.ps1
#>
[CmdletBinding()]
param(
[int]$Jobs = 3,
[int]$SessionsPerJob = 2,
[string]$OutFile = (Join-Path $env:TEMP 'vb365-collector-diagnostics.txt')
)
$ErrorActionPreference = 'Stop'
$MaxDepth = 4
if (-not (Get-Command Get-VBOJob -ErrorAction SilentlyContinue)) {
if (Get-Module -ListAvailable -Name Veeam.Archiver.PowerShell) {
Import-Module Veeam.Archiver.PowerShell
} else {
Import-Module (Join-Path $env:ProgramFiles 'Veeam\Backup365\Veeam.Archiver.PowerShell\Veeam.Archiver.PowerShell.psd1')
}
}
function Test-Scalar($Value) {
return $null -eq $Value -or $Value -is [string] -or $Value -is [ValueType] -or $Value -is [guid]
}
# Recursive "name (type) = value" listing. Collections show their first 3 items.
function Show-Object($Value, [string]$Indent = ' ', [int]$Depth = 0) {
if (Test-Scalar $Value) { return }
if ($Depth -ge $MaxDepth) { "$Indent..."; return }
if ($Value -is [Collections.IEnumerable]) {
$items = @($Value)
"$Indent[$($items.Count) item(s)]"
$i = 0
foreach ($item in $items | Select-Object -First 3) {
"$Indent[$i] ($($item.GetType().FullName))$(if (Test-Scalar $item) { " = $item" })"
Show-Object $item "$Indent " ($Depth + 1)
$i++
}
return
}
foreach ($p in $Value.PSObject.Properties) {
try { $v = $p.Value } catch { "$Indent$($p.Name) = <error: $($_.Exception.Message)>"; continue }
$type = if ($null -eq $v) { 'null' } else { $v.GetType().FullName }
if (Test-Scalar $v) {
$kind = if ($v -is [datetime]) { " [Kind=$($v.Kind)]" } else { '' }
"$Indent$($p.Name) ($type) = $v$kind"
} else {
$text = [string]$v
if ($text.Length -gt 100) { $text = $text.Substring(0, 97) + '...' }
"$Indent$($p.Name) ($type) = $text"
Show-Object $v "$Indent " ($Depth + 1)
}
}
}
function Show-Section([string]$Title, [scriptblock]$Get) {
"== $Title"
try {
$result = @(& $Get)
if ($result.Count -eq 0 -or ($result.Count -eq 1 -and $null -eq $result[0])) { ' (nothing returned)' }
$n = 0
foreach ($item in $result) {
if ($null -eq $item) { continue }
" [$n] ($($item.GetType().FullName))"
Show-Object $item ' '
$n++
}
} catch {
" error: $($_.Exception.Message)"
}
''
}
$report = & {
$module = Get-Module Veeam.Archiver.PowerShell
"Generated $(Get-Date -Format o) on $env:COMPUTERNAME, PowerShell $($PSVersionTable.PSVersion), module $($module.Version)"
"Local time zone: $([TimeZoneInfo]::Local.Id), UTC offset now $([TimeZoneInfo]::Local.GetUtcOffset((Get-Date))), culture $([Globalization.CultureInfo]::CurrentCulture.Name)"
''
"== Organizations"
try {
Get-VBOOrganization | ForEach-Object { " $($_.Name) [$($_.Type)]" }
} catch { " error: $($_.Exception.Message)" }
''
$allJobs = @(Get-VBOJob)
"== Jobs: $($allJobs.Count) in total"
$allJobs | ForEach-Object { " $($_.Name) [$($_.Id)] enabled=$($_.IsEnabled) next=$($_.NextRun)" }
''
Show-Section "Get-VBOJob (first $Jobs, full detail)" { $allJobs | Select-Object -First $Jobs }
foreach ($j in $allJobs | Select-Object -First $Jobs) {
Show-Section "Get-VBOJobSession -Job '$($j.Name)' (latest $SessionsPerJob)" {
Get-VBOJobSession -Job $j | Sort-Object CreationTime -Descending | Select-Object -First $SessionsPerJob
}
}
if (Get-Command Get-VBOCopyJob -ErrorAction SilentlyContinue) {
Show-Section 'Get-VBOCopyJob (first 2)' { Get-VBOCopyJob | Select-Object -First 2 }
if (Get-Command Get-VBOCopyJobSession -ErrorAction SilentlyContinue) {
Show-Section 'Get-VBOCopyJobSession (latest 2)' {
Get-VBOCopyJobSession | Sort-Object CreationTime -Descending | Select-Object -First 2
}
}
} else {
'== Get-VBOCopyJob: not available'
}
} | Out-String -Width 300
$report
$report | Set-Content -Path $OutFile -Encoding UTF8
Write-Host "Saved to $OutFile"
+105
View File
@@ -0,0 +1,105 @@
<#
.SYNOPSIS
Installs the Veeam Backup for Microsoft 365 collector as a SYSTEM scheduled task.
.DESCRIPTION
- Copies Send-Vb365BackupStatus.ps1 to Program Files (only admins can modify it, which
matters because it runs as SYSTEM).
- Writes the config to ProgramData\BackupDashboard and restricts that folder to SYSTEM and
Administrators, since it contains the API key.
- Registers a scheduled task that runs every IntervalMinutes, under Windows PowerShell 5.1
(powershell.exe) unless -PwshPath points at PowerShell 7.
Safe to re-run to change settings or update the script. Can sit alongside the other collectors.
Dashboard company per job: -OrganizationTenants maps Microsoft 365 organization names to
companies; jobs of unmapped organizations use -Tenant, or the organization's name if no -Tenant.
.EXAMPLE
.\Install-Vb365Collector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
One client per VB365 server. Prompts for the API key.
.EXAMPLE
.\Install-Vb365Collector.ps1 -ApiUrl https://backups.example.com -OrganizationTenants @{
'acme.onmicrosoft.com' = 'Acme Corp'; 'bluewater.onmicrosoft.com' = 'Bluewater Dental' }
A shared VB365 server protecting several organizations.
#>
#Requires -RunAsAdministrator
[CmdletBinding()]
param(
[Parameter(Mandatory)][string]$ApiUrl,
[string]$Tenant,
[hashtable]$OrganizationTenants = @{},
[string]$SourceSystem = 'Veeam Backup for M365',
[ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
[ValidateRange(1, 720)][int]$LookbackHours = 48,
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard'),
# Run under PowerShell 7 instead of Windows PowerShell, e.g. 'C:\Program Files\PowerShell\7\pwsh.exe'
[string]$PwshPath
)
$ErrorActionPreference = 'Stop'
$TaskName = 'Backup Dashboard - VB365 collector'
$DataDir = Join-Path $env:ProgramData 'BackupDashboard'
$ConfigPath = Join-Path $DataDir 'vb365-collector.json'
$Shell = Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\powershell.exe'
if ($PwshPath) {
if (-not (Test-Path $PwshPath)) { throw "PowerShell 7 not found at '$PwshPath'." }
$Shell = $PwshPath
}
# Check VB365 is here before changing anything
$moduleFile = Join-Path $env:ProgramFiles 'Veeam\Backup365\Veeam.Archiver.PowerShell\Veeam.Archiver.PowerShell.psd1'
if (-not (Get-Module -ListAvailable -Name Veeam.Archiver.PowerShell) -and -not (Test-Path $moduleFile)) {
throw 'The Veeam Backup for Microsoft 365 PowerShell module (Veeam.Archiver.PowerShell) is not installed on this server.'
}
if (-not $Tenant -and $OrganizationTenants.Count -eq 0) {
Write-Warning 'No -Tenant or -OrganizationTenants: jobs will be listed under their Microsoft 365 organization name.'
}
if ($ApiUrl -notmatch '^https://') {
Write-Warning 'ApiUrl is not HTTPS: the API key would be sent in clear text.'
}
# Read the key as a SecureString so it isn't echoed to the console or kept in shell history
$secure = Read-Host -AsSecureString 'API key (from: python manage.py create-key <name>)'
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
try { $apiKey = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) }
finally { [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr) }
if (-not $apiKey.StartsWith('bdk_')) { throw "That doesn't look like a dashboard API key (expected 'bdk_...')." }
# Config folder: SYSTEM + Administrators only (SIDs so this works on non-English Windows)
New-Item -ItemType Directory -Path $DataDir -Force | Out-Null
& icacls.exe $DataDir /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' | Out-Null
if ($LASTEXITCODE) { throw "icacls failed on $DataDir" }
$config = [ordered]@{
ApiUrl = $ApiUrl.TrimEnd('/')
ApiKey = $apiKey
SourceSystem = $SourceSystem
LookbackHours = $LookbackHours
OrganizationTenants = $OrganizationTenants
}
if ($Tenant) { $config.Tenant = $Tenant }
$config | ConvertTo-Json | Set-Content -Path $ConfigPath -Encoding UTF8
New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null
$scriptPath = Join-Path $InstallDir 'Send-Vb365BackupStatus.ps1'
Copy-Item -Path (Join-Path $PSScriptRoot 'Send-Vb365BackupStatus.ps1') -Destination $scriptPath -Force
Unblock-File -Path $scriptPath
$action = New-ScheduledTaskAction -Execute $Shell `
-Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$scriptPath`""
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date).AddMinutes(1) `
-RepetitionInterval (New-TimeSpan -Minutes $IntervalMinutes)
$principal = New-ScheduledTaskPrincipal -UserId 'S-1-5-18' -LogonType ServiceAccount -RunLevel Highest
$settings = New-ScheduledTaskSettingsSet -ExecutionTimeLimit (New-TimeSpan -Minutes 30) `
-MultipleInstances IgnoreNew -StartWhenAvailable
Register-ScheduledTask -TaskName $TaskName -Action $action -Trigger $trigger -Principal $principal `
-Settings $settings -Description 'Sends Veeam Backup for Microsoft 365 results to the Backup Dashboard' -Force | Out-Null
Write-Host "Installed. Config: $ConfigPath"
Write-Host "Task '$TaskName' runs every $IntervalMinutes minutes as SYSTEM, using $Shell."
Write-Host ''
Write-Host 'Next steps:'
Write-Host " 1. Preview what will be sent: & '$Shell' -File '$scriptPath' -DryRun"
Write-Host " 2. Backfill 30 days of history: & '$Shell' -File '$scriptPath' -LookbackHours 720"
Write-Host " 3. Check the log: Get-Content '$DataDir\vb365-collector.log' -Tail 20"
+86
View File
@@ -0,0 +1,86 @@
# Veeam Backup for Microsoft 365 collector
A PowerShell script that runs on the Veeam Backup for Microsoft 365 (VB365) server and reports its
backup jobs (and backup copy jobs) to the dashboard, one dashboard job per VB365 job.
> **Checked against:** VB365 8.4 under Windows PowerShell 5.1, a shared server protecting 11
> organizations: successful, warning and retried sessions. Not yet seen on a real server: a failed
> session, a session in progress, or a backup copy job. If one looks wrong, run
> `Get-Vb365CollectorDiagnostics.ps1` while it's in that state and send the output.
## Which company a job appears under
One VB365 server can protect several Microsoft 365 organizations, so the company is picked per job
from the job's organization:
1. `OrganizationTenants` maps an organization name (as `Get-VBOOrganization` shows it, e.g.
`acme.onmicrosoft.com`; not case-sensitive) to a dashboard company.
2. Otherwise `Tenant`, for a VB365 server dedicated to one client.
3. Otherwise the organization's own name.
Changing a job's company later creates a new dashboard job under the new company; delete the old
one with `python manage.py delete-job <id>`.
## Install (on the VB365 server, as an administrator)
1. On the dashboard host, create a key for this server:
```bash
python manage.py create-key vb365-01
```
2. Copy this folder to the server and run one of:
```powershell
# VB365 server for one client
.\Install-Vb365Collector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
# Shared VB365 server
.\Install-Vb365Collector.ps1 -ApiUrl https://backups.example.com -OrganizationTenants @{
'acme.onmicrosoft.com' = 'Acme Corp'
'bluewater.onmicrosoft.com' = 'Bluewater Dental'
}
```
It prompts for the key, then:
- writes `C:\ProgramData\BackupDashboard\vb365-collector.json` (readable only by SYSTEM and Administrators)
- copies the script to `C:\Program Files\BackupDashboard\`
- registers the scheduled task **Backup Dashboard - VB365 collector** (SYSTEM, every 60 minutes)
It runs under Windows PowerShell 5.1. If your VB365 version's module needs PowerShell 7, add
`-PwshPath 'C:\Program Files\PowerShell\7\pwsh.exe'`.
3. Preview the payloads, then backfill 30 days so the history view is populated straight away:
```powershell
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-Vb365BackupStatus.ps1' -DryRun
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-Vb365BackupStatus.ps1' -LookbackHours 720
```
It can be installed alongside the other collectors: it uses its own config, state, log and task.
Re-run the installer to change settings (including the organization mapping) or update the script.
## What it sends
| Dashboard | VB365 |
|---|---|
| Company | See above |
| Job | Each backup job and backup copy job. The job's GUID is the ID, so renaming it in VB365 doesn't duplicate it |
| Status | Session status: Success / Warning / Failed. Running or queued → **Running**. Stopped (cancelled) or unrecognised → **Warning** |
| Started / finished | Session `CreationTime` / `EndTime` |
| Next scheduled | Job's `NextRun`. Empty for disabled jobs |
| Message | Warning and error lines from the session log (items titled `[Warning] ...`, `[Failed] ...`), plus "Veeam will retry this job." when a retry is pending |
| Retries | VB365 runs a retry as a separate session, so it appears as its own run, marked `details.retry_of` (the session it retried). The retried run is marked `details.will_retry` |
| Size | Data transferred (`Statistics.TransferredData`) |
| Details | Organization, objects processed, job kind (`backup` / `copy`) |
Each run covers the last 48 hours (`LookbackHours`). Sessions already delivered are recorded in
`vb365-collector.state.json`.
## Troubleshooting
- Log: `C:\ProgramData\BackupDashboard\vb365-collector.log`. Exit codes: `0` OK, `1` some jobs
failed to send (retried next run), `2` setup failed (config or VB365 module).
- Wrong status, times or messages: run `.\Get-Vb365CollectorDiagnostics.ps1` (read-only). It prints
the real structure of jobs and sessions, including whether each date is local or UTC.
## Testing without VB365
```powershell
. ./tests/MockVb365.ps1
./Send-Vb365BackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
```
+372
View File
@@ -0,0 +1,372 @@
<#
.SYNOPSIS
Reports Veeam Backup for Microsoft 365 (VB365) job results to the Backup Dashboard.
.DESCRIPTION
Runs on the VB365 server, normally as a SYSTEM scheduled task every 60 minutes
(see Install-Vb365Collector.ps1). For every backup job (and backup copy job) it sends:
- the job's next scheduled run, which drives the dashboard's Overdue flag, and
- every session from the last LookbackHours that the dashboard hasn't already received in
its final state. Running sessions are re-sent each time until they finish.
One VB365 server can protect several Microsoft 365 organizations, so the dashboard company is
chosen per job from its organization: OrganizationTenants in the config maps an organization
name (e.g. "contoso.onmicrosoft.com") to a company; otherwise Tenant is used; otherwise the
organization's own name.
Already-reported sessions are tracked in a state file, and the ingest API is idempotent, so a
failed or repeated run is harmless: anything not confirmed is retried next time.
.PARAMETER ConfigPath
JSON config with ApiUrl and ApiKey (optional: Tenant, OrganizationTenants, SourceSystem,
LookbackHours).
.PARAMETER LookbackHours
Overrides the config value. Use 720 once to backfill 30 days of history.
.PARAMETER DryRun
Print the payloads instead of sending them. Doesn't touch the state or log files.
.EXAMPLE
.\Send-Vb365BackupStatus.ps1 -DryRun
.EXAMPLE
.\Send-Vb365BackupStatus.ps1 -LookbackHours 720
#>
[CmdletBinding()]
param(
[string]$ConfigPath = (Join-Path $env:ProgramData 'BackupDashboard\vb365-collector.json'),
[int]$LookbackHours = 0,
[switch]$DryRun
)
$ErrorActionPreference = 'Stop'
$DataDir = Split-Path -Parent $ConfigPath
$LogPath = Join-Path $DataDir 'vb365-collector.log'
$StatePath = Join-Path $DataDir 'vb365-collector.state.json'
$MaxLogBytes = 5MB
$Invariant = [Globalization.CultureInfo]::InvariantCulture
# ---------- Helpers ----------
function Write-Log {
param([string]$Message, [string]$Level = 'INFO')
$line = '{0} [{1}] {2}' -f (Get-Date).ToString('yyyy-MM-dd HH:mm:ss'), $Level, $Message
Write-Host $line
if ($DryRun) { return }
if ((Test-Path $LogPath) -and (Get-Item $LogPath).Length -gt $MaxLogBytes) {
Move-Item $LogPath "$LogPath.1" -Force
}
Add-Content -Path $LogPath -Value $line -Encoding UTF8
}
function Get-ErrorText($ErrorRecord) {
# Windows PowerShell puts the HTTP response body (e.g. the API's 422 explanation) here
if ($ErrorRecord.ErrorDetails -and $ErrorRecord.ErrorDetails.Message) {
return "$($ErrorRecord.Exception.Message) $($ErrorRecord.ErrorDetails.Message)"
}
return $ErrorRecord.Exception.Message
}
# The API needs an explicit UTC offset. DateTimeOffset keeps a Local/Unspecified time as local and a
# Utc time as +00:00.
function ConvertTo-IsoTime($Value) {
if ($null -eq $Value) { return $null }
$dt = [datetime]$Value
if ($dt.Year -le 1900) { return $null } # "not set" (DateTime.MinValue and friends)
return ([DateTimeOffset]$dt).ToString("yyyy-MM-dd'T'HH:mm:sszzz", $Invariant)
}
# First property of $Object, out of $Names, that exists and isn't empty. The VB365 object model isn't
# fully documented, so reads go through this rather than assuming one spelling.
function Get-FirstValue($Object, [string[]]$Names) {
if ($null -eq $Object) { return $null }
foreach ($name in $Names) {
try {
$prop = $Object.PSObject.Properties[$name]
if (-not $prop) { continue }
$value = $prop.Value
# Check emptiness by type: casting to [string] would make a one-item list of objects ''
if ($null -eq $value) { continue }
if ($value -is [string] -and $value -eq '') { continue }
if ($value -is [Collections.ICollection] -and $value.Count -eq 0) { continue }
return $value
} catch { }
}
return $null
}
function Import-Vb365 {
if (Get-Command Get-VBOJob -ErrorAction SilentlyContinue) { return } # already loaded
if (Get-Module -ListAvailable -Name Veeam.Archiver.PowerShell) {
Import-Module Veeam.Archiver.PowerShell
return
}
# Fall back to the default install path
Import-Module (Join-Path $env:ProgramFiles 'Veeam\Backup365\Veeam.Archiver.PowerShell\Veeam.Archiver.PowerShell.psd1')
}
function Get-Time($Object, [string]$Field) {
$value = Get-FirstValue $Object @($Field)
if ($null -eq $value) { return $null }
$dt = [datetime]$value
if ($dt.Year -le 1900) { return $null } # e.g. EndTime while a session is running
return $dt
}
# "1.5 GB", "512 MB", "123456" -> bytes; $null when it can't be read
function ConvertTo-Bytes($Value) {
if ($null -eq $Value) { return $null }
if ($Value -is [ValueType]) { return [long]$Value }
$match = [regex]::Match([string]$Value, '^\s*([\d.,]+)\s*([KMGTP]?B)?\s*$', 'IgnoreCase')
if (-not $match.Success) { return $null }
$number = 0.0
if (-not [double]::TryParse(($match.Groups[1].Value -replace ',', '.'), [Globalization.NumberStyles]::Float, $Invariant, [ref]$number)) {
return $null
}
$power = @{ '' = 0; 'B' = 0; 'KB' = 1; 'MB' = 2; 'GB' = 3; 'TB' = 4; 'PB' = 5 }[$match.Groups[2].Value.ToUpperInvariant()]
return [long]($number * [math]::Pow(1024, $power))
}
# ---------- Jobs and tenants ----------
function Get-OrganizationName($Job) {
$org = Get-FirstValue $Job @('Organization')
if ($null -eq $org) {
# Copy jobs point at their source backup job
$org = Get-FirstValue (Get-FirstValue $Job @('BackupJob')) @('Organization')
}
if ($org -is [string]) { return $org }
$name = Get-FirstValue $org @('Name', 'OfficeName', 'DisplayName')
if ($name) { return [string]$name }
return $null
}
function Get-TenantForJob($Job) {
$org = Get-OrganizationName $Job
if ($org -and $Config.OrganizationTenants) {
$mapped = $Config.OrganizationTenants.PSObject.Properties[$org] # case-insensitive lookup
if ($mapped -and $mapped.Value) { return [string]$mapped.Value }
}
if ($Config.Tenant) { return [string]$Config.Tenant }
if ($org) { return $org }
return $env:COMPUTERNAME
}
function Get-NextRun($Job) {
try {
$enabled = Get-FirstValue $Job @('IsEnabled', 'Enabled')
if ($enabled -eq $false) { return $null }
return ConvertTo-IsoTime (Get-Time $Job 'NextRun')
} catch {
Write-Log "Couldn't read schedule for job '$($Job.Name)': $(Get-ErrorText $_)" 'WARN'
return $null
}
}
# ---------- Sessions ----------
function Get-RunStatus($Session) {
$status = [string](Get-FirstValue $Session @('Status', 'Result', 'State'))
if ($status -match 'Fail|Error') { return 'failed' }
if ($status -match 'Warn') { return 'warning' }
if ($status -match 'Succe') { return 'success' }
if ($status -match 'Run|Queue|Progress|Start|Wait') { return 'running' }
if (-not (Get-Time $Session 'EndTime')) { return 'running' }
# Stopped (cancelled), Disconnected, or a state we don't recognise: surface it rather than hide it
return 'warning'
}
function Test-EmptyGuid($Value) {
return (-not $Value) -or [string]$Value -eq [string][guid]::Empty
}
# Warning/error lines from the session log. Each log item's status is a prefix of its Title, e.g.
# "[Warning] Item was skipped" (there's no separate status property), so match on that rather than
# on words in the text, which would catch e.g. "[Success] Processing site: .../error-reports".
function Get-RunMessage($Session, [string]$Status) {
if ($Status -eq 'running') { return '' }
$lines = New-Object 'System.Collections.Generic.List[string]'
$sessionStatus = [string](Get-FirstValue $Session @('Status', 'Result', 'State'))
if ($Status -ne 'success') {
foreach ($entry in @(Get-FirstValue $Session @('Log', 'Logs') | Where-Object { $_ })) {
$title = ([string](Get-FirstValue $entry @('Title', 'Message', 'Text'))).Trim()
$prefix = [regex]::Match($title, '^\[(\w+)\]\s*')
if ($prefix.Success) {
if ($prefix.Groups[1].Value -notmatch 'Warn|Fail|Error') { continue }
$title = $title.Substring($prefix.Length)
} elseif ([string](Get-FirstValue $entry @('Status', 'Type', 'Severity')) -notmatch 'Warn|Fail|Error') {
continue
}
if ($title -and -not $lines.Contains($title)) { $lines.Add($title) }
}
}
if ($lines.Count -eq 0) {
if ($Status -eq 'success') { $lines.Add('Job completed successfully.') }
else { $lines.Add("Job finished with status '$sessionStatus'.") }
} elseif ($sessionStatus -and $sessionStatus -notmatch 'Fail|Warn') {
$lines.Insert(0, "Session status: $sessionStatus")
}
# A warning/failure VB365 is about to retry; the retry arrives as its own run
if ($Status -ne 'success' -and (Get-FirstValue $Session @('JobWillBeRetried')) -eq $true) {
$lines.Add('Veeam will retry this job.')
}
return ($lines -join "`n")
}
function ConvertTo-Run($Session, $Job, [string]$JobKind) {
$status = Get-RunStatus $Session
$details = [ordered]@{ vb365_server = $env:COMPUTERNAME; job_kind = $JobKind }
$org = Get-OrganizationName $Job
if ($org) { $details.organization = $org }
$sessionStatus = Get-FirstValue $Session @('Status', 'Result', 'State')
if ($sessionStatus) { $details.vb365_status = [string]$sessionStatus }
# Retries are separate sessions pointing at the session they retry
$mainSession = Get-FirstValue $Session @('MainSessionId')
if (-not (Test-EmptyGuid $mainSession)) { $details.retry_of = [string]$mainSession }
if ((Get-FirstValue $Session @('JobWillBeRetried')) -eq $true) { $details.will_retry = $true }
$run = [ordered]@{
external_id = [string](Get-FirstValue $Session @('Id', 'SessionId'))
status = $status
started_at = ConvertTo-IsoTime (Get-Time $Session 'CreationTime')
message = Get-RunMessage $Session $status
details = $details
}
if ($status -ne 'running') {
$finished = ConvertTo-IsoTime (Get-Time $Session 'EndTime')
if ($finished) { $run.finished_at = $finished } # otherwise the API uses receipt time
$stats = Get-FirstValue $Session @('Statistics')
$processed = Get-FirstValue $stats @('ProcessedObjects')
if ($null -ne $processed) { $details.processed_objects = $processed }
$transferred = ConvertTo-Bytes (Get-FirstValue $stats @('TransferredData', 'TransferredDataSize'))
if ($transferred -gt 0) { $run.size_bytes = $transferred }
}
return $run
}
function Send-Report($Body) {
$json = $Body | ConvertTo-Json -Depth 6 -Compress
if ($DryRun) {
Write-Host $json
return
}
# Send bytes: Windows PowerShell would otherwise encode a string body as Latin-1
$params = @{
Uri = "$($Config.ApiUrl.TrimEnd('/'))/api/v1/ingest"
Method = 'Post'
Headers = @{ Authorization = "Bearer $($Config.ApiKey)" }
ContentType = 'application/json; charset=utf-8'
Body = [Text.Encoding]::UTF8.GetBytes($json)
TimeoutSec = 30
}
Invoke-RestMethod @params | Out-Null
}
# ---------- Main ----------
try {
$Config = Get-Content -Path $ConfigPath -Raw | ConvertFrom-Json
foreach ($key in 'ApiUrl', 'ApiKey') {
if (-not $Config.$key) { throw "Config $ConfigPath is missing '$key'" }
}
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Veeam Backup for M365' }
if ($LookbackHours -le 0) {
$LookbackHours = if ($Config.LookbackHours) { [int]$Config.LookbackHours } else { 48 }
}
# Windows PowerShell 5.1 may default to TLS 1.0
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
Import-Vb365
# Session ID -> start time (Unix seconds), for sessions already delivered in their final state
$sent = @{}
if (Test-Path $StatePath) {
$saved = Get-Content -Path $StatePath -Raw | ConvertFrom-Json
if ($saved) { foreach ($p in $saved.PSObject.Properties) { $sent[$p.Name] = [long]$p.Value } }
}
$since = (Get-Date).AddHours(-$LookbackHours)
# (job, kind, sessions) for backup jobs, plus backup copy jobs where the cmdlets exist
$work = New-Object 'System.Collections.Generic.List[object]'
foreach ($job in @(Get-VBOJob)) {
$sessions = @(Get-VBOJobSession -Job $job | Where-Object { (Get-Time $_ 'CreationTime') -ge $since })
$work.Add([pscustomobject]@{ Job = $job; Kind = 'backup'; Sessions = $sessions })
}
if ((Get-Command Get-VBOCopyJob -ErrorAction SilentlyContinue) -and (Get-Command Get-VBOCopyJobSession -ErrorAction SilentlyContinue)) {
$copyJobs = @(Get-VBOCopyJob)
if ($copyJobs.Count) {
# Fetch once and match by job ID, since the per-job parameter name isn't consistent
$copySessions = @(Get-VBOCopyJobSession | Where-Object { (Get-Time $_ 'CreationTime') -ge $since })
foreach ($job in $copyJobs) {
$id = [string]$job.Id
$work.Add([pscustomobject]@{
Job = $job; Kind = 'copy'
Sessions = @($copySessions | Where-Object { [string](Get-FirstValue $_ @('JobId')) -eq $id })
})
}
}
}
} catch {
Write-Log "Collector setup failed: $(Get-ErrorText $_)" 'ERROR'
exit 2
}
$failures = 0
$runsSent = 0
foreach ($item in $work) {
$job = $item.Job
$tenant = Get-TenantForJob $job
$jobInfo = [ordered]@{
external_id = [string]$job.Id
name = [string]$job.Name
# Always sent (null when disabled/unscheduled) so the dashboard's value tracks VB365's
next_run_at = Get-NextRun $job
}
$pending = @($item.Sessions |
Where-Object { $_ -and -not $sent.ContainsKey([string](Get-FirstValue $_ @('Id', 'SessionId'))) } |
Sort-Object { Get-Time $_ 'CreationTime' })
try {
if ($pending.Count -eq 0) {
# Keep next_run_at fresh even when nothing new ran
Send-Report ([ordered]@{ tenant = $tenant; source_system = $SourceSystem; job = $jobInfo })
}
foreach ($session in $pending) {
$run = ConvertTo-Run $session $job $item.Kind
Send-Report ([ordered]@{ tenant = $tenant; source_system = $SourceSystem; job = $jobInfo; run = $run })
$runsSent++
if ($run.status -ne 'running') {
$sent[$run.external_id] = ([DateTimeOffset](Get-Time $session 'CreationTime')).ToUnixTimeSeconds()
}
}
} catch {
$failures++
Write-Log "Failed to report job '$($job.Name)': $(Get-ErrorText $_)" 'ERROR'
$httpStatus = 0
try { $httpStatus = [int]$_.Exception.Response.StatusCode } catch { }
if ($httpStatus -eq 401) {
Write-Log 'API key rejected; check ApiKey in the config. Stopping.' 'ERROR'
break
}
}
}
if (-not $DryRun) {
# Forget sessions that have aged out of the lookback window (plus a day of slack)
$cutoff = [DateTimeOffset]::Now.AddHours(-$LookbackHours - 24).ToUnixTimeSeconds()
$keep = [ordered]@{}
foreach ($id in $sent.Keys) {
if ($sent[$id] -ge $cutoff) { $keep[$id] = $sent[$id] }
}
try {
$keep | ConvertTo-Json -Compress | Set-Content -Path $StatePath -Encoding UTF8
} catch {
Write-Log "Couldn't save state file: $(Get-ErrorText $_)" 'WARN'
}
}
Write-Log ("Reported {0} jobs, {1} runs, {2} failures" -f $work.Count, $runsSent, $failures)
if ($failures) { exit 1 }
+78
View File
@@ -0,0 +1,78 @@
# Stand-ins for the Veeam Backup for Microsoft 365 cmdlets. Object shapes follow
# Get-Vb365CollectorDiagnostics.ps1 output from a real VB365 8.4 server: session and NextRun times
# are local, Statistics.TransferredData is a string ("187.2 MB"), log items carry their status as a
# Title prefix ("[Warning] ..."), and retries are separate sessions whose MainSessionId points at
# the session they retry (which has JobWillBeRetried = True).
#
# . ./tests/MockVb365.ps1
# ./Send-Vb365BackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
# Fixed to the top of the hour, and IDs derived from names/times, so repeated runs see the same data
$now = Get-Date -Minute 0 -Second 0 -Millisecond 0
function New-StableGuid([string]$Seed) {
[guid]::new([Security.Cryptography.MD5]::Create().ComputeHash([Text.Encoding]::UTF8.GetBytes($Seed)))
}
function New-MockOrg($Name) { [pscustomobject]@{ Name = $Name; OfficeName = $Name; Type = 'Office365'; IsBackedUp = $true } }
$acme = New-MockOrg 'acme.onmicrosoft.com'
$blue = New-MockOrg 'bluewater.onmicrosoft.com'
function New-MockJob($Name, $Org, $NextRun, [bool]$Enabled = $true) {
[pscustomobject]@{
Id = New-StableGuid $Name; Name = $Name; Organization = $Org; JobType = 'Backup'
IsEnabled = $Enabled; NextRun = $NextRun; LastStatus = 'Success'
LastRun = $(if ($NextRun) { $NextRun.AddDays(-1).ToUniversalTime() }) # UTC, as on the real server
}
}
function New-LogItem($Status, $Text, $At) {
[pscustomobject]@{ Id = [guid]::NewGuid(); USN = 1; Title = "[$Status] $Text"; CreationTime = $At; EndTime = $At }
}
function New-MockSession($Job, $Start, $End, $Status, $Transferred = '0 B', [long]$Objects = 0, $Log = @(),
$MainSessionId = [guid]::Empty, [bool]$WillRetry = $false) {
$logItems = @((New-LogItem 'Success' "Job started at: $($Start.ToString('G'))" $Start)) + @($Log)
[pscustomobject]@{
JobId = $Job.Id; JobName = $Job.Name; Status = $Status
Statistics = [pscustomobject]@{ ProcessingRate = '1.6 MB/s (4 items/s)'; TransferredData = $Transferred; ProcessedObjects = $Objects; Bottleneck = 'Target' }
JobSessionConfigType = 'Incremental'
Id = New-StableGuid "$($Job.Name)|$($Start.Ticks)"
CreationTime = $Start
EndTime = $(if ($End) { $End } else { [datetime]::MinValue })
Progress = 49; MainSessionId = $MainSessionId; JobWillBeRetried = $WillRetry
Log = $logItems
}
}
$global:MockVboJobs = @(
(New-MockJob 'Acme 365 Backup' $acme $now.Date.AddHours(17).AddDays($(if ($now.Hour -ge 17) { 1 } else { 0 }))),
(New-MockJob 'Bluewater 365 Backup' $blue $now.Date.AddHours(17.75).AddDays($(if ($now.Hour -ge 17) { 1 } else { 0 }))),
(New-MockJob 'Bluewater Old Mailboxes' $blue $now.AddDays(1) $false) # disabled
)
$acmeJob, $blueJob, $oldJob = $global:MockVboJobs
$y = $now.Date.AddDays(-1) # yesterday
$blueWarn = New-MockSession $blueJob $y.AddHours(17.75) $y.AddHours(17.75).AddMinutes(3.5) 'Warning' '32.9 MB' 47 @(
(New-LogItem 'Success' 'Found 94 objects' $y.AddHours(17.75)),
# A success line mentioning "error" must not end up in the message
(New-LogItem 'Success' 'Processing site: https://bluewater.sharepoint.com/sites/error-reports' $y.AddHours(17.76)),
(New-LogItem 'Warning' 'Processing mailbox [email protected] finished with warning: Item "Q3.pst" exceeds the maximum size' $y.AddHours(17.78))
) -WillRetry $true
$global:MockVboSessions = @(
(New-MockSession $acmeJob $y.AddHours(17) $y.AddHours(17).AddMinutes(4.8) 'Success' '187.2 MB' 568 @(
(New-LogItem 'Success' 'Found 49 objects' $y.AddHours(17)), (New-LogItem 'Success' 'Connected to organization' $y.AddHours(17)))),
(New-MockSession $acmeJob $now.AddMinutes(-20) $null 'Running' '0 B' 12),
$blueWarn,
# Its retry 13 minutes later, which succeeded
(New-MockSession $blueJob $y.AddHours(17.75).AddMinutes(13.5) $y.AddHours(17.75).AddMinutes(16.8) 'Success' '0 B' 0 -MainSessionId $blueWarn.Id),
(New-MockSession $blueJob $now.AddHours(-30) $now.AddHours(-29.9) 'Failed' '0 B' 0 @(
(New-LogItem 'Failed' 'Failed to connect to organization: The remote server returned an error: (401) Unauthorized.' $now.AddHours(-30)))),
(New-MockSession $oldJob $now.AddDays(-5) $now.AddDays(-5).AddHours(1) 'Success') # outside 48h lookback
)
function Get-VBOJob { $global:MockVboJobs }
function Get-VBOJobSession { param($Job) $global:MockVboSessions | Where-Object { -not $Job -or $_.JobId -eq $Job.Id } }
function Get-VBOOrganization { $acme, $blue }
function Get-VBOCopyJob { } # none, as on the real server
function Get-VBOCopyJobSession { }
@@ -9,6 +9,7 @@
2. sessions whose JobId matches no job (the runs that are going missing) 2. sessions whose JobId matches no job (the runs that are going missing)
3. for jobs with no matching sessions, their type/link-related properties and methods 3. for jobs with no matching sessions, their type/link-related properties and methods
4. what other job cmdlets (e.g. Get-VBRBackupCopyJob) return 4. what other job cmdlets (e.g. Get-VBRBackupCopyJob) return
5. each job's schedule properties, which the collector reads for the next run
Nothing is changed in Veeam or the dashboard. Output is also saved to a text file. Nothing is changed in Veeam or the dashboard. Output is also saved to a text file.
.EXAMPLE .EXAMPLE
@@ -42,14 +43,14 @@ function Format-Value($Value) {
# Property and method names matching $LinkPattern, with property values. Every read is guarded: # Property and method names matching $LinkPattern, with property values. Every read is guarded:
# some Veeam properties throw when accessed. # some Veeam properties throw when accessed.
function Show-LinkMembers($Object) { function Show-LinkMembers($Object, [string]$Pattern = $LinkPattern, [string]$Indent = ' ') {
foreach ($p in $Object.PSObject.Properties | Where-Object { $_.Name -match $LinkPattern }) { foreach ($p in $Object.PSObject.Properties | Where-Object { $_.Name -match $Pattern }) {
$value = try { Format-Value $p.Value } catch { "<error: $($_.Exception.Message)>" } $value = try { Format-Value $p.Value } catch { "<error: $($_.Exception.Message)>" }
' {0} = {1}' -f $p.Name, $value '{0}{1} = {2}' -f $Indent, $p.Name, $value
} }
$methods = @($Object | Get-Member -MemberType Methods -ErrorAction SilentlyContinue | $methods = @($Object | Get-Member -MemberType Methods -ErrorAction SilentlyContinue |
Where-Object { $_.Name -match $LinkPattern } | ForEach-Object Name) Where-Object { $_.Name -match $Pattern } | ForEach-Object Name)
if ($methods) { ' methods: ' + ($methods -join ', ') } if ($methods) { $Indent + 'methods: ' + ($methods -join ', ') }
} }
function Get-Prop($Object, [string]$Name) { function Get-Prop($Object, [string]$Name) {
@@ -147,6 +148,39 @@ $report = & {
" ${cmd}: error: $($_.Exception.Message)" " ${cmd}: error: $($_.Exception.Message)"
} }
} }
''
# The collector uses IsScheduleEnabled and ScheduleOptions.NextRun; show what else is there in
# case this version keeps the next run somewhere different
"== 5. Schedules (culture: $([Globalization.CultureInfo]::CurrentCulture.Name))"
$SchedulePattern = 'Schedul|Next|Latest|Run|Enabled|Daily|Periodic|Monthly|Continuous|After|Option'
foreach ($j in $jobs) {
" $($j.Name) [$(Get-Prop $j 'JobType')]"
Show-LinkMembers $j $SchedulePattern ' '
foreach ($source in 'ScheduleOptions', 'GetScheduleOptions()', 'Get-VBRJobScheduleOptions') {
try {
$opts = switch ($source) {
'ScheduleOptions' { $j.ScheduleOptions }
'GetScheduleOptions()' { $j.GetScheduleOptions() }
'Get-VBRJobScheduleOptions' { Get-VBRJobScheduleOptions -Job $j -WarningAction SilentlyContinue }
}
if ($null -eq $opts) { " ${source}: <null>"; continue }
" ${source} ($($opts.GetType().FullName)):"
foreach ($p in $opts.PSObject.Properties) {
$value = try { $p.Value } catch { "<error: $($_.Exception.Message)>" }
$type = if ($null -eq $value) { 'null' } else { $value.GetType().Name }
if ($value -is [string] -or $value -is [ValueType] -or $null -eq $value) {
' {0} ({1}) = {2}' -f $p.Name, $type, (Format-Value $value)
} else {
' {0} ({1})' -f $p.Name, $type
}
}
} catch {
" ${source}: not available: $($_.Exception.Message)"
}
}
''
}
} | Out-String -Width 300 } | Out-String -Width 300
$report $report
+1 -1
View File
@@ -22,7 +22,7 @@ param(
[Parameter(Mandatory)][string]$ApiUrl, [Parameter(Mandatory)][string]$ApiUrl,
[Parameter(Mandatory)][string]$Tenant, [Parameter(Mandatory)][string]$Tenant,
[string]$SourceSystem = 'Veeam B&R', [string]$SourceSystem = 'Veeam B&R',
[ValidateRange(5, 1440)][int]$IntervalMinutes = 15, [ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
[ValidateRange(1, 720)][int]$LookbackHours = 48, [ValidateRange(1, 720)][int]$LookbackHours = 48,
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard'), [string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard'),
# Defaults to the machine-wide PowerShell 7 install # Defaults to the machine-wide PowerShell 7 install
+2 -2
View File
@@ -20,7 +20,7 @@ account can't run it. The installer checks for this before changing anything.
It prompts for the key, then: It prompts for the key, then:
- writes `C:\ProgramData\BackupDashboard\veeam-collector.json` (readable only by SYSTEM and Administrators) - writes `C:\ProgramData\BackupDashboard\veeam-collector.json` (readable only by SYSTEM and Administrators)
- copies the script to `C:\Program Files\BackupDashboard\` - copies the script to `C:\Program Files\BackupDashboard\`
- registers the scheduled task **Backup Dashboard - Veeam collector** (SYSTEM, every 15 minutes, - registers the scheduled task **Backup Dashboard - Veeam collector** (SYSTEM, every 60 minutes,
run by `C:\Program Files\PowerShell\7\pwsh.exe`; use `-PwshPath` if it's installed elsewhere) run by `C:\Program Files\PowerShell\7\pwsh.exe`; use `-PwshPath` if it's installed elsewhere)
3. Preview the payloads, then backfill 30 days so the history view is populated straight away 3. Preview the payloads, then backfill 30 days so the history view is populated straight away
(from `pwsh`): (from `pwsh`):
@@ -40,7 +40,7 @@ delete the scheduled task and both `BackupDashboard` folders.
| Job | Job name. The job's GUID is the ID, so renaming a job in Veeam doesn't duplicate it | | Job | Job name. The job's GUID is the ID, so renaming a job in Veeam doesn't duplicate it |
| Status | Session result (Success / Warning / Failed). Any state other than Stopped is **Running**. A session that stopped with no result (e.g. cancelled) is reported as **Warning** | | Status | Session result (Success / Warning / Failed). Any state other than Stopped is **Running**. A session that stopped with no result (e.g. cancelled) is reported as **Warning** |
| Started / finished | Session `CreationTime` / `EndTime` | | Started / finished | Session `CreationTime` / `EndTime` |
| Next scheduled | Job's next run. Sent as empty for disabled jobs and chained ("run after") jobs, so those are never flagged Overdue | | Next scheduled | Job's next run, from the live schedule (`GetScheduleOptions()`). Sent as empty, so never flagged Overdue, for disabled jobs, chained ("run after") jobs, and immediate-mode backup copy jobs, which run whenever their source does |
| Message | Session reason, each VM or object that didn't succeed, and warning/error lines from the session log | | Message | Session reason, each VM or object that didn't succeed, and warning/error lines from the session log |
| Size | `BackupStats.BackupSize` | | Size | `BackupStats.BackupSize` |
+16 -5
View File
@@ -3,7 +3,7 @@
Reports Veeam Backup & Replication job results to the Backup Dashboard. Reports Veeam Backup & Replication job results to the Backup Dashboard.
.DESCRIPTION .DESCRIPTION
Runs on the VBR server, normally as a SYSTEM scheduled task every 15 minutes Runs on the VBR server, normally as a SYSTEM scheduled task every 60 minutes
(see Install-VeeamCollector.ps1). For every backup, replica and backup copy job it sends: (see Install-VeeamCollector.ps1). For every backup, replica and backup copy job it sends:
- the job's next scheduled run (drives the dashboard's Overdue flag), and - the job's next scheduled run (drives the dashboard's Overdue flag), and
- every session from the last LookbackHours that the dashboard hasn't already received - every session from the last LookbackHours that the dashboard hasn't already received
@@ -85,15 +85,26 @@ function Import-Veeam {
function Get-NextRun($Job) { function Get-NextRun($Job) {
try { try {
if (-not $Job.IsScheduleEnabled) { return $null } if (-not $Job.IsScheduleEnabled) { return $null }
$raw = $Job.ScheduleOptions.NextRun # Immediate-mode backup copy jobs run whenever their source job does. Their schedule still
# carries a NextRun, but it's a stale date (months old on VBR 13) that would show as Overdue.
if ($Job.IsContinuous) { return $null }
# The ScheduleOptions property is a stale snapshot on VBR 13 (NextRun empty, last run years
# old); GetScheduleOptions() reads the live schedule. Fall back for versions without it.
$options = $null
try { $options = $Job.GetScheduleOptions() } catch { }
if (-not $options) { $options = $Job.ScheduleOptions }
$raw = $options.NextRun
if (-not $raw) { return $null } # e.g. chained jobs that run "after" another job if (-not $raw) { return $null } # e.g. chained jobs that run "after" another job
if ($raw -is [datetime]) { return ConvertTo-IsoTime $raw } if ($raw -is [datetime]) { return ConvertTo-IsoTime $raw }
# Older versions return a string formatted with the server's culture, so parse with it # A string formatted by VBR ("2026-10-09 12:15:00 AM" on VBR 13, en-CA). Try the server's
# culture first, then the invariant one
foreach ($culture in [Globalization.CultureInfo]::CurrentCulture, [Globalization.CultureInfo]::InvariantCulture) {
$parsed = [datetime]::MinValue $parsed = [datetime]::MinValue
if ([datetime]::TryParse([string]$raw, [Globalization.CultureInfo]::CurrentCulture, if ([datetime]::TryParse([string]$raw, $culture, [Globalization.DateTimeStyles]::None, [ref]$parsed)) {
[Globalization.DateTimeStyles]::None, [ref]$parsed)) {
return ConvertTo-IsoTime $parsed return ConvertTo-IsoTime $parsed
} }
}
Write-Log "Couldn't parse next run '$raw' for job '$($Job.Name)'" 'WARN' Write-Log "Couldn't parse next run '$raw' for job '$($Job.Name)'" 'WARN'
} catch { } catch {
Write-Log "Couldn't read schedule for job '$($Job.Name)': $(Get-ErrorText $_)" 'WARN' Write-Log "Couldn't read schedule for job '$($Job.Name)': $(Get-ErrorText $_)" 'WARN'
+17 -7
View File
@@ -12,12 +12,22 @@ function New-StableGuid([string]$Seed) {
} }
$notSet = Get-Date '1900-01-01' $notSet = Get-Date '1900-01-01'
function New-MockJob($Name, $Type, $NextRun, [bool]$Enabled = $true) { # As on VBR 13: the ScheduleOptions property is a stale snapshot with an empty NextRun, and the live
[pscustomobject]@{ # schedule (NextRun as a "2026-10-09 12:15:00 AM" string) comes from GetScheduleOptions().
# -Continuous: an immediate-mode copy job, whose live NextRun is a stale date in the past.
function New-MockJob($Name, $Type, $NextRun, [bool]$Enabled = $true, [switch]$Continuous) {
$job = [pscustomobject]@{
Id = New-StableGuid $Name; Name = $Name; JobType = $Type Id = New-StableGuid $Name; Name = $Name; JobType = $Type
IsScheduleEnabled = $Enabled IsScheduleEnabled = $Enabled
ScheduleOptions = [pscustomobject]@{ NextRun = $NextRun } IsContinuous = [bool]$Continuous
ScheduleOptions = [pscustomobject]@{ NextRun = '' }
LiveNextRun = $NextRun
} }
$job | Add-Member ScriptMethod GetScheduleOptions { [pscustomobject]@{ NextRun = $this.LiveNextRun } }
$job
}
function Format-VbrTime([datetime]$Value) {
$Value.ToString('yyyy-MM-dd h:mm:ss tt', [Globalization.CultureInfo]::InvariantCulture)
} }
function New-MockSession($Job, $Start, $End, $State, $Result, $Reason = '', $Tasks = @(), $LogRecords = @(), $Size = 0) { function New-MockSession($Job, $Start, $End, $State, $Result, $Reason = '', $Tasks = @(), $LogRecords = @(), $Size = 0) {
@@ -40,13 +50,13 @@ function New-MockTask($Name, $Status, $Reason = '') {
} }
$global:MockJobs = @( $global:MockJobs = @(
(New-MockJob 'File Server – Nightly' 'Backup' $now.Date.AddDays(1).AddHours(22)), (New-MockJob 'File Server – Nightly' 'Backup' (Format-VbrTime $now.Date.AddDays(1).AddMinutes(15))),
# Older VBR versions return NextRun as a culture-formatted string # Some versions return NextRun as a DateTime
(New-MockJob 'SQL01 – Logs' 'Backup' $now.AddMinutes(40).ToString()), (New-MockJob 'SQL01 – Logs' 'Backup' $now.AddMinutes(40)),
(New-MockJob 'Offsite Copy' 'BackupSync' $null), (New-MockJob 'Offsite Copy' 'BackupSync' $null),
(New-MockJob 'Old Exchange' 'Backup' $now.AddDays(1) $false), (New-MockJob 'Old Exchange' 'Backup' $now.AddDays(1) $false),
# VBR 12+ backup copy job (immediate mode, no schedule): the "policy" job Get-VBRJob returns # VBR 12+ backup copy job (immediate mode, no schedule): the "policy" job Get-VBRJob returns
(New-MockJob 'Cloud Copy' 'SimpleBackupCopyPolicy' $null) (New-MockJob 'Cloud Copy' 'SimpleBackupCopyPolicy' (Format-VbrTime $now.AddMonths(-6)) -Continuous)
) )
$fs, $sql, $copy, $old, $cloud = $global:MockJobs $fs, $sql, $copy, $old, $cloud = $global:MockJobs
@@ -0,0 +1,103 @@
<#
.SYNOPSIS
Read-only dump of what the Windows Server Backup cmdlets return, for troubleshooting the collector.
.DESCRIPTION
The collector reads properties of Get-WBJob / Get-WBSummary results whose names and time zones
aren't fully documented. This prints those objects' real structure (property names, types and
values, a few levels deep), plus recent events from the Microsoft-Windows-Backup log, whose
timestamps are reliable local times to compare against. Nothing is changed. Output is also
saved to a text file.
.EXAMPLE
.\Get-WsbCollectorDiagnostics.ps1
#>
[CmdletBinding()]
param(
[int]$Jobs = 3,
[int]$Events = 15,
[string]$OutFile = (Join-Path $env:TEMP 'wsb-collector-diagnostics.txt')
)
$ErrorActionPreference = 'Stop'
$MaxDepth = 6
if (-not (Get-Command Get-WBJob -ErrorAction SilentlyContinue)) {
Import-Module WindowsServerBackup
}
function Test-Scalar($Value) {
return $null -eq $Value -or $Value -is [string] -or $Value -is [ValueType] -or $Value -is [guid]
}
# Recursive "name (type) = value" listing. Collections show their first 3 items.
function Show-Object($Value, [string]$Indent = ' ', [int]$Depth = 0) {
if (Test-Scalar $Value) { return }
if ($Depth -ge $MaxDepth) { "$Indent..."; return }
if ($Value -is [Collections.IEnumerable]) {
$items = @($Value)
"$Indent[$($items.Count) item(s)]"
$i = 0
foreach ($item in $items | Select-Object -First 3) {
"$Indent[$i] ($($item.GetType().FullName))$(if (Test-Scalar $item) { " = $item" })"
Show-Object $item "$Indent " ($Depth + 1)
$i++
}
return
}
foreach ($p in $Value.PSObject.Properties) {
try { $v = $p.Value } catch { "$Indent$($p.Name) = <error: $($_.Exception.Message)>"; continue }
$type = if ($null -eq $v) { 'null' } else { $v.GetType().FullName }
if (Test-Scalar $v) {
"$Indent$($p.Name) ($type) = $v"
} else {
$text = [string]$v
if ($text.Length -gt 100) { $text = $text.Substring(0, 97) + '...' }
"$Indent$($p.Name) ($type) = $text"
Show-Object $v "$Indent " ($Depth + 1)
}
}
}
function Show-Section([string]$Title, [scriptblock]$Get) {
"== $Title"
try {
$result = @(& $Get)
if ($result.Count -eq 0 -or ($result.Count -eq 1 -and $null -eq $result[0])) { ' (nothing returned)' }
$n = 0
foreach ($item in $result) {
if ($null -eq $item) { continue }
" [$n] ($($item.GetType().FullName))"
Show-Object $item ' '
$n++
}
} catch {
" error: $($_.Exception.Message)"
}
''
}
$report = & {
"Generated $(Get-Date -Format o) on $env:COMPUTERNAME, PowerShell $($PSVersionTable.PSVersion)"
"Local time zone: $([TimeZoneInfo]::Local.Id), UTC offset now $([TimeZoneInfo]::Local.GetUtcOffset((Get-Date)))"
''
Show-Section 'Get-WBSummary' { Get-WBSummary }
Show-Section 'Get-WBJob (in progress)' { Get-WBJob }
Show-Section "Get-WBJob -Previous $Jobs" { Get-WBJob -Previous $Jobs }
Show-Section 'Get-WBPolicy | Get-WBSchedule' {
$policy = Get-WBPolicy
if ($policy) { Get-WBSchedule -Policy $policy } else { '(no scheduled backup policy)' }
}
"== Microsoft-Windows-Backup event log (last $Events, local times)"
try {
Get-WinEvent -LogName 'Microsoft-Windows-Backup' -MaxEvents $Events |
Select-Object TimeCreated, Id, LevelDisplayName, @{ n = 'Message'; e = { ($_.Message -split "`r?`n")[0] } } |
Format-Table -AutoSize -Wrap | Out-String -Width 300
} catch {
" error: $($_.Exception.Message)"
}
} | Out-String -Width 300
$report
$report | Set-Content -Path $OutFile -Encoding UTF8
Write-Host "Saved to $OutFile"
+89
View File
@@ -0,0 +1,89 @@
<#
.SYNOPSIS
Installs the Windows Server Backup collector as a SYSTEM scheduled task.
.DESCRIPTION
- Copies Send-WsbBackupStatus.ps1 to Program Files (only admins can modify it, which
matters because it runs as SYSTEM).
- Writes the config to ProgramData\BackupDashboard and restricts that folder to SYSTEM and
Administrators, since it contains the API key.
- Registers a scheduled task that runs every IntervalMinutes under Windows PowerShell 5.1
(powershell.exe), which the WindowsServerBackup module targets.
Safe to re-run to change settings or update the script. Can sit alongside the other collectors.
.EXAMPLE
.\Install-WsbCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
(prompts for the API key)
#>
#Requires -RunAsAdministrator
[CmdletBinding()]
param(
[Parameter(Mandatory)][string]$ApiUrl,
[Parameter(Mandatory)][string]$Tenant,
# Name shown on the dashboard; defaults to "<server> - Windows Server Backup"
[string]$JobName,
[string]$SourceSystem = 'Windows Server Backup',
[ValidateRange(5, 1440)][int]$IntervalMinutes = 60,
[ValidateRange(1, 720)][int]$LookbackHours = 48,
[string]$InstallDir = (Join-Path $env:ProgramFiles 'BackupDashboard')
)
$ErrorActionPreference = 'Stop'
$TaskName = 'Backup Dashboard - WSB collector'
$DataDir = Join-Path $env:ProgramData 'BackupDashboard'
$ConfigPath = Join-Path $DataDir 'wsb-collector.json'
$WindowsPowerShell = Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\powershell.exe'
# Check Windows Server Backup is here before changing anything
if (-not (Get-Module -ListAvailable -Name WindowsServerBackup)) {
throw 'The WindowsServerBackup PowerShell module is not available. Install the Windows Server Backup feature (Install-WindowsFeature Windows-Server-Backup).'
}
if ($ApiUrl -notmatch '^https://') {
Write-Warning 'ApiUrl is not HTTPS: the API key would be sent in clear text.'
}
# Read the key as a SecureString so it isn't echoed to the console or kept in shell history
$secure = Read-Host -AsSecureString 'API key (from: python manage.py create-key <name>)'
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
try { $apiKey = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) }
finally { [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr) }
if (-not $apiKey.StartsWith('bdk_')) { throw "That doesn't look like a dashboard API key (expected 'bdk_...')." }
# Config folder: SYSTEM + Administrators only (SIDs so this works on non-English Windows)
New-Item -ItemType Directory -Path $DataDir -Force | Out-Null
& icacls.exe $DataDir /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' | Out-Null
if ($LASTEXITCODE) { throw "icacls failed on $DataDir" }
$config = [ordered]@{
ApiUrl = $ApiUrl.TrimEnd('/')
ApiKey = $apiKey
Tenant = $Tenant
SourceSystem = $SourceSystem
LookbackHours = $LookbackHours
}
if ($JobName) { $config.JobName = $JobName }
$config | ConvertTo-Json | Set-Content -Path $ConfigPath -Encoding UTF8
New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null
$scriptPath = Join-Path $InstallDir 'Send-WsbBackupStatus.ps1'
Copy-Item -Path (Join-Path $PSScriptRoot 'Send-WsbBackupStatus.ps1') -Destination $scriptPath -Force
Unblock-File -Path $scriptPath
$action = New-ScheduledTaskAction -Execute $WindowsPowerShell `
-Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$scriptPath`""
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date).AddMinutes(1) `
-RepetitionInterval (New-TimeSpan -Minutes $IntervalMinutes)
$principal = New-ScheduledTaskPrincipal -UserId 'S-1-5-18' -LogonType ServiceAccount -RunLevel Highest
$settings = New-ScheduledTaskSettingsSet -ExecutionTimeLimit (New-TimeSpan -Minutes 30) `
-MultipleInstances IgnoreNew -StartWhenAvailable
Register-ScheduledTask -TaskName $TaskName -Action $action -Trigger $trigger -Principal $principal `
-Settings $settings -Description 'Sends Windows Server Backup results to the Backup Dashboard' -Force | Out-Null
Write-Host "Installed. Config: $ConfigPath"
Write-Host "Task '$TaskName' runs every $IntervalMinutes minutes as SYSTEM, using Windows PowerShell."
Write-Host ''
Write-Host 'Next steps:'
Write-Host " 1. Preview what will be sent: & '$WindowsPowerShell' -File '$scriptPath' -DryRun"
Write-Host " 2. Backfill 30 days of history: & '$WindowsPowerShell' -File '$scriptPath' -LookbackHours 720"
Write-Host " 3. Check the log: Get-Content '$DataDir\wsb-collector.log' -Tail 20"
+69
View File
@@ -0,0 +1,69 @@
# Windows Server Backup collector
A PowerShell script that runs on each server using Windows Server Backup (WSB) and reports its
backups to the dashboard. Each server appears as one job, named `<SERVER> - Windows Server Backup`
by default.
> **Checked against:** Server 2019 Hyper-V host backing up VMs, successful backups only.
> Not yet seen on a real server: a failed backup, a failed VM or volume, or a backup in progress.
> Those paths are written to the same object model, but if one looks wrong, run
> `Get-WsbCollectorDiagnostics.ps1` while it's in that state and send the output.
## Install (on the server, as an administrator)
Requires the Windows Server Backup feature with its PowerShell module
(`Install-WindowsFeature Windows-Server-Backup`). Runs under Windows PowerShell 5.1
(`powershell.exe`), which that module targets.
1. On the dashboard host, create a key for this server:
```bash
python manage.py create-key wsb-acme-dc01
```
2. Copy this folder to the server and run:
```powershell
.\Install-WsbCollector.ps1 -ApiUrl https://backups.example.com -Tenant "Acme Corp"
```
Add `-JobName "Domain Controller"` to choose the name shown on the dashboard. It prompts for the
key, then:
- writes `C:\ProgramData\BackupDashboard\wsb-collector.json` (readable only by SYSTEM and Administrators)
- copies the script to `C:\Program Files\BackupDashboard\`
- registers the scheduled task **Backup Dashboard - WSB collector** (SYSTEM, every 60 minutes)
3. Preview the payloads, then backfill 30 days so the history view is populated straight away:
```powershell
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-WsbBackupStatus.ps1' -DryRun
powershell.exe -File 'C:\Program Files\BackupDashboard\Send-WsbBackupStatus.ps1' -LookbackHours 720
```
It can be installed alongside the other collectors: it uses its own config, state, log and task.
Re-run the installer to change settings or update the script.
## What it sends
| Dashboard | Windows Server Backup |
|---|---|
| Company | `Tenant` from the config |
| Job | One per server, ID `wsb:<computer name>`, so changing `JobName` renames rather than duplicates it |
| Run | One per backup job from `Get-WBJob`, identified by its start time (WSB jobs have no ID). Restores are ignored. `details.items` lists what was backed up: the VMs for a Hyper-V backup, otherwise volumes |
| Status | Error HRESULT → **Failed**. Success, but a detailed error or a failed item (e.g. one volume) → **Warning**. Cancelled or unrecognised → **Warning**. In progress → **Running** |
| Started / finished | Job start / end time. WSB gives these as local-time text to the minute, in the server's date format; the collector parses them with that format |
| Next scheduled | `Get-WBSummary`'s `NextBackupTime`. Empty when there's no scheduled backup |
| Message | Error description, error codes (as `0x8XXXXXXX`), failed VMs/volumes, and the failure log path |
| Size | Bytes processed across the backup's items |
Each run covers the last 48 hours (`LookbackHours`). Runs already delivered are recorded in
`wsb-collector.state.json`.
## Troubleshooting
- Log: `C:\ProgramData\BackupDashboard\wsb-collector.log`. Exit codes: `0` OK, `1` some runs
failed to send (retried next run), `2` setup failed (config or WindowsServerBackup module).
- Wrong status, times or messages: run `.\Get-WsbCollectorDiagnostics.ps1` (read-only). It prints
the real structure of what the cmdlets return, plus recent `Microsoft-Windows-Backup` events to
compare times against.
## Testing without Windows Server Backup
```powershell
. ./tests/MockWsb.ps1 # or: . ./tests/MockWsb.ps1 -WithRunningJob
./Send-WsbBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
```
+395
View File
@@ -0,0 +1,395 @@
<#
.SYNOPSIS
Reports Windows Server Backup (WSB) results to the Backup Dashboard.
.DESCRIPTION
Runs on a server with the Windows Server Backup feature, normally as a SYSTEM scheduled task
every 60 minutes (see Install-WsbCollector.ps1). The server appears on the dashboard as one job.
Each run sends:
- the next scheduled backup (Get-WBSummary's NextBackupTime), which drives the dashboard's
Overdue flag, and
- every backup from the last LookbackHours that the dashboard hasn't already received in its
final state. A backup in progress is sent as Running until it finishes.
WSB jobs have no ID, so a run is identified by its start time. Already-reported runs are
tracked in a state file, and the ingest API is idempotent, so a failed or repeated run is
harmless: anything not confirmed is retried next time.
Written for Windows PowerShell 5.1, which the WindowsServerBackup module targets.
.PARAMETER ConfigPath
JSON config with ApiUrl, ApiKey and Tenant (optional: JobName, SourceSystem, LookbackHours).
.PARAMETER LookbackHours
Overrides the config value. Use 720 once to backfill 30 days of history.
.PARAMETER DryRun
Print the payloads instead of sending them. Doesn't touch the state or log files.
.EXAMPLE
.\Send-WsbBackupStatus.ps1 -DryRun
.EXAMPLE
.\Send-WsbBackupStatus.ps1 -LookbackHours 720
#>
[CmdletBinding()]
param(
[string]$ConfigPath = (Join-Path $env:ProgramData 'BackupDashboard\wsb-collector.json'),
[int]$LookbackHours = 0,
[switch]$DryRun
)
$ErrorActionPreference = 'Stop'
$DataDir = Split-Path -Parent $ConfigPath
$LogPath = Join-Path $DataDir 'wsb-collector.log'
$StatePath = Join-Path $DataDir 'wsb-collector.state.json'
$MaxLogBytes = 5MB
# Get-WBJob -Previous accepts at most 100. That covers a 30-day backfill at up to 3 backups a day;
# servers backing up more often only get their latest 100 runs backfilled.
$MaxPreviousJobs = 100
$Invariant = [Globalization.CultureInfo]::InvariantCulture
# ---------- Helpers ----------
function Write-Log {
param([string]$Message, [string]$Level = 'INFO')
$line = '{0} [{1}] {2}' -f (Get-Date).ToString('yyyy-MM-dd HH:mm:ss'), $Level, $Message
Write-Host $line
if ($DryRun) { return }
if ((Test-Path $LogPath) -and (Get-Item $LogPath).Length -gt $MaxLogBytes) {
Move-Item $LogPath "$LogPath.1" -Force
}
Add-Content -Path $LogPath -Value $line -Encoding UTF8
}
function Get-ErrorText($ErrorRecord) {
# Windows PowerShell puts the HTTP response body (e.g. the API's 422 explanation) here
if ($ErrorRecord.ErrorDetails -and $ErrorRecord.ErrorDetails.Message) {
return "$($ErrorRecord.Exception.Message) $($ErrorRecord.ErrorDetails.Message)"
}
return $ErrorRecord.Exception.Message
}
# The API needs an explicit UTC offset. DateTimeOffset keeps a Local/Unspecified time as local and a
# Utc time as +00:00.
function ConvertTo-IsoTime($Value) {
if ($null -eq $Value) { return $null }
$dt = [datetime]$Value
if ($dt.Year -le 1900) { return $null } # "not set" (DateTime.MinValue and friends)
return ([DateTimeOffset]$dt).ToString("yyyy-MM-dd'T'HH:mm:sszzz", $Invariant)
}
# First property of $Object, out of $Names, that exists and isn't empty. The WSB object model isn't
# fully documented, so reads go through this rather than assuming one spelling.
function Get-FirstValue($Object, [string[]]$Names) {
if ($null -eq $Object) { return $null }
foreach ($name in $Names) {
try {
$prop = $Object.PSObject.Properties[$name]
if (-not $prop) { continue }
$value = $prop.Value
# Check emptiness by type: casting to [string] would make a one-item list of objects ''
if ($null -eq $value) { continue }
if ($value -is [string] -and $value -eq '') { continue }
if ($value -is [Collections.ICollection] -and $value.Count -eq 0) { continue }
return $value
} catch { }
}
return $null
}
function Import-Wsb {
if (Get-Command Get-WBJob -ErrorAction SilentlyContinue) { return } # already loaded
# Present only when the Windows Server Backup feature (with its command-line tools) is installed
Import-Module WindowsServerBackup
}
# ---------- Jobs ----------
# WBJob times are strings in local time, to the minute, formatted with the current culture
# ("10/6/2026 9:00 PM" on en-US). Parse with that same culture: PowerShell's [datetime] cast would
# use the invariant (US) format and misread dd/MM dates. Empty while not set (e.g. EndTime of a
# running backup).
function Get-JobTime($Job, [string]$Field) {
$value = Get-FirstValue $Job @($Field)
if ($null -eq $value) { return $null }
if ($value -is [datetime]) {
$dt = $value
} else {
$dt = [datetime]::MinValue
if (-not [datetime]::TryParse([string]$value, [Globalization.CultureInfo]::CurrentCulture,
[Globalization.DateTimeStyles]::AssumeLocal, [ref]$dt)) {
Write-Log "Couldn't parse $Field '$value'" 'WARN'
return $null
}
}
if ($dt.Year -le 1900) { return $null }
return $dt
}
# HRESULTs as numbers; 0 = OK. $null when the object doesn't have the property. WSB returns them
# as strings ("0"); PowerShell's conversion also accepts hex ("0x80042302") and negative decimals.
function Get-HResult($Object, [string]$Field) {
$value = Get-FirstValue $Object @($Field)
if ($null -eq $value) { return $null }
try { return [long]$value } catch { return $null }
}
function Format-HResult([long]$Value) {
if ($Value -lt 0) { $Value += 4294967296 } # show as the familiar unsigned 0x8XXXXXXX
return '0x{0:X8}' -f $Value
}
function Get-JobItems($Job) {
return @(Get-FirstValue $Job @('JobItems', 'Items') | Where-Object { $_ })
}
function Get-ItemName($Item) {
$name = Get-FirstValue $Item @('Name', 'VolumeLabel', 'MountPath', 'Path', 'Type')
if ($name) { return [string]$name }
return 'Item'
}
# The things actually backed up. Items can nest: a Hyper-V backup has one VirtualMachineList item
# whose SubItemList holds each VM (and "Host Component").
function Get-LeafItems($Items) {
foreach ($item in $Items) {
$subItems = @(Get-FirstValue $item @('SubItemList') | Where-Object { $_ })
if ($subItems.Count) { Get-LeafItems $subItems } else { $item }
}
}
function Test-ItemFailed($Item) {
$hr = Get-HResult $Item 'HResult'
$state = [string](Get-FirstValue $Item @('State', 'JobState'))
return ($hr -and $hr -ne 0) -or $state -match 'Fail'
}
# Items that reported an error: failed leaves, or a parent that counts failed sub-items without
# saying which
function Get-FailedItems($Job) {
$items = Get-JobItems $Job
$failed = @(Get-LeafItems $items | Where-Object { Test-ItemFailed $_ })
if ($failed.Count) { return $failed }
return @($items | Where-Object {
(Test-ItemFailed $_) -or ((Get-FirstValue $_ @('SubItemFailed')) -and [long](Get-FirstValue $_ @('SubItemFailed')) -gt 0)
})
}
# Bytes processed, from the top-level items (sub-items are already counted in their parent)
function Get-BytesProcessed($Job) {
$sum = [long]0
foreach ($item in Get-JobItems $Job) {
$value = Get-FirstValue $item @('BytesProcessed')
if ($value) { $sum += [long]$value }
}
return $sum
}
function Get-RunStatus($Job) {
$state = [string](Get-FirstValue $Job @('JobState', 'State'))
if (-not (Get-JobTime $Job 'EndTime') -and $state -notmatch 'Complet|Fail|Cancel|Abort|Stop|Succe') {
return 'running'
}
if ($state -match 'Fail') { return 'failed' }
if ($state -match 'Cancel|Abort|Stop') { return 'warning' }
$hr = Get-HResult $Job 'HResult'
if ($hr -and $hr -ne 0) { return 'failed' }
# Overall success but something underneath went wrong (e.g. a volume or files skipped)
$detailed = Get-HResult $Job 'DetailedHResult'
if (($detailed -and $detailed -ne 0) -or (Get-FailedItems $Job).Count) { return 'warning' }
if (-not $state -or $state -match 'Complet|Succe') { return 'success' }
return 'warning' # finished in a state we don't recognise: surface it rather than hide it
}
function Get-RunMessage($Job, [string]$Status) {
if ($Status -eq 'running') { return '' }
$lines = New-Object 'System.Collections.Generic.List[string]'
$state = [string](Get-FirstValue $Job @('JobState', 'State'))
$description = Get-FirstValue $Job @('ErrorDescription', 'DetailedMessage')
if ($description) { $lines.Add(([string]$description).Trim()) }
$hr = Get-HResult $Job 'HResult'
$detailed = Get-HResult $Job 'DetailedHResult'
if (($hr -and $hr -ne 0) -or ($detailed -and $detailed -ne 0)) {
$codes = @()
if ($hr -and $hr -ne 0) { $codes += "error $(Format-HResult $hr)" }
if ($detailed -and $detailed -ne 0) { $codes += "detailed error $(Format-HResult $detailed)" }
$lines.Add('Windows Server Backup reported ' + ($codes -join ', ') + '.')
}
foreach ($item in Get-FailedItems $Job) {
$itemError = Get-FirstValue $item @('ErrorDescription', 'DetailedMessage')
$itemHr = Get-HResult $item 'HResult'
$subFailed = Get-FirstValue $item @('SubItemFailed')
$line = "$(Get-ItemName $item) - " # names are often volumes like "D:", so no colon
if ($itemError) { $line += ([string]$itemError).Trim() }
elseif ($itemHr) { $line += "error $(Format-HResult $itemHr)" }
elseif ($subFailed -and [long]$subFailed -gt 0) { $line += "$subFailed sub-item(s) failed" }
else { $line += [string](Get-FirstValue $item @('State', 'JobState')) }
$lines.Add($line)
}
# WSB fills in FailureLogPath even for successful backups, so only point at it when relevant
if ($Status -ne 'success') {
$failureLog = Get-FirstValue $Job @('FailureLogPath')
if ($failureLog) { $lines.Add("Failure log: $failureLog") }
}
if ($lines.Count -eq 0) {
if ($Status -eq 'success') { return 'Backup completed successfully.' }
return "Backup finished with state '$state'."
}
# WSB reports "Completed" even for a backup that failed with an error code, so only mention
# the state when it says something else (e.g. cancelled)
if ($state -and $Status -ne 'success' -and $state -notmatch 'Complet') { $lines.Insert(0, "Job state: $state") }
return ($lines -join "`n")
}
function ConvertTo-Run($Job) {
$status = Get-RunStatus $Job
$details = [ordered]@{ server = $env:COMPUTERNAME }
$items = @(Get-LeafItems (Get-JobItems $Job) | ForEach-Object { Get-ItemName $_ })
if ($items.Count) { $details.items = $items }
$state = Get-FirstValue $Job @('JobState', 'State')
if ($state) { $details.wsb_state = [string]$state }
$version = Get-FirstValue $Job @('VersionId')
if ($version) { $details.version_id = [string]$version }
# No external_id: WSB jobs have no ID, so the API keys the run on started_at
$run = [ordered]@{
status = $status
started_at = ConvertTo-IsoTime (Get-JobTime $Job 'StartTime')
message = Get-RunMessage $Job $status
details = $details
}
if ($status -ne 'running') {
$finished = ConvertTo-IsoTime (Get-JobTime $Job 'EndTime')
if ($finished) { $run.finished_at = $finished } # otherwise the API uses receipt time
$bytes = Get-BytesProcessed $Job
if ($bytes -gt 0) { $run.size_bytes = $bytes }
}
return $run
}
function Test-IsBackupJob($Job) {
# Get-WBJob also lists recoveries; only backups belong on the dashboard
$type = [string](Get-FirstValue $Job @('JobType', 'Type'))
return (-not $type) -or $type -match 'Backup'
}
function Get-NextRun {
try {
$summary = Get-WBSummary
return ConvertTo-IsoTime (Get-FirstValue $summary @('NextBackupTime'))
} catch {
Write-Log "Couldn't read the backup summary: $(Get-ErrorText $_)" 'WARN'
return $null
}
}
function Send-Report($Body) {
$json = $Body | ConvertTo-Json -Depth 6 -Compress
if ($DryRun) {
Write-Host $json
return
}
# Send bytes: Windows PowerShell would otherwise encode a string body as Latin-1
$params = @{
Uri = "$($Config.ApiUrl.TrimEnd('/'))/api/v1/ingest"
Method = 'Post'
Headers = @{ Authorization = "Bearer $($Config.ApiKey)" }
ContentType = 'application/json; charset=utf-8'
Body = [Text.Encoding]::UTF8.GetBytes($json)
TimeoutSec = 30
}
Invoke-RestMethod @params | Out-Null
}
# ---------- Main ----------
try {
$Config = Get-Content -Path $ConfigPath -Raw | ConvertFrom-Json
foreach ($key in 'ApiUrl', 'ApiKey', 'Tenant') {
if (-not $Config.$key) { throw "Config $ConfigPath is missing '$key'" }
}
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Windows Server Backup' }
$JobName = if ($Config.JobName) { [string]$Config.JobName } else { "$env:COMPUTERNAME - Windows Server Backup" }
if ($LookbackHours -le 0) {
$LookbackHours = if ($Config.LookbackHours) { [int]$Config.LookbackHours } else { 48 }
}
# Windows PowerShell 5.1 may default to TLS 1.0
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
Import-Wsb
# Run start time (ISO, as sent) -> Unix seconds, for runs already delivered in their final state
$sent = @{}
if (Test-Path $StatePath) {
$saved = Get-Content -Path $StatePath -Raw | ConvertFrom-Json
if ($saved) { foreach ($p in $saved.PSObject.Properties) { $sent[$p.Name] = [long]$p.Value } }
}
$since = (Get-Date).AddHours(-$LookbackHours)
# Get-WBJob with no arguments returns the job in progress (if any); -Previous lists finished ones
$jobs = @(@(Get-WBJob) + @(Get-WBJob -Previous $MaxPreviousJobs) | Where-Object {
$_ -and (Test-IsBackupJob $_) -and (Get-JobTime $_ 'StartTime')
})
$byStart = [ordered]@{}
foreach ($job in $jobs) {
$start = Get-JobTime $job 'StartTime'
if ($start -lt $since) { continue }
$byStart[(ConvertTo-IsoTime $start)] = $job # a job in both lists: the finished entry (later) wins
}
$pending = @($byStart.Keys | Sort-Object | Where-Object { -not $sent.ContainsKey($_) } |
ForEach-Object { $byStart[$_] })
$jobInfo = [ordered]@{
# Stable per server, so renaming the dashboard job via JobName doesn't create a new one
external_id = "wsb:$($env:COMPUTERNAME.ToLowerInvariant())"
name = $JobName
# Always sent (null when unscheduled) so the dashboard's value tracks WSB's
next_run_at = Get-NextRun
}
} catch {
Write-Log "Collector setup failed: $(Get-ErrorText $_)" 'ERROR'
exit 2
}
$failures = 0
$runsSent = 0
try {
if ($pending.Count -eq 0) {
# Keep next_run_at fresh even when nothing new ran
Send-Report ([ordered]@{ tenant = $Config.Tenant; source_system = $SourceSystem; job = $jobInfo })
}
foreach ($job in $pending) {
$run = ConvertTo-Run $job
Send-Report ([ordered]@{ tenant = $Config.Tenant; source_system = $SourceSystem; job = $jobInfo; run = $run })
$runsSent++
if ($run.status -ne 'running') {
$sent[$run.started_at] = ([DateTimeOffset](Get-JobTime $job 'StartTime')).ToUnixTimeSeconds()
}
}
} catch {
$failures++
Write-Log "Failed to report: $(Get-ErrorText $_)" 'ERROR'
}
if (-not $DryRun) {
# Forget runs that have aged out of the lookback window (plus a day of slack)
$cutoff = [DateTimeOffset]::Now.AddHours(-$LookbackHours - 24).ToUnixTimeSeconds()
$keep = [ordered]@{}
foreach ($key in $sent.Keys) {
if ($sent[$key] -ge $cutoff) { $keep[$key] = $sent[$key] }
}
try {
$keep | ConvertTo-Json -Compress | Set-Content -Path $StatePath -Encoding UTF8
} catch {
Write-Log "Couldn't save state file: $(Get-ErrorText $_)" 'WARN'
}
}
Write-Log ("Reported {0} runs, {1} failures" -f $runsSent, $failures)
if ($failures) { exit 1 }
+99
View File
@@ -0,0 +1,99 @@
# Stand-ins for the WindowsServerBackup cmdlets. Object shapes follow Get-WsbCollectorDiagnostics.ps1
# output from a real Hyper-V host (Server 2019): job times are culture-formatted local-time strings
# to the minute ("10/6/2026 9:00 PM" on en-US), HRESULTs are strings, and a Hyper-V backup has one
# VirtualMachineList item whose SubItemList holds the VMs. When nothing is running, Get-WBJob
# returns an empty job (JobType None).
#
# . ./tests/MockWsb.ps1 # nothing running
# . ./tests/MockWsb.ps1 -WithRunningJob # a backup in progress
# ./Send-WsbBackupStatus.ps1 -ConfigPath ./my-test-config.json -DryRun
param([switch]$WithRunningJob)
# Fixed to the top of the hour so repeated runs see the same jobs
$now = Get-Date -Minute 0 -Second 0 -Millisecond 0
# Same format as the real cmdlet: short date + short time in the current culture
function Format-WbTime($Value) {
if ($null -eq $Value) { return '' }
return ([datetime]$Value).ToString('g', [Globalization.CultureInfo]::CurrentCulture)
}
function New-MockItem($Name, $Type = 'Volume', [long]$Bytes = 0, [string]$HResult = '0', $ErrorDescription = '', $SubItems = $null) {
$failedSubs = @($SubItems | Where-Object { $_ -and $_.HResult -ne '0' }).Count
[pscustomobject]@{
Name = $Name
Type = $Type
State = $(if ($HResult -ne '0') { 'Failed' } else { 'Completed' })
HResult = $HResult
DetailedHResult = '0'
ErrorDescription = $ErrorDescription
BytesProcessed = $Bytes
TotalBytes = $Bytes
CurrentItem = $null
SubItemProcessed = [long]@($SubItems | Where-Object { $_ }).Count
SubItemFailed = [long]$failedSubs
TotalSubItem = [long]@($SubItems | Where-Object { $_ }).Count
SubItemList = $SubItems
}
}
function New-VmList([long]$VmBytes, [string]$VmHResult = '0', $VmError = '') {
$vms = @(
(New-MockItem 'AAOG-EDM-AD1' 'Component' $VmBytes $VmHResult $VmError),
(New-MockItem 'Host Component' 'Component' 447729)
)
New-MockItem 'VirtualMachineList' 'VirtualmachineList' ($VmBytes + 447729) -SubItems $vms
}
function New-MockJob($Type, $Start, $End, $State, [string]$HResult = '0', [string]$DetailedHResult = '0', $ErrorDescription = '', $Items = $null) {
[pscustomobject]@{
JobType = $Type
StartTime = Format-WbTime $Start
EndTime = Format-WbTime $End
JobState = $State
CurrentOperation = ''
HResult = $HResult
DetailedHResult = $DetailedHResult
ErrorDescription = $ErrorDescription
JobItems = $Items
VersionId = $(if ($End) { $Start.ToUniversalTime().ToString('MM/dd/yyyy-HH:mm', [Globalization.CultureInfo]::InvariantCulture) } else { '' })
SuccessLogPath = 'C:\Windows\Logs\WindowsServerBackup\Backup.log'
# Populated even for successful backups, as on the real server
FailureLogPath = 'C:\Windows\Logs\WindowsServerBackup\Backup_Error.log'
}
}
$global:MockWsbRunning = if ($WithRunningJob) {
@(New-MockJob 'Backup' $now.AddMinutes(-10) $null 'Running' -Items @(New-VmList 10GB))
} else {
@(New-MockJob 'None' $null $null 'Unknown')
}
$global:MockWsbPrevious = @(
(New-MockJob 'Backup' $now.AddHours(-12) $now.AddHours(-11.6) 'Completed' -Items @(New-VmList 124756706036)),
# Completed overall, but one VM failed
(New-MockJob 'Backup' $now.AddHours(-36) $now.AddHours(-35.6) 'Completed' '0' '0' '' @(
New-VmList 0 '-2147024891' 'Access is denied.')),
# Failed outright: backup target not found (0x80042302)
(New-MockJob 'Backup' $now.AddHours(-30) $now.AddHours(-29.9) 'Completed' '-2147212542' '0' `
'The backup storage location could not be found.'),
# A volume-based backup, for servers that aren't Hyper-V hosts
(New-MockJob 'Backup' $now.AddHours(-20) $now.AddHours(-19.5) 'Completed' -Items @(
(New-MockItem 'C:' 'Volume' 40GB), (New-MockItem 'D:' 'Volume' 200GB))),
(New-MockJob 'Recovery' $now.AddHours(-5) $now.AddHours(-4) 'Completed'), # a restore: must be ignored
(New-MockJob 'Backup' $now.AddDays(-5) $now.AddDays(-5).AddHours(1) 'Completed') # outside 48h lookback
)
$global:MockWsbSummary = [pscustomobject]@{
NextBackupTime = $now.Date.AddHours(21).AddDays($(if ($now.Hour -ge 21) { 1 } else { 0 }))
NumberOfVersions = 1
LastSuccessfulBackupTime = $now.AddHours(-12)
LastBackupResultHR = [uint32]0
CurrentOperationStatus = $(if ($WithRunningJob) { 'BackupInProgress' } else { 'NoOperationInProgress' })
}
# Same limit as the real cmdlet
function Get-WBJob { param([ValidateRange(1, 100)][int]$Previous) if ($PSBoundParameters.ContainsKey('Previous')) { $global:MockWsbPrevious } else { $global:MockWsbRunning } }
function Get-WBSummary { $global:MockWsbSummary }
function Get-WBPolicy { [pscustomobject]@{ PolicyName = 'Scheduled' } }
function Get-WBSchedule { param($Policy) @($global:MockWsbSummary.NextBackupTime) }
+5 -4
View File
@@ -95,8 +95,8 @@ function statusBadge(status, { asButton = false, opensDialog = false, dataset =
const runningTitle = (since) => (since ? `Running since ${formatDateTime(since)}` : null); const runningTitle = (since) => (since ? `Running since ${formatDateTime(since)}` : null);
function timeCell(date, extra = null) { function timeCell(date, { extra = null, emptyLabel = 'Never' } = {}) {
if (!date) return el('td', {}, el('span', { class: 'muted' }, 'Never')); if (!date) return el('td', {}, el('span', { class: 'muted' }, emptyLabel));
return el('td', {}, return el('td', {},
el('time', { class: 'time', datetime: date.toISOString(), title: date.toString() }, el('time', { class: 'time', datetime: date.toISOString(), title: date.toString() },
el('span', {}, formatDateTime(date), extra), el('span', {}, formatDateTime(date), extra),
@@ -112,7 +112,8 @@ function renderRow(job) {
asButton: true, opensDialog: true, dataset: { jobId: job.id }, title: runningTitle(job.runningSince), asButton: true, opensDialog: true, dataset: { jobId: job.id }, title: runningTitle(job.runningSince),
})), })),
timeCell(job.lastRun), timeCell(job.lastRun),
timeCell(job.nextRun, overdueTag)); // No next run: unscheduled, or e.g. a Veeam copy job in immediate mode
timeCell(job.nextRun, { extra: overdueTag, emptyLabel: 'Not scheduled' }));
} }
const collator = new Intl.Collator(undefined, { sensitivity: 'base', numeric: true }); const collator = new Intl.Collator(undefined, { sensitivity: 'base', numeric: true });
@@ -304,7 +305,7 @@ function renderSummarySection() {
// While a job is running, its "last run" is the previous one — say how that went // While a job is running, its "last run" is the previous one — say how that went
status === 'running' && lastStatus ? detailRow('Previous result', STATUS[lastStatus].label) : [], status === 'running' && lastStatus ? detailRow('Previous result', STATUS[lastStatus].label) : [],
detailRow('Last finished', formatDateTime(detail ? parseDate(detail.last_run_at) : job.lastRun)), detailRow('Last finished', formatDateTime(detail ? parseDate(detail.last_run_at) : job.lastRun)),
detailRow('Next scheduled', formatDateTime(detail ? parseDate(detail.next_run_at) : job.nextRun)), detailRow('Next scheduled', formatDateTime(detail ? parseDate(detail.next_run_at) : job.nextRun, 'Not scheduled')),
detailRow('Duration', formatDuration(detail?.duration_seconds)), detailRow('Duration', formatDuration(detail?.duration_seconds)),
detailRow('Size', formatBytes(detail?.size_bytes)), detailRow('Size', formatBytes(detail?.size_bytes)),
detailRow('Source', detail?.source_system))); detailRow('Source', detail?.source_system)));
+1 -1
View File
@@ -13,7 +13,7 @@ export function parseDate(value) {
return Number.isNaN(d.getTime()) ? null : d; return Number.isNaN(d.getTime()) ? null : d;
} }
export const formatDateTime = (d) => (d ? dateTimeFmt.format(d) : '—'); export const formatDateTime = (d, emptyLabel = '—') => (d ? dateTimeFmt.format(d) : emptyLabel);
export const formatTime = (d) => timeFmt.format(d); export const formatTime = (d) => timeFmt.format(d);
export function formatRelative(d, now = Date.now()) { export function formatRelative(d, now = Date.now()) {