kyleandClaude Opus 5.5
dd3e2fe885
On VBR 13 a job's ScheduleOptions property is a stale snapshot: NextRun
is empty (and its LatestRunLocal years old), so every Veeam job showed
no next run and could never be flagged Overdue. GetScheduleOptions() /
Get-VBRJobScheduleOptions return the live schedule ("2026-10-09
12:15:00 AM"). Use GetScheduleOptions(), falling back to the property.
Immediate-mode backup copy jobs (IsContinuous) have a live NextRun too,
but it's a stale date months in the past, which would show as Overdue;
keep reporting no next run for them.
Parsing tries the current culture, then invariant; VBR 13's format
parsed correctly under en-CA, en-US, fr-CA and en-GB. Mock updated to
VBR 13's schedule shape.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
347 lines
14 KiB
PowerShell
347 lines
14 KiB
PowerShell
<#
|
|
.SYNOPSIS
|
|
Reports Veeam Backup & Replication job results to the Backup Dashboard.
|
|
|
|
.DESCRIPTION
|
|
Runs on the VBR server, normally as a SYSTEM scheduled task every 60 minutes
|
|
(see Install-VeeamCollector.ps1). For every backup, replica and backup copy job it sends:
|
|
- the job's next scheduled run (drives the dashboard's Overdue flag), and
|
|
- every session from the last LookbackHours that the dashboard hasn't already received
|
|
in its final state. Running sessions are re-sent each time until they finish.
|
|
|
|
Already-reported sessions are tracked in a state file, and the ingest API is idempotent,
|
|
so a failed or repeated run is harmless: anything not confirmed is retried next time.
|
|
|
|
Compatible with Windows PowerShell 5.1 and PowerShell 7.
|
|
|
|
.PARAMETER ConfigPath
|
|
JSON config with ApiUrl, ApiKey and Tenant (optional: SourceSystem, LookbackHours).
|
|
|
|
.PARAMETER LookbackHours
|
|
Overrides the config value. Use 720 once to backfill 30 days of history.
|
|
|
|
.PARAMETER DryRun
|
|
Print the payloads instead of sending them. Doesn't touch the state or log files.
|
|
|
|
.EXAMPLE
|
|
.\Send-VeeamBackupStatus.ps1 -DryRun
|
|
.EXAMPLE
|
|
.\Send-VeeamBackupStatus.ps1 -LookbackHours 720
|
|
#>
|
|
[CmdletBinding()]
|
|
param(
|
|
[string]$ConfigPath = (Join-Path $env:ProgramData 'BackupDashboard\veeam-collector.json'),
|
|
[int]$LookbackHours = 0,
|
|
[switch]$DryRun
|
|
)
|
|
|
|
$ErrorActionPreference = 'Stop'
|
|
|
|
$DataDir = Split-Path -Parent $ConfigPath
|
|
$LogPath = Join-Path $DataDir 'veeam-collector.log'
|
|
$StatePath = Join-Path $DataDir 'veeam-collector.state.json'
|
|
$MaxLogBytes = 5MB
|
|
$Invariant = [Globalization.CultureInfo]::InvariantCulture
|
|
|
|
# ---------- Helpers ----------
|
|
|
|
function Write-Log {
|
|
param([string]$Message, [string]$Level = 'INFO')
|
|
$line = '{0} [{1}] {2}' -f (Get-Date).ToString('yyyy-MM-dd HH:mm:ss'), $Level, $Message
|
|
Write-Host $line
|
|
if ($DryRun) { return }
|
|
if ((Test-Path $LogPath) -and (Get-Item $LogPath).Length -gt $MaxLogBytes) {
|
|
Move-Item $LogPath "$LogPath.1" -Force
|
|
}
|
|
Add-Content -Path $LogPath -Value $line -Encoding UTF8
|
|
}
|
|
|
|
function Get-ErrorText($ErrorRecord) {
|
|
# Windows PowerShell puts the HTTP response body (e.g. the API's 422 explanation) here
|
|
if ($ErrorRecord.ErrorDetails -and $ErrorRecord.ErrorDetails.Message) {
|
|
return "$($ErrorRecord.Exception.Message) $($ErrorRecord.ErrorDetails.Message)"
|
|
}
|
|
return $ErrorRecord.Exception.Message
|
|
}
|
|
|
|
# The API needs an explicit UTC offset. Veeam returns local times, often with Kind=Unspecified,
|
|
# which DateTimeOffset treats as local, exactly what we want.
|
|
function ConvertTo-IsoTime($Value) {
|
|
if ($null -eq $Value) { return $null }
|
|
$dt = [datetime]$Value
|
|
if ($dt.Year -le 1900) { return $null } # Veeam's "not set" value (e.g. EndTime while running)
|
|
return ([DateTimeOffset]$dt).ToString("yyyy-MM-dd'T'HH:mm:sszzz", $Invariant)
|
|
}
|
|
|
|
function Import-Veeam {
|
|
if (Get-Command Get-VBRJob -ErrorAction SilentlyContinue) { return } # already loaded
|
|
if (Get-Module -ListAvailable -Name Veeam.Backup.PowerShell) {
|
|
Import-Module Veeam.Backup.PowerShell -WarningAction SilentlyContinue # VBR 11+
|
|
} else {
|
|
Add-PSSnapin VeeamPSSnapin # VBR 10 and earlier
|
|
}
|
|
}
|
|
|
|
function Get-NextRun($Job) {
|
|
try {
|
|
if (-not $Job.IsScheduleEnabled) { return $null }
|
|
# Immediate-mode backup copy jobs run whenever their source job does. Their schedule still
|
|
# carries a NextRun, but it's a stale date (months old on VBR 13) that would show as Overdue.
|
|
if ($Job.IsContinuous) { return $null }
|
|
|
|
# The ScheduleOptions property is a stale snapshot on VBR 13 (NextRun empty, last run years
|
|
# old); GetScheduleOptions() reads the live schedule. Fall back for versions without it.
|
|
$options = $null
|
|
try { $options = $Job.GetScheduleOptions() } catch { }
|
|
if (-not $options) { $options = $Job.ScheduleOptions }
|
|
$raw = $options.NextRun
|
|
if (-not $raw) { return $null } # e.g. chained jobs that run "after" another job
|
|
if ($raw -is [datetime]) { return ConvertTo-IsoTime $raw }
|
|
# A string formatted by VBR ("2026-10-09 12:15:00 AM" on VBR 13, en-CA). Try the server's
|
|
# culture first, then the invariant one
|
|
foreach ($culture in [Globalization.CultureInfo]::CurrentCulture, [Globalization.CultureInfo]::InvariantCulture) {
|
|
$parsed = [datetime]::MinValue
|
|
if ([datetime]::TryParse([string]$raw, $culture, [Globalization.DateTimeStyles]::None, [ref]$parsed)) {
|
|
return ConvertTo-IsoTime $parsed
|
|
}
|
|
}
|
|
Write-Log "Couldn't parse next run '$raw' for job '$($Job.Name)'" 'WARN'
|
|
} catch {
|
|
Write-Log "Couldn't read schedule for job '$($Job.Name)': $(Get-ErrorText $_)" 'WARN'
|
|
}
|
|
return $null
|
|
}
|
|
|
|
function Get-RunStatus($Session) {
|
|
if ([string]$Session.State -ne 'Stopped') { return 'running' }
|
|
switch ([string]$Session.Result) {
|
|
'Success' { return 'success' }
|
|
'Warning' { return 'warning' }
|
|
'Failed' { return 'failed' }
|
|
# Stopped without a result (e.g. cancelled): surface it rather than hide it
|
|
default { return 'warning' }
|
|
}
|
|
}
|
|
|
|
# Builds a readable summary: the session's reason, any task (VM/object) that didn't succeed,
|
|
# and warning/error lines from the session log.
|
|
function Get-RunMessage($Session, [string]$Status, $Tasks) {
|
|
$lines = New-Object 'System.Collections.Generic.List[string]'
|
|
try {
|
|
$reason = [string]$Session.Info.Reason
|
|
if ($reason.Trim()) { $lines.Add($reason.Trim()) }
|
|
} catch { }
|
|
|
|
foreach ($task in $Tasks) {
|
|
$taskStatus = [string]$task.Status
|
|
if ($taskStatus -and $taskStatus -ne 'Success' -and $taskStatus -ne 'InProgress' -and $taskStatus -ne 'Pending') {
|
|
$taskReason = ''
|
|
try { $taskReason = ([string]$task.Info.Reason).Trim() } catch { }
|
|
$line = "$($task.Name): $taskStatus"
|
|
if ($taskReason) { $line += " - $taskReason" }
|
|
$lines.Add($line)
|
|
}
|
|
}
|
|
|
|
if ($Status -eq 'warning' -or $Status -eq 'failed') {
|
|
try {
|
|
foreach ($record in $Session.Logger.GetLog().UpdatedRecords) {
|
|
$recordStatus = [string]$record.Status
|
|
if ($recordStatus -eq 'EWarning' -or $recordStatus -eq 'EFailed') {
|
|
$title = ([string]$record.Title).Trim()
|
|
if ($title -and -not $lines.Contains($title)) { $lines.Add($title) }
|
|
}
|
|
}
|
|
} catch { }
|
|
}
|
|
|
|
if ($lines.Count -eq 0) {
|
|
switch ($Status) {
|
|
'success' { return 'Job completed successfully.' }
|
|
'running' { return '' }
|
|
default { return "Job finished with result '$($Session.Result)'." }
|
|
}
|
|
}
|
|
return ($lines -join "`n")
|
|
}
|
|
|
|
# VBR 12+ backup copy jobs: Get-VBRJob returns the copy (policy) job, but Get-VBRBackupSession
|
|
# returns only the sessions of its hidden worker jobs, one per source job, never the policy's own.
|
|
# Each worker job records its policy in PolicyJobId, so those sessions can be filed under it.
|
|
function Get-PolicyJobId($Session, [hashtable]$KnownJobIds) {
|
|
try {
|
|
$policyId = [string]$Session.GetJob().PolicyJobId
|
|
if ($policyId -and $KnownJobIds.ContainsKey($policyId)) { return $policyId }
|
|
} catch { }
|
|
return ''
|
|
}
|
|
|
|
function ConvertTo-Run($Session, $Job) {
|
|
$status = Get-RunStatus $Session
|
|
$tasks = @()
|
|
try { $tasks = @($Session.GetTaskSessions()) } catch { }
|
|
|
|
$details = [ordered]@{
|
|
job_type = [string]$Job.JobType
|
|
vbr_server = $env:COMPUTERNAME
|
|
objects = $tasks.Count
|
|
}
|
|
$failedTasks = @($tasks | Where-Object { [string]$_.Status -eq 'Failed' }).Count
|
|
$warningTasks = @($tasks | Where-Object { [string]$_.Status -eq 'Warning' }).Count
|
|
if ($failedTasks) { $details.objects_failed = $failedTasks }
|
|
if ($warningTasks) { $details.objects_warning = $warningTasks }
|
|
try { if ($Session.IsRetryMode) { $details.retry = $true } } catch { }
|
|
# Worker session of a backup copy job: say which source job it copied
|
|
# (worker sessions are named "<copy job>\<source job>")
|
|
if ($Session.JobId.ToString() -ne $Job.Id.ToString()) {
|
|
$details.copied_from = ([string]$Session.JobName -split '\\')[-1]
|
|
}
|
|
|
|
$run = [ordered]@{
|
|
external_id = $Session.Id.ToString()
|
|
status = $status
|
|
started_at = ConvertTo-IsoTime $Session.CreationTime
|
|
message = Get-RunMessage $Session $status $tasks
|
|
details = $details
|
|
}
|
|
if ($status -ne 'running') {
|
|
$finished = ConvertTo-IsoTime $Session.EndTime
|
|
if ($finished) { $run.finished_at = $finished } # otherwise the API uses receipt time
|
|
try {
|
|
$size = [long]$Session.BackupStats.BackupSize
|
|
if ($size -gt 0) { $run.size_bytes = $size }
|
|
$dataSize = [long]$Session.BackupStats.DataSize
|
|
if ($dataSize -gt 0) { $details.data_size_bytes = $dataSize }
|
|
} catch { }
|
|
}
|
|
return $run
|
|
}
|
|
|
|
function Send-Report($Body) {
|
|
$json = $Body | ConvertTo-Json -Depth 6 -Compress
|
|
if ($DryRun) {
|
|
Write-Host $json
|
|
return
|
|
}
|
|
# Send bytes: Windows PowerShell would otherwise encode a string body as Latin-1,
|
|
# mangling non-ASCII job names (e.g. en dashes).
|
|
$params = @{
|
|
Uri = "$($Config.ApiUrl.TrimEnd('/'))/api/v1/ingest"
|
|
Method = 'Post'
|
|
Headers = @{ Authorization = "Bearer $($Config.ApiKey)" }
|
|
ContentType = 'application/json; charset=utf-8'
|
|
Body = [Text.Encoding]::UTF8.GetBytes($json)
|
|
TimeoutSec = 30
|
|
}
|
|
Invoke-RestMethod @params | Out-Null
|
|
}
|
|
|
|
# ---------- Main ----------
|
|
|
|
try {
|
|
$Config = Get-Content -Path $ConfigPath -Raw | ConvertFrom-Json
|
|
foreach ($key in 'ApiUrl', 'ApiKey', 'Tenant') {
|
|
if (-not $Config.$key) { throw "Config $ConfigPath is missing '$key'" }
|
|
}
|
|
$SourceSystem = if ($Config.SourceSystem) { [string]$Config.SourceSystem } else { 'Veeam B&R' }
|
|
if ($LookbackHours -le 0) {
|
|
$LookbackHours = if ($Config.LookbackHours) { [int]$Config.LookbackHours } else { 48 }
|
|
}
|
|
|
|
# Windows PowerShell 5.1 may default to TLS 1.0
|
|
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
|
|
|
|
Import-Veeam
|
|
|
|
# Session ID -> start time (Unix seconds), for sessions already delivered in their final state.
|
|
# Numbers rather than ISO strings: PowerShell 7's ConvertFrom-Json turns date strings into
|
|
# DateTimes and loses the offset.
|
|
$sent = @{}
|
|
if (Test-Path $StatePath) {
|
|
$saved = Get-Content -Path $StatePath -Raw | ConvertFrom-Json
|
|
if ($saved) { foreach ($p in $saved.PSObject.Properties) { $sent[$p.Name] = [long]$p.Value } }
|
|
}
|
|
|
|
$since = (Get-Date).AddHours(-$LookbackHours)
|
|
$jobs = @(Get-VBRJob -WarningAction SilentlyContinue)
|
|
|
|
$jobIds = @{}
|
|
foreach ($j in $jobs) { $jobIds[$j.Id.ToString()] = $true }
|
|
|
|
# Get-VBRBackupSession returns every session in the database, so fetch once and bucket by job
|
|
$sessionsByJob = @{}
|
|
$policyByWorker = @{} # worker job ID -> policy job ID ('' if it isn't a worker of a listed job)
|
|
foreach ($s in Get-VBRBackupSession -WarningAction SilentlyContinue) {
|
|
if ($s.CreationTime -lt $since) { continue }
|
|
$key = $s.JobId.ToString()
|
|
if (-not $jobIds.ContainsKey($key)) {
|
|
if (-not $policyByWorker.ContainsKey($key)) { $policyByWorker[$key] = Get-PolicyJobId $s $jobIds }
|
|
if (-not $policyByWorker[$key]) { continue }
|
|
$key = $policyByWorker[$key]
|
|
}
|
|
if (-not $sessionsByJob.ContainsKey($key)) { $sessionsByJob[$key] = New-Object 'System.Collections.Generic.List[object]' }
|
|
$sessionsByJob[$key].Add($s)
|
|
}
|
|
} catch {
|
|
Write-Log "Collector setup failed: $(Get-ErrorText $_)" 'ERROR'
|
|
exit 2
|
|
}
|
|
|
|
$failures = 0
|
|
$runsSent = 0
|
|
foreach ($job in $jobs) {
|
|
$jobInfo = [ordered]@{
|
|
external_id = $job.Id.ToString()
|
|
name = $job.Name
|
|
# Always sent (null when unscheduled/disabled) so the dashboard's value tracks Veeam's
|
|
next_run_at = Get-NextRun $job
|
|
}
|
|
$pending = @()
|
|
if ($sessionsByJob.ContainsKey($jobInfo.external_id)) {
|
|
$pending = @($sessionsByJob[$jobInfo.external_id] |
|
|
Where-Object { -not $sent.ContainsKey($_.Id.ToString()) } |
|
|
Sort-Object CreationTime)
|
|
}
|
|
|
|
try {
|
|
if ($pending.Count -eq 0) {
|
|
# Keep next_run_at fresh even when nothing new ran
|
|
Send-Report ([ordered]@{ tenant = $Config.Tenant; source_system = $SourceSystem; job = $jobInfo })
|
|
}
|
|
foreach ($session in $pending) {
|
|
$run = ConvertTo-Run $session $job
|
|
Send-Report ([ordered]@{ tenant = $Config.Tenant; source_system = $SourceSystem; job = $jobInfo; run = $run })
|
|
$runsSent++
|
|
if ($run.status -ne 'running') {
|
|
$sent[$run.external_id] = ([DateTimeOffset][datetime]$session.CreationTime).ToUnixTimeSeconds()
|
|
}
|
|
}
|
|
} catch {
|
|
$failures++
|
|
Write-Log "Failed to report job '$($job.Name)': $(Get-ErrorText $_)" 'ERROR'
|
|
$httpStatus = 0
|
|
try { $httpStatus = [int]$_.Exception.Response.StatusCode } catch { }
|
|
if ($httpStatus -eq 401) {
|
|
Write-Log 'API key rejected; check ApiKey in the config. Stopping.' 'ERROR'
|
|
break
|
|
}
|
|
}
|
|
}
|
|
|
|
if (-not $DryRun) {
|
|
# Forget sessions that have aged out of the lookback window (plus a day of slack)
|
|
$cutoff = [DateTimeOffset]::Now.AddHours(-$LookbackHours - 24).ToUnixTimeSeconds()
|
|
$keep = [ordered]@{}
|
|
foreach ($id in $sent.Keys) {
|
|
if ($sent[$id] -ge $cutoff) { $keep[$id] = $sent[$id] }
|
|
}
|
|
try {
|
|
$keep | ConvertTo-Json -Compress | Set-Content -Path $StatePath -Encoding UTF8
|
|
} catch {
|
|
Write-Log "Couldn't save state file: $(Get-ErrorText $_)" 'WARN'
|
|
}
|
|
}
|
|
|
|
Write-Log ("Reported {0} jobs, {1} runs, {2} failures" -f $jobs.Count, $runsSent, $failures)
|
|
if ($failures) { exit 1 }
|