Cache Me If You Can: Remote Monitoring macOS Services with PowerShell

I needed to monitor if a daemon was running on multiple remote macOS devices from a Windows PowerShell script. Sounds simple enough, right? The daemon in question was a Python-based agent that collects caching server data and calculates hourly data transfer rates to clients. If this agent stops running, we lose visibility into our cache server performance.

The Mission

I have six macOS servers running cache services, and each needs a monitoring agent (cache_monitor.py) running continuously to:

• Collect caching statistics every hour
• Calculate data transferred to clients between queries
• Store metrics for performance analysis

The challenge: ensure these agents are always running, and automatically restart them if they're not - all from a Windows machine using PowerShell.

The Plan : In a "nutshell"

My plan was straightforward:

1. SSH into each macOS server
2. Check if the cache_monitor process is running
3. Start it if it's not
4. Report the status

Here's where I started:

$Servers = @{
    "10.84.220.1" = "London"
    "10.85.220.1"  = "Manchester"
    "10.86.220.1" = "Birmingham"
    "10.87.225.1" = "Leeds"
    "10.88.220.1"  = "Glasgow"
    "10.89.220.1" = "Liverpool"
}

$CheckCommand = "cd /System/Volumes/Data/Users/LeeCroucher && ./cache_monitor.py start 2>&1"

Problem 1: SSH Authentication Noise

When using PuTTY's plink.exe for SSH connections, I got this mess:

Keyboard-interactive authentication prompts from server:
End of keyboard-interactive prompts from server
Service already running

The authentication messages were contaminating my output, making it impossible to detect the actual service status, so lets filter those out:

$CleanOutput = ($Output -split "`r?`n" | Where-Object { 
    $_ -notmatch "Keyboard-interactive" -and 
    $_ -notmatch "End of keyboard-interactive" -and
    $_.Trim() -ne ""
}) -join "`n"

Problem 2: False Status Reports

Running ./cache_monitor.py start when the service was already running didn't give me reliable feedback. I needed to check if the process was actually running first.

# First check if the process is already running
$CheckProcessCommand = "ps aux | grep -E 'cache_monitor\.py|1005' | grep -v grep"

$ProcessCheck = & $PuttyPath -batch -l $Username -pw $Password $RemoteHost $CheckProcessCommand 2>&1

if ($CleanProcessOutput -match "cache_monitor\.py") {
    Write-Host "✓ $ServerName: Service already running" -ForegroundColor Green
    # Extract PID and user information
} else {
    # Process not found, try to start it
    $StartCommand = "cd /System/Volumes/Data/Users/LCrouc2 && sudo ./cache_monitor.py start 2>&1"
}

Problem 3: Verification After Starting

I couldn't trust that a start command succeeded. I needed to verify the daemon was actually running after attempting to start it.

# Try to start the service
$StartOutput = & $PuttyPath -batch -l $Username -pw $Password $RemoteHost $StartCommand 2>&1

# Wait for service to initialize
Start-Sleep -Seconds 2

# Verify it's now running
$VerifyCheck = & $PuttyPath -batch -l $Username -pw $Password $RemoteHost $CheckProcessCommand 2>&1

if ($VerifyOutput -match "cache_monitor\.py") {
    Write-Host "✓ $ServerName: Service started successfully" -ForegroundColor Green
} else {
    Write-Host "✗ $ServerName: Failed to start service" -ForegroundColor Red
}

Sample CSV File Ouput:

✓ London: Service already running (found in process list)
  Process info: root 7992 0.0 0.0 410607968 1552 s001 S Thu03PM 0:13.55 /bin/bash ./cache_mo...
✓ Liverpool: Service already running (found in process list)
  Process info: root 41772 0.0 0.0 410598672 1552 ?? S 10:46AM 0:00.01 /bin/bash ./cache_mo...

This monitoring solution ensures our cache data collection never stops, giving us continuous visibility into cache server performance and client data transfer rates. The hourly metrics collected by these agents are crucial for capacity planning and performance optimization.