This morning, I discovered that all six of our Mac mini Content Cache servers were showing as unavailable. These servers had been running smoothly, serving cached content to our network, but suddenly all of them were failing with identical errors. What followed was a deep investigation into what appeared to be a mysterious failure affecting our entire caching infrastructure.
Initial Discovery
When checking the status of our Content Cache servers, I was greeted with an abnormal sight:
AssetCacheManagerUtil status
The output showed:
PublicAddress: [REDACTED]
RegistrationError: 500 Internal Server Error
RegistrationResponseCode: 500
RegistrationStarted: 2025-11-04 02:43:10
RegistrationStatus: -1
StartupStatus: FAILED
TotalBytesReturnedToClients: 57.94 GB
TotalBytesStoredFromOrigin: 2.94 GB
The servers had clearly been working - they'd served nearly 58GB of cached content - but now they were all failing to register with Apple's services.
Troubleshooting Steps
When attempting to reactivate the caching service, I encountered a timeout error:
sudo AssetCacheManagerUtil activate
Result:
2025-11-04 11:54:03.689 AssetCacheManagerUtil[61608:1347149] Failed to activate content caching:
Error Domain=NSPOSIXErrorDomain Code=60 "Operation timed out"
UserInfo={NSLocalizedDescription=timed out waiting for Active=YES}
This timeout, combined with the 500 error, suggested either a network issue on our end or a problem with Apple's registration servers.
Network Investigation
To rule out local network issues, I began with basic connectivity tests. First, I tested if I could reach Apple's Content Cache registration endpoint:
curl -v -X POST "https://lcdn-registration.apple.com/lcdn/register"
Surprisingly, this returned:
< HTTP/2 400
< server: Apple
< content-type: text/plain; charset=UTF-8
< content-length: 22
MALFORMED_REQUEST_BODY
This was actually good news! The 400 error with "MALFORMED_REQUEST_BODY" is the expected response when hitting this endpoint with curl. It meant Apple's servers were reachable and responding.
Diving Deeper with Packet Capture
To understand what was happening during the actual Content Cache registration, I set up a packet capture:
sudo tcpdump -i any host lcdn-registration.apple.com
Then I attempted, in a new terminal to activate Content Cache in another terminal. The packet capture revealed something interesting:
- The TCP three-way handshake completed successfully
- The TLS/HTTPS connection was established
- Data was exchanged between my server and Apple's server (17.156.129.135)
- Then... complete silence for 30 seconds
- Finally, my client timed out
The critical observation was that after the initial data exchange, there were no packets from Apple's server for 30 seconds, leading to a client-side timeout.
What This Revealed
The packet capture told a clear story:
- Network connectivity: ✅ Working perfectly
- TLS/SSL handshake: ✅ Successful
- Data sent to Apple: ✅ Transmitted
- Response from Apple: ❌ Never received
This pattern is highly unusual. In normal HTTP communication, servers respond immediately - whether with success (200), client error (400), or server error (500). A 30-second silence followed by client timeout indicates the server received the request but failed to generate any response.
Isolated Troubleshooting
I attempted several fixes on one device - this is more informational as they did not fix the problem but one device I need to rule out the basics!
Step 1 : Reset the Content Cache configuration:sudo AssetCacheManagerUtil deactivate
sudo rm /Library/Preferences/com.apple.AssetCache.plist
sudo AssetCacheManagerUtil activate
Result: Same timeout error
Step 2 : Check system logs for more details:log show --predicate 'subsystem == "com.apple.AssetCache"' --last 1h
The logs confirmed repeated registration attempts, all failing with the same pattern.
Step 3 : Verify the issue across all servers:Investigation Results
After extensive testing, I concluded:
- This is not a local network issue - Connectivity to Apple's servers is working perfectly
- This is not a configuration issue - All servers fail identically, even after configuration resets
- This appears to be an Apple service issue - Their registration endpoint accepts connections but fails to respond to Content Cache registration requests
The most telling evidence is the packet capture showing successful connection establishment but no response from Apple's servers. This indicates their registration service is in a failed state where it can accept connections but cannot process Content Cache registrations.
Environment Details
- Hardware: 6x Mac mini servers
- OS Version: macOS 26.0 (Build 25A354)
- Last Known Working: October 30, 2025 (based on cache statistics)
- Failure Mode: All servers failing identically with HTTP 500 errors
Invisible outages
Unfortunately, this does not seem to be a service that appears on the iCloud health statusAnd it also does not seem to feature on the developer service status - this is because those pages are only updated when people call into report problems and it’s not stopping devices activating it’s stopping our content activating.
In this particular instance, when iOS and macOS updates on these servers, take the brunt of the damage with utilization, but unfortunately, when they’re not I the raw Internet connection was taking the hit with excessively utilized bandwidth constraints.
Apple Status Pages are no help
The Apple status pages do not focus on content cache activation servers at all the Developer status page is the one you need which is https://developer.apple.com/system-status/
Unfortunately this will tell you that all the service are online as below but this "service" is not really advertised as a service which can be affected:
Log "silent" support ticket
The only option you have at this point is to log a "silent" support ticket where Apple will never get back to you, that is now they roll, but you can do this for this example using the link: https://developer.apple.com/contact/ you will need to sign in to your Apple ID to get a email ticket logged.
I then put detailed information in that ticket about the problem and step taken to try and resolve the issue then it was submitted, this was done later on the day of the issue, the next option was to wait and hope it resolved itself by the next day (Apple do not get back to you)
When I checked the next day, I was glad tro see the problem had been resolved and all the content cache servers were back online as you can see below:
If I look at my log file I can see that the content cache service was restored at 18:00 on the same day:
2025-11-04 14:00:07,0,0,0,0
2025-11-04 15:00:08,0,0,0,0
2025-11-04 16:00:10,0,0,0,0
2025-11-04 17:00:12,0,0,0,0
2025-11-04 18:00:14,1.36,1.35,1.36,1.35
2025-11-04 19:00:16,13.88,7.36,12.52,6.01
2025-11-04 20:00:18,16.5,8.55,2.62,1.19
2025-11-04 21:00:20,16.51,8.55,.01,0
2025-11-04 22:00:22,17.92,10.83,1.41,2.28
2025-11-04 23:00:24,20.97,13.38,3.05,2.55
2025-11-05 00:00:26,29.65,14.29,8.68,.91
2025-11-05 01:00:28,31.93,16.44,2.28,2.15
2025-11-05 02:00:30,31.96,16.44,.03,0
2025-11-05 03:00:32,33.38,16.95,1.42,.51
2025-11-05 04:00:34,34.05,16.95,.67,0
2025-11-05 05:00:36,34.97,17.86,.92,.91
2025-11-05 06:00:37,36.64,19.21,1.67,1.35
2025-11-05 07:00:40,46.65,23.67,10.01,4.46Lessons Learned
- HTTP 500 errors really do mean server-side issues - Despite the temptation to troubleshoot locally, the error code was telling the truth
- Packet captures provide invaluable insights - They can definitively show whether the problem is local or remote
- Apple's infrastructure services aren't always visible on status pages - Content Cache registration failures don't appear on Apple's system status
- Multiple identical failures point to systemic issues - When six independent servers fail identically, look for common external dependencies
Conclusion
While frustrating, this investigation clearly showed that the Content Cache registration failures were due to issues on Apple's end, not our local infrastructure. The combination of HTTP 500 errors, successful network connectivity, and the telling 30-second silence in packet captures all point to a service failure at Apple.
For organizations running Content Cache servers, this serves as a reminder that even well-functioning infrastructure can fail due to external dependencies. When troubleshooting, always validate your assumptions with concrete evidence like packet captures, and don't hesitate to conclude that sometimes, the problem really is on the other end.