NDES, SCEP and no Certificates delivered on the devices......what?

 Right this is a weird one, but lets get started this is the remit for this article:

If you’re distributing certificates to managed devices in Microsoft Intune, there’s a good chance that’s it’s done through using the SCEP protocol with NDES in the background enrolling the actual certificate to the device. In the case that your organization is not used SCEP/NDES for certificate distribution,

This was the error we got:

Connection to the backend server failed. Error: (0x80072f00).

Details:

Transaction ID: {75811b00-93a0-4940-a093-ff74aae2dc68}
Session ID: {75811b00-93a0-4940-a093-ff74aae2dc68}
Published Application Name: 
Published Application ID: 
Published Application External URL: https://grizzybear.bear.clouds/
Published Backend URL: https://bearbackend.bear.clouds/
User: <Unknown>
User-Agent: profiled/1.0 CFNetwork/1240.0.4 Darwin/20.5.0
Device ID: <Not Applicable>
Token State: NotFound
Cookie State: NotFound
Client Request URL: https://bearbackend.bear.clouds/certsrv/mscep/mscep.dll?operation=PKIOperation
Backend Request URL: https://grizzlybear.bear.clouds/certsrv/mscep/mscep.dll?operation=PKIOperation
Preauthentication Flow: PassThrough
Backend Server Authentication Mode: PassThrough
State Machine State: BEHeadersReading
Response Code to Client: <Not Applicable>
Response Message to Client: <Not Applicable>
Client Certificate Issuer: <Not Found>
Response Code from Backend: <Not Applicable>
Frontend Response Location Header: <Not Applicable>
Backend Response Location Header: <Not Applicable>
Backend Request Http Verb: POST
Client Request Http Verb: POST

We also got this error, but you can ignore this error as it does not mean anything and does not affect the service at all:

{

"Metric":{

"Dimensions":{

"UploadState":"0",

"UploadStateString":"CertificateData",

"InstanceId":"95E03FDE-DD2A-4815-82B7-7CC248334F12",

"DiagnosticCode":"0x0FFFFFFF",

"DiagnosticText":"We are unable to complete your request because a server-side error occurred. Please try again. [Exception Message: \"DiagnosticException: 0x0FFFFFFF. We are unable to complete your request because a server-side error occurred. Please try again.\"] [Exception Message: \"String was not recognized as a valid DateTime.\"]"

},

"Name":"Upload_Failure",

"Value":0

}

}

We got this >> When the certificates do not distribute:

When setting up certificate distribution for managed devices with Intune, the Intune Connector software requires you to enroll a certificate to the NDES server from a given certificate template that you’ve crafted. By doing this, you should be aware of that the certificate enrolled to the server needs to be renewed on a given interval depending on your certificate template configuration.

However, in case you’re not fully aware of the complete picture for keeping certificate distribution up and running smoothly, you could very easily find yourself in a situation where the NDES server with the Intune Connector software eventually stops responding and a non-descriptive HTTP error of 500 is shown, however not for me it wasn't 

The website bound had a valid certificate so that was not the cause of the issue at all, the cause of the issue led me down a rabbit hole, or rabbit holes, for example this link was completely wrong for out use case indeed:

https://msendpointmgr.com/2020/06/15/how-to-renew-ndes-service-certificates-for-usage-with-microsoft-intune/

Creating .inf files was not the solution as this article describes, a perfect example of the issue yes, but not the fix this time, then we move on to this, which again was not helpful at all, good information however this was NOT the issue for us, these were healthy and online, so we thought:

https://docs.microsoft.com/en-us/azure/active-directory/app-proxy/active-directory-app-proxy-protect-ndes

Then there was this article which did not fix the issue but did lead me in the direction of the fix for the issue, its a helpful article but most of it is irrelevant:

https://docs.microsoft.com/en-us/troubleshoot/mem/intune/troubleshoot-scep-certificate-device-to-ndes

But if you navigate on that article to the section "HTTP Error 503" which we were not getting, under cause 4 which says:

The CAPI2 log (see Cause 2's resolution) will show errors relating to the certificate referenced by HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\MSCEP\Modules\NDESPolicy\NDESCertThumbprint being outside of the certificate's validity period.

And that got me curious why did none of the other articles mention this key at all, so I took the value NDESCertThumbprint and then used certlm.msc to find that certificate and indeed it had expired, but as this was not the certificate bound to the IIS website it was not that obvious.

Therefore I generated a new certificate, got the thumbprint of the new certificate thumbprint and then replaced the value of this key below and override it with the new thumbprint:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\MSCEP\Modules\NDESPolicy\NDESCertThumbprint 

Then I restarted the following services:

NDESConnectorSvc
WAPCUpdaterSvc
WAPCSvc

How it was fixed:

  1. Generate new certificate (as ours was expired)
  2. Get Thumbprint from new certificate from the new certificate
  3. Update that thumbprint for this key : HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\MSCEP\Modules\NDESPolicy\NDESCertThumbprint 
Note : If you renew the certificate if expired or not the thumbprint is changed, so I would image you would need to use the NDES setup to register each time the cert is changed, or get a longer running certificate - but if you do that, you need to remember to do the fix above.