⚠️ Hybrid Exchange Certificate Expired

Though the title gives away the issue, its more complex than that when you are in the mist of the issue you are trying to fix, so this is something I call the "ohhh nooo" second, you know something is wrong but at the time you have that feeling you are not sure how wrong stuff is.....

Hybrid Exchange?

If you have local Exchange and Exchange Online and you require both of these environments, you have what is called a Hybrid setup, which means if you have local mailboxes on Exchange they use the Hybrid to send mail to EXO and then in EXO mails designated for Local Exchange will use the Hybrid connection to be sent locally to Exchange.

Hybrid DNS and TCP Ports?

This is usually routed via a DNS name that is secured with a certificate that identifies that Hybrid connection, which for the majority required TCP:25 and TCP:587, which TCP:587 is more the native secure port, you can also secure TCP:25 with a certificate using STARTTLS, so when the certificate has expired it can be a little shock......

Exchange Queue Checking

This is the check I do for queuing mails in Exchange, the command queries all the Mailbox servers (the one with queues) and then retrieves all the queues that have a message account above 10 from all the mailbox servers....

Get-MailboxServer | Get-Queue -Filter "MessageCount -gt 10"

This usually returns a line feed like this, which is good, nothing in the queue.....

[PS] C:\Windows\system32>Get-MailboxServer | Get-Queue -Filter "MessageCount -gt 10" [PS] C:\Windows\system32>

The ohhh-no Feeling

When you run the command above and what you see is not normal at all, you get that sinking feeling that something has gone very wrong somewhere.....

BearA\6  Retry  1443  Normal    0  <exo connector name>  BearB\6  Retry  1122  Normal    0  <exo connector name> BearC\5  Retry  1442  Normal    0  <exo connector name>  BearD\6  Retry  800   Normal    0  <exo connector name> BearE\6  Retry  2887  Normal    0  <exo connector name>

The <exo connector name> will be your actual connector name but its been changed for security here, but here you can see rather than a clean queue we have 7,694 messages queuing up, well that's not good at all

Get with the Power of Powershell

The first thing I think of is can I resume the queue with this command:

Resume-Queue BearA\6

This should then make the queue say "Active" and processing the backlog will commence, however this command did not fix that issue, the queue stayed in "Retry" - therefore someone more fundamental is wrong here, all the queues behaved like that, they would not enter "Active" but say in "retry"

Locate the real error, with LastError

So we need to look a little deeper, so we need to know what it means by retry, so lets get that from the queue with this command:

Get-Queue -Identity BearA\6 | fl

This will return a list of variables and the "LastError" is the one you are interested in.....

If you only want the see the error you can run this, that will only show you the error field.... 

Get-Queue -Identity BearA\6 | fl LastError

Now we can where the error is the actual error is this: The certificate specified in TlsCertificateName of the SendConnector could not be found.

Right, this helps a lot, this is telling me that the certificate use to send and receive e-mails does not have a valid name in the TLSCertificateName on the send connector, right something go work with

Confirm with the Send Connector

The send connector is how the e-mails are sent from your local Exchange to the outside world and if you are in a hybrid environment the EXO mailboxes as well, so we need to check that with this:

That will return all your Send Connectors, like this

Internet Mail (Primary)            {SMTP:*;10} Outbound to Office 365      {smtp:growllikeabear.mail.onmicrosoft.com;1} 

This shows we have two send connectors, so the question is which one is giving us the issues, well lets get some more PowerShell out to fix this
Get-SendConnector | fl Name,TLS*

This will tell us, that both send connectors use a TLS certificate to send to both the internet and Office 365 or EXO like this:
Name : Internet Mail (Primary) TlsDomain : TlsAuthLevel : TlsCertificateName : <I>CN=Some Random CA, C=US<S>CN=mailrelay.pokebearswithsticks.com, O=BearWorld, L=Bearville, C=US 

 Name : Outbound to Office 365 TlsDomain : mail.protection.outlook.com TlsAuthLevel : DomainValidation TlsCertificateName : <I>CN=Some Random CA, C=US<S>CN=mailrelay.pokebearswithsticks.com, O=BearWorld, L=Bearville, C=US

Right, so this is affecting both connectors and it does not like something in the TlsCertificateName so it does not like something with the certificate mailrelay.pokebearswithsticks.com

Query Certificates in Exchange

Right, that helps with the issues, next we need to see what is wrong with the certificate and why it is not working.....so lets look at those certificates with this:
Get-ExchangeCertificate | fl Subject,NotAfter,Status

That will tell us this.... 

Subject : CN=mailrelay.pokebearswithsticks.com, O=BearWorld, L=BearVille, C=UNotAfter : 06/06/2023 17:04:42  Status : DateInvalid  Subject : CN=Bear1.bear.local, C=US  NotAfter : 05/03/2029 17:48:30  Status : Valid

Well, what you do you know the certificate has expired, this now explains the issue we are having with mail queuing, the certificate being used is invalid and there cannot be used by the send connector, that also explains why the resume command fails immediately, you cannot resume with an expired certificate.

SSL Monitoring: Don't let it trip you up!

When monitoring certificates, and with hindsight, I would use an external monitor, as the record internally in your company may be served by an internal load balancer which will give you the internal certificate that will not expire at the same time as the external certificate depending on your setup, this was the case here.

mailrelay.pokebearswithsticks.com | Mar  5 17:48:30 2024 GMT  | 264        | Healthy

When the actual truth was this.....

mailrelay.pokebearswithsticks.com |Jun  6 23:45:22 2024 GMT  | 0        | Expired

Monitoring was querying the internally assigned certificate as from an internal point for view, that record was being queried by a delegated DNS zone that was in turn querying the wrong certificate, well that an oversight, but something we need to fix here later on.

Generate the new certificate

Now we have found the source of the problem. We now need to get it resolved, this will involve generating. Can you certificate with your certificate authority, this will need to be a public facing certificate authority, and cannot be an internal certificate authority.

The certificate required will handle communications with the Internet, so it’s not acceptable for you to use an internal certificate authority that’s only trusted locally in your company.

The process is the same for all the certificate authorities however, you will need to generate a CSR or a certificate signing, which will generate a public and private key, the private key, obviously stays on your server, you generated it on, and the public key is the one you sent your certificate authority in the form of a CER file

Once you had the response from your certificate authority, you will then need to complete the Sonia request by marrying the response you get from the certificate authority back to the original certificate, which should then give you a certificate with a private key, if you do not have the private key attached to the certificate, you cannot find the certificate to the connector to get mail flowing again.

Visual certificate check….

Before proceeding, it’s definitely worth with a visual indicator double check to confirm you have a certificate that’s valid, so without further ado…..

This is an example of a certificate that will not work, as the red box highlights the fact that the certificate should have a private key, notice this is blank meaning here you have the public key for the certificate but not the private key

However, this one looks healthy, here you can see you have the public and private key with the tell-tale icon here

Right now you have a valid certificate you can move on, lets get cracking.......

Install the Certificate on all Exchange servers

If you have one Exchange server then create a share another server that houses the PFX file and then you can run this.........

Import-ExchangeCertificate -Server "<server_name>" -FileData ([System.IO.File]::ReadAllBytes('<share_path>')) -PrivateKeyExportable:$true -Password (ConvertTo-SecureString -String '<pfx_password>' -AsPlainText -Force)

<server_name> is the name of the Exchange Server like BearA.bear.local

<share_path> is the path to the share like this : \\pfxshare\pfx\mailcert.pfx.pfx

<pfx_password> is the password for the PFX file like this : S3c34tPa55word

If you have more than one server then update the <server_name> for all the servers in your environment, once installed they will appear when you query the server below for the certificates.

Activate the Certificate for SMTP

Now you need to activate the certificate for SMTP services, for this you will require the thumbprint, so to get this run this command:

Get-ExchangeCertificate | fl CertificateDomains,NotBefore,NotAfter,Services,Thumbprint

This will return something like this, redacted for privacy, you will notice that the new certificate (green) has no services linked to it, and the expired certificate (red) does have SMTP linked to it, this is wrong......and needs to be fixed.

CertificateDomains : {mailrelay.pokebearswithsticks.com}
NotBefore          : 07/05/2022 17:04:42
NotAfter           : 06/06/2023 17:04:42
Services           : SMTP
Thumbprint         : 061B4B3812C91BF9727242AC4C33C3F384E45617

CertificateDomains : {mailrelay.pokebearswithsticks.com}
NotBefore          : 07/06/2023 07:23:29
NotAfter           : 07/06/2024 07:23:29
Services           :
Thumbprint         : 117958926ACC3A5571B7FF683C3C6693007ADFE9

Right so, we have the thumbprint so lets this certificate activated with this:

Note: You need to issue this to command to every Exchange server.

Enable-ExchangeCertificate -Thumbprint 117958926ACC3A5571B7FF683C3C6693007ADFE9 -Services SMTP

Then when you run the certificate check command again it will say this:

CertificateDomains : {mailrelay.pokebearswithsticks.com}
NotBefore          : 07/06/2023 07:23:29
NotAfter           : 07/06/2024 07:23:29
Services           : SMTP
Thumbprint         : 117958926ACC3A5571B7FF683C3C6693007ADFE9

Run the Hybrid Wizard

Last step, is we need to run the Hybrid wizard, which should be installed on one of your Exchange servers, this will then use the certificate and activate the mail flow both EXO and local Exchange side, which will get e-mails flowing once again.

Queue still in "Retry" - Why?

Well if after your have done all the above you notice that only once queues resumes processing, or a couple of queues resume processing and the other do not, you have missed something out, if this is your situation it will look like this:

BearA\6  Active 443  Normal    -0.72  <exo connector name> 
BearB\6  Retry  1122  Normal    0  <exo connector name>
BearC\5  Retry  1442  Normal    0  <exo connector name> 
BearD\6  Retry  800   Normal    0  <exo connector name> 
BearE\6  Retry  2887  Normal    0  <exo connector name>

In this example we have BearA as Active and dropping (as shown by the -0.72 value) and all the other queues still in Retry, this could be puzzling, but the solution is very simple, this means you have not activated the certificate from earlier on the other servers

So they cannot use it for SMTP mail flow. Therefore to make this easier you can do this, which will do it from BearA (in the case, the server already ran the command on)

$Servers = "BearB", "BearC", "BearD", "BearE"  

$Servers | ForEach {Enable-ExchangeCertificate -Thumbprint 117958926ACC3A5571B7FF683C3C6693007ADFE9 -Services "SMTP" -Server $_ -Confirm:$False -Force} 

Once the command runs you may need to resume the queue with: 

Resume-Queue Bearx\x

Then you should see the Active status and the queues should be going down, and you have fixed the issue.

BearA\6  Active 2  Normal    -0.72  <exo connector name> 
BearB\6  Active  111  Normal    -1.1  <exo connector name>
BearC\5  Active  17  Normal    -0.9  <exo connector name> 
BearD\6  Active  52   Normal    -0.81  <exo connector name>
BearE\6  Active 87  Normal    -0.94  <exo connector name>

Excellent, you have now got the mail flow back online, and by the power of grey skull you can sigh a breath of relief, however, before you back into your normal daily routine, it’s not forget about the monitoring aspect.

Monitoring for certificate expiry

It’s best not to have the same problem happen again, monitoring was in place, however, the monitoring was being done from a server that was inside the network, as mentioned earlier internally, we do some DNS delegated zones to point directly at certain exchange  servers - this caused the certificate reported in the monitoring to not be the same certificate that would be used outside the company.

This is also an external servers, so monitoring an external services from inside your own network, especially if you have internal records that differ from their external counterparts, he’s not a very good idea, also, if you wanted to from inside your own Nettwerk, you usually have to rely on a proxy server to get out to the Internet, which is again some thing people coming from outside your net work won’t do.

There are many services that will offer you external monitoring, however, the key bit here is not only getting monitoring available, but being able to monitor your SSL certificate status, or more specifically when that certificate is going to expire.
Previous Post Next Post

Ω†Ω…ΩˆΨ°Ψ¬ Ψ§Ω„Ψ§ΨͺΨ΅Ψ§Ω„