🤯 Phantom ProxyAddress with EXO

Weird one the other day when trying to assign licenses to users I got the error "Non-unique proxy address in Exchange Online" as you can see below:



This left the users with no Office E5 services, but I wonder why, so lets take a look into this and get it fixed, so first you need to understand how the proxyAddress attribute is assigned in Exchange Online, for a excellent insight into this you can check out the article here

This article is helpful for the understanding but does not help you fix the issue at all, the issue here is the process called MOERA, which is defined as this, with added bears for goodness:

Microsoft Online Email Routing Address (MOERA): The address constructed from the user's userPrincipalName prefix, plus the initial domain suffix, which is automatically added to the proxyAddresses in Azure AD. For example, smtp:broken.bear@bearland.onmicrosoft.com . UserPrincipalName (UPN): The sign-in address of the user

The first clue here is linked off the UPN, which is interesting, but lets give the full story for this scenario:

  1. Active Directory domain
  2. AD-Connect using AD with Entra (previously called AAD)
  3. Exchange Hybrid mode (local and cloud version of Exchange)
  4. Mailbox migrated to Exchange online
  5. Attributes are read only as they are synced with AD
Right, so in this example we have an AD synced object which means we cannot remove attributes that are "misconfigured" as you need to do that from the AD domain, if you try to do this from Office 365 you will quite rightly get an error as below:


Therefore lets check the account using that process AD > AD-Connect > Metaverse > Entra and see where the error is, if indeed there is an error at all?

Active Directory Check

First start with the ECP of your local Exchange, find the user, for this I will use my account as an example, there I am with my Office 365 mailbox reporting correctly to local exchange:


Double click that entry and choose email address then from there you can see the SMTP addresses and the proxy address, you can see here I have my e-mail address and the routing address which is correct, so everything is fine there.

<new images required>

AD-Connect Check

Now we need to check AD-Connect and when I say AD-Connect I mean the Metaverse database, so login to your AD-Connect server and then run the executable below from the Run box

"C:\Program Files\Microsoft Azure AD Sync\UIShell\miisclient.exe"

Like this, simples:



This will display the synchronisation window, from here you will see all the syncs that have run, you need the Metaverse Search option as below:


That will show you a screen like this, where you need to select the following use the red boxes, then when you search you will notice you have a result which is shown by the green box:

  1. Search for : Person
  2. Attribute : mail
  3. Operator : equals
  4. Value : <email address of person>
  5. Search

If you double click the user you will get another window pop-up for that user, then choose the connections tab as below:



You will then see two entries in this list this is how it works:
  1. Top Entry : Cloud Account
  2. Bottom Entry : Local AD account
Right, so now you can see what Metaverse sees, which is the data being sent from your local AD to Entra, so double click the top entry and find the section for ProxyAddress like this:



You will notice in the last sync the changes are "none" so from the last sync nothing has been updated, however if you click on the button under value, you will see that no changes have been made on the last sync and all the values are correct:


Click OK to the first window, and chose the second window, then choose the bottom entry which is your local AD and do the same with that one, and this is my result:


Excellent, this is the same as the other value, so we are good, therefore so far we have proved our AD is correct and AD-Connect is correct, this means that everything should be in sync - but its not, we get a licensing error.

Deeper Dive

Now we have checked the synchronisation is all good, why do we get the error, well if you navigate to the Exchange Online management website and find the user under mailboxes like this:



If we click on the user and on the side menu choose "Manage email address types"


We we will see that in Exchange we have another <tenant>.onmicrosoft.com - where did this come from, this value is not in AD or AD-Connect but it is in Entra - what the deuce?


This is also the cause of the error, as you no longer have a unique routing address for Office 365, or more specifically Exchange Online to use, this means that something occurred with the account when it was created which mean MOERA (the Microsoft Online e-mail routing address) has added some attributes that should not be there. 

Now this is where you are between a rock and hard place as the following is now true:
  1. The additional attributes are not in AD so you cannot remove them
  2. The additional are in the cloud, but as its synced to AD you cannot remove them
Hmmmm, checkmate it would appear, well not quite, while the mailbox is synced to AD then you cannot remove the additional broken attributes, but you can make that mailbox a cloud account and then remove the attributes, or I wonder if a "unsync" and a "re-sync" would fix the issue?

Unsync and re-sync - from AD-Connect

This will need to be done from AD-Connect, so go back to that server, and open the same synchronisation manager from earlier, this time we require the Connection option then you need to double click on the AD connection, as below:

Note : You cannot amend the connections when active, if you need to, you can either wait between the synchronisation cycles or you can suspend the synchronisation while you do this with this command:

Set-ADSyncScheduler -SyncCycleEnabled $False

DO NOT FORGET to enable it again with this command:

Set-ADSyncScheduler -SyncCycleEnabled $True


Then from here you want the "Containers" option as below:



Then you will need to enter the password for the service account used for AD-Connect and once you have you will get this:



You need to make no changes, but you need to check which OU (or Organizational Units) are not syncing with your domain to Entra, here we can see that "Users" does not sync with Entra.

Note : If you sync your whole domain, you will need to create a new OU and untick that OU for this to work.

Move actual users to users OU

Now we have to move the "affected" people to that OU, I would also make a note of where they came from before, which in my example is:

Old > OU=Technical,CN=Users,DC=bear,DC=local
New > CN=Users,DC=bear,DC=local

Now you have moved the users you can wait 30 minutes for a synchronisation to run of you can use this command:

Start-ADSyncSyncCycle -PolicyType Delta

Once the changes have been synced this will remove the account the AD sync to Entra and those account will vanish along with the mailbox as that action has moved them to deleted items.

Magically make the Entra account appear

Now the accounts are no longer there in Entra or Exchange Online Admin, you need to make the account re-appear so to complete this action below, by moving the accounts back to their original locations:

Current > CN=Users,DC=bear,DC=local
New > OU=Technical,CN=Users,DC=bear,DC=local

Then you can wait 30 minutes for the synchronisation to start or you can issue this command again:

Start-ADSyncSyncCycle -PolicyType Delta

Once the synchronisation is complete the accounts should once again appear in Entra and Exchange Online, when they do, if you try to assign the licenses to them, this time they are Active and you do not get the error like you did before:



Excellent, that means that "unsync" and a "resync" fixed the issue, but why did it occur in the first place I ask, and that is not the root cause before that is the only takeaway you get from that!

Why did it happen?

Well, not that people like to read this, but this particular problem is caused by human error, when you create an account that account has a UPN, which is short for User principal name

If in the scenario you have an Active Directory domain it’s common place to use non-routable domains, for example, I use bear.local - which means I will have the ending of the UPN with the options of:

bear.local
pokebearswithsticks.con

Please do not confuse my domain name with my office 365 tenant Prefix which happens to be “bearland”

now, if I create an account with the bear.local prefix and sync it to Entra, because my non-routable domain is not in office 365 it is unable to assign me bear.local

Therefore, what will happen is in office 365 I will be assigned Lee.Croucher@bearland.onmicrosoft.com due to the incorrectly set UPN.

If this was an account that did not have a license that used exchange online then that’s not a massive problem at the moment, because MOERA has not tried to assign any address routing because it doesn’t have a license for exchange.

At this point if the creator of the account notices that mistake and fixes the UPN, then the account will change from Lee.Croucher@bearland.onmicrosoft.com to Lee.Croucher@pokebearswithsticks.com - which is absolutely what it should do.

UPN Incorrect and MOERA Issues

However, if you create that account incorrectly, and then migrate that mailbox to Exchange Online and assign it a valid Exchange license, that’s where the fun begins, because once it has a license to use exchange, MOERA will kick in, and do it thing, that will cause the incorrect addition of that funny email address we saw earlier in this post.

Now you have that additional attribute which has been created Office 365, you are not able to remove that attribute because it’s synced with your local Active Directory, and ironically, since the additional "wrong" routing address is present, when you review the status of the license you get the an error saying there is a not a unique address.

This means, all this excitement was caused because the account was created wrong to start with, that also sounds logical, but where did I go wrong?

Well when you create a new user in the domain the default ending to the user is the domain name, which for me is bear.local makes up the section after the @ in the UPN for example:

Lee.Croucher@bear.local

I have to change this to @pokebearswithsticks.com at the time of the account creation to get the correct UPN of :

Lee.Croucher@pokebearswithsticks.com

This is a visual of what I am explaining above, here the red box is the default domain name and the green box is the option you need to select manually to avoid this error occurring, as any new user will inherit the domain name:



Check your automation!

Finally, if you have any automation to crate new users, you also need to make sure this is setting the correct UPN as well, so you cannot really be running this, as the results of the command below is a UPN of bear.local@

New-ADUser lee.croucher

You would need to specify this as the command, the command below with actually set the UPN correctly whcih is how it really should be done (rather than fixing errors in code)

New-ADUser lee.croucher -UserPrincipalName lee.croucher@pokebearwithsticks.com


Address Book Policy Issues?

I can also think of address book policies causing issues here as well if they are not setup for your environment correctly, these will be found on your local exchange management website, which are under Mail Flow > email Address Policies and this is what that looks like:


These mail policies work with a priority but only if it applies to the mailbox in question, so lets take a look at the policies one by one to see if we can find anomaly's with any of these policies, so first up is the HealthMailbox.......

This policy only applies to mailboxes that start with "HealthMailbox" so that does not apply to users, so that policy is not an issue, that is below:



The next one called e-mail address policy, does apply to users, here you can see it applies to all users with Exchange mailboxes which is definitely all the users for local Exchange.....this one looks good as well, here it is:


However the last policy is a problem, this applies to all recipient types, and when you move an account from the local Exchange mailbox to a EXO mailbox the mailbox is then a remote mailbox, which no longer falls in the Exchange mailbox (as its remote)

However here you can see it applies the mail address as @bears.local to the Default policy, this should really be @pokebearswithsticks.com which is another reason why this issue is occurring.......


Local and Remote - how can you tell?

If you start a Exchange PowerShell session on the on-premises Exchange and run the command for a mailbox that is hosted on-premises you get this, which is classed as an Exchange mailbox:

Get-Mailbox lee.croucher@pokebearswithsticks.com

Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
Lee Croucher              lee.croucher           mailbear1         Unlimited


However once this mailbox is migrated to the EXO if you try the same command you get this:

Get-Mailbox lee.croucher@Severntrent.co.uk

The operation couldn't be performed because object 'lee.croucher@pokebearswithsticks.om' couldn't be found on 'overloardbear1.stwater.intra'.
    + CategoryInfo          : NotSpecified: (:) [Get-Mailbox], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : [Server=MAILBEAR1,RequestId=aa099902-3710-4369-904e-678d844afd17,TimeStamp=01/09/2023 13:06:19] [FailureCategory=Cmdlet-Manage
   mentObjectNotFoundException] C88F8BA8,Microsoft.Exchange.Management.RecipientTasks.GetMailbox
    + PSComputerName        : mailbear1.bear.local

This means in the bottom command the mailbox is no longer classed as local, once its a remote mailbox it will fall under the "Default" address policy, which means the e-mail address will change to @bear.local - causing MOERA to append more mail addresses on the EXO mailbox.

On boarding to EXO updates e-mail address?

Yes it does, while you are on-premises you are getting the e-mail address policy, which is set correctly, however when you migrate to Exchange Online you are a remote mailbox and fall under the default policy.

Once your migration completes your e-mail address is being updated to @bears.local after the migration is complete, which means unless you untick the "automatically update" tick box for each user your migrate to EXO and manually set the correct address - this could be a systemic problem.


Update that Default address policy

Simple one to fix, update the default policy from @bear.local to @pokebearswithsticks.com and save the policy, once saved you will need to apply it as well.

Summary

Simple in a couple of numbered bullet points:
  1. Create user accounts with the UPN specified correctly
  2. Ensure you address policies are correct for all users in the domain (including EXO)
  3. Check the accounts are correct before the sync to Entra via AD-Connect

Previous Post Next Post

نموذج الاتصال