When you Privatelink DNS does not work and you are wondering why, first you need to assess the scenario:
- Storage Account that uses a private endpoint and a private DNS zone
- Conditional forwarders on-prem that ultimately point to 168.63.129.16 which is Azure DNS
- Conditional forwarders are AD Integrated therefore replicating to all DC's in the forest
However when you try to resolve, or ping these addresses you get host no found, you do not get the reponse, however if you connect to Azure DNS from an Azure DC and perform these commands:
- nslookup
- server 168.63.129.16
- testsite.blob.core.windows.net
- Correct response
However if you do this from the domain controller either Azure or local you get "no response" or "host not found"
The flow path looks like this from clients:
Client > DNS > DC > Azure DC > Azure DNS > Privatelink
However you connection does not get to Privatelink for the record it fails well before that, well actually this is what is going on here:
Client > DNS > DC > Azure DC > Fail
So why is this failing at the Azure DC, it is like the forwarders re being ignored, well this is where it gets interesting, at the start of the article step 3 the AD Integrated zone is where we need to look further.
We need the local DCs that are no hosted in Azure to point at the Azure DC's and then we need the Azure DC to point at Azure DNS for the Privatelink, but the failures seem to occur with Azure DNS, but not when we point at it???? What????
Now the issue is with step 3 in this case, where the conditional forwarder is overriding the forwarder as you will notice if you view the forwarding tab on the DNS properties in the MMC, the bit in the red box at the bottom of the image........
This warning really should be on the conditional forwarding rule when you create it, but it is not, what is even worse is the fact that if you use PowerShell to create forwarders or conditional forwarders you do not even get a warning of this, you need to use the GUI MMC via remote management to even get this warning.
Therefore the solution is to do this for all your local Domain controllers and NOT replicate this with the domain, so it is stored in the registry not AD, this is called "PassThru"
Add-DnsServerConditionalForwarderZone -Name "blob.core.windows.net" -ComputerName local-DC -MasterServers azuredc1,azuredc2 -PassThru
NOTE ; You need to replace local-DC with your DC server IP then replace azuredc1 with the Azure DC.
If you add a conditional forwarder to an Azure DC where it points at itself it will ignore the forwarders completely this means if you take this situation
a6n.co.uk resolves like this:
nslookup
Default Server: st1w10222.stwater.intra
Address: 10.245.161.45
> a6n.co.uk
Server: dc1.private.link
Address: 10.845.77.44
Non-authoritative answer:
Name: a6n.co.uk
Addresses: 2001:4860:4802:34::15
2001:4860:4802:36::15
2001:4860:4802:32::15
2001:4860:4802:38::15
216.239.36.21
216.239.34.21
216.239.32.21
216.239.38.21
This is working as expected, but now lets add a conditional forwarder for a6n.co.uk pointing at the Azure DC itself......
> a6n.co.uk
Server: dc1.private.link
Address: 10.845.77.44
*** dc1.private.link can't find a84543534n.co.uk: Non-existent domain
Broken, this confirms indeed that the conditional forwarder has not only broke Azure DNS, but it has ignored any of entries in the forwarders tab even if you point at Google or OpenDNS for DNS queries.
If you are in this situation you need to delete all your AD Integrated DNS zones and manually create the zone using the PowerShell above ONLY on the local DC's.
Finally if after this you notice than some of your domain controllers do no resolve the zone and some do, on the private link DNS in Azure select the zone in question then under Settings and "Virtual network links" as shown below:
Ensure that you have added the this zone to the virtual network on which the DC is installed on, failure to complete the step will mean the DC will still not be able to resolve the zone,
So for more clarity if you have one DC in North Europe and one is South Europe, if you only add the North Europe subnet then on the DC's only in North Europe will be able to resolve the Privatelink addresses, the ones in South Europe will fail.
Sounds obvious reading a blog, with the added bonus of hindsight and no pressure, but this is the last thing you think about when a production issue is running and you are trying to fix it.