Looking to fix or repair a broken Intune tunnel, in this particular example both of the servers were showing as server off-line, even though the host was online and active.
If you navigate to this link: Tenant admin - Microsoft Intune admin center you will notice that the servers are offline or unhealthy.
Check the logs from intune
Click on one of the servers, this will show you this, and that tells you that the MDM thinks the server is offline:
From there choose the "Logs" option and then generate a report for the last 7 days, then when you click generate you will be able to download the logs to see if there is anything helpful in them:
This will be the logs from the server that is offline from when it was online:
10/4/2023 12:59:34 PM Warning GnuTLS error (at worker-vpn.c:861): An unexpected TLS packet was received.
10/4/2023 12:59:34 PM Warning GnuTLS error (at worker-vpn.c:861): An unexpected TLS packet was received.
10/4/2023 12:59:34 PM Warning GnuTLS error (at worker-vpn.c:861): An unexpected TLS packet was received.
10/4/2023 12:59:34 PM Warning GnuTLS error (at worker-vpn.c:861): An unexpected TLS packet was received.
10/4/2023 12:59:35 PM Warning GnuTLS error (at worker-vpn.c:861): An unexpected TLS packet was received.
10/4/2023 12:59:35 PM Warning GnuTLS error (at worker-vpn.c:861): An unexpected TLS packet was received.
10/4/2023 12:59:35 PM Warning GnuTLS error (at worker-vpn.c:861): An unexpected TLS packet was received.
10/4/2023 1:28:58 PM Warning GnuTLS error (at worker-vpn.c:861): Error in the pull function.
10/4/2023 2:42:04 PM Warning GnuTLS error (at worker-vpn.c:861): Error in the pull function.
10/4/2023 2:42:06 PM Warning GnuTLS error (at worker-vpn.c:861): Error in the pull function.
10/4/2023 2:42:11 PM Warning GnuTLS error (at worker-vpn.c:861): Error in the pull function.
10/4/2023 2:42:18 PM Warning GnuTLS error (at worker-vpn.c:861): The TLS connection was non-properly terminated.
10/4/2023 2:42:24 PM Warning GnuTLS error (at worker-vpn.c:861): The TLS connection was non-properly terminated.
10/4/2023 3:16:17 PM Warning GnuTLS error (at worker-vpn.c:861): No supported cipher suites have been found.
10/4/2023 4:06:28 PM Warning GnuTLS error (at worker-vpn.c:861): No supported cipher suites have been found.
These servers seem to be having issues with the pull function which is how they are installed, as these servers use Docker, this is how the image is installed....
Reboot/Power Cycle?
Rebooting the server does not fix this particular problem as there’s a more fundamental failure going on here, therefore If you wish to try rebooting the server, you will notice that will fail, Furthermore, if you try to power off the server and power it back on again this will also fail - It’s not an Xbox.
Login and Password Maintanance 👮
These servers will listen on SSH, so connect to the using SSH, my weapon choice here is putty, if you have misplaced the username and password (Yes, it does happen) then find the virtual machine in your azure Subscription and find the “reset password” option - this will tell you what the username is, especially helpful if you have changed it from the default of “root”
Then in the reset password options, you will see the username which here is "crazyroot" so now you know the login and you can also set the new password if required:
SSH to server 👀
Anyway, I’ve digressed, once you have successfully connected to the server and logged in via SSH, once you are logged in use this command, you will notice from below that the service cannot be started, not good:
mstunnel_monitor
fail: Error executing ContainerStatus
/usr/sbin/mst-cli: line 1589: /etc/mstunnel/messages/out/LastError: Permission denied
fail: Error executing FailLogger
fail: Failed to start service mstunnel-agent service
sudo mst-cli uninstall
This will then remove it :
info: Uninstalling...
Now the files are in place you can go back to your setup SSH session when you can press enter for it to continue:
Just before it complete you will get a request that says something like this:
This is where you need to load a browser and visit the website as above, and use the code you have been given in the box below:
You will need to be a Global Administrator or an Intune Administrator else you will get this error:
Error: You must be assigned to role 'Global administrator' or 'Intune administrator'
Shown below in all its glory:
Verify service is online and running
mstunnel_monitor
fail: Error executing ContainerStatus
/usr/sbin/mst-cli: line 1589: /etc/mstunnel/messages/out/LastError: Permission denied
fail: Error executing FailLogger
fail: Failed to start service mstunnel-agent service
You you can try to start it with this command:
sudo mst-cli server start
So once started if you try to query the service with this command, but that says its offline, weird:
sudo mst-cli server show status
Status: offline
This means its not starting, but if you are quick with the status you will see it does actually start running but then it fails again, so if you run this quickly, you will see starts briefly before failing again.
sudo mst-cli server start
sudo mst-cli server show status
General info:
Status: online
Server PID: 1
Sec-mod PID: 733
Sec-mod instance count: 8
Up since: 2023-10-05 11:35 ( 1s)
Active sessions: 0
Total sessions: 0
Total authentication failures: 0
IPs in ban list: 0
Median latency: <1ms
STDEV latency: <1ms
Remove tunnel ☠️
To fix this you need re-install the tunnel configuration
sudo mst-cli uninstall
This will then remove it :
info: Uninstalling...
info: Removing maintenance task
info: Removing service mstunnel-server
info: Stopping service mstunnel-server
Failed to create CoreCLR, HRESULT: 0x80070008
info: Removing service mstunnel-agent
info: Stopping service mstunnel-agent
info: Removing directory /var/mstunnel
info: Removing directory /etc/mstunnel
info: Removing all unreferenced container images
Deleted Images:
untagged: mcr.microsoft.com/tunnel/gateway/agent@sha256:<removed?
deleted: sha256:6a8cb5298c272577fe0354cf5c6eb59446ef3d62b4560ab2547323c6754835a3
deleted: sha256:cf286ae9184e7e8069c3da4970e30e5d5d3d45ffa5c42e87d8d25019e3226424
Reinstall tunnel 🧐
Now you need to re-install it, so lets get started with that, so from the link from the start of this article, choose Servers then create
When you click "Create" you will get two downloads, if this has worked before you really do not need the readiness tool but its good to check for sanity, and then you do need the script to install the service.....
Download those files and lets take a look, once downloaded I used winscp to transfer the files the users home directory, in this case the local user.....
Navigate to this folder, so for me this was /home/crazyroot then you need to make the files executable and able to run with the command below:
chmod -x mst-readiness
Install the Tunnel ✅
When you click "Create" you will get two downloads, if this has worked before you really do not need the readiness tool but its good to check for sanity, and then you do need the script to install the service.....
Download those files and lets take a look, once downloaded I used winscp to transfer the files the users home directory, in this case the local user.....
Readiness Checker
chmod -x mst-readiness
chmod -777 mst-readiness
Then you can run
./mst-readiness
Then you can run
./mst-readiness
This will give you the options you can use:
So this is for checking everything is going to work, so if we choose the network connectivity check when you run this with:
./mst-readiness network
./mst-readiness network
That will cycle though all the websites and confirm the application can access them like this:
Checking connectivity:
Issuing Command: curl https://sts.windows.net/*/v2.0/.well-known/openid-configuration --resolve sts.windows.net:443:20.190.177.20 -v -L -S -s 2>&1 > /dev/null
proxy: NA
->20.190.177.20
> Host: sts.windows.net
< HTTP/1.1 200 OK
Success
The utility can also test the account you’re about to use as well as prerequisites that need to be installed before the tunnel can be installed, however, all you really need to make sure you have the intune administrator or a global administrator.
chmod -x mstunnel-setup
chmod -777 mstunnel-setup
Then you can run
./mstunnel-setup
Then you can run
./mstunnel-setup
This will then start the install and the pull request from the source repo:
Remember you need to enter "yes" not "y" like I did - read the options correctly!
Then it will all the services and then leave you on the screen about the certificate used in conjunction with the tunnel as you can see below:
Start a new putty session to do some file moves, which is required to get the files in the correct place, first you need the files with are the PFX files split into the server key and private key, and the folders for these keys are above.
Remember you need to enter "yes" not "y" like I did - read the options correctly!
Then it will all the services and then leave you on the screen about the certificate used in conjunction with the tunnel as you can see below:
Get the certificate files
I need to be vague in this section because I don’t know the requirements for your naming convention or subject, alternate name in the certificate - however, once you have the PFX file, you can then follow this guide.
I have used Windows for this with OpenSSL installed, so to get the files to the tunnel server I will use WinSCP.
When you have the PFX file to split them using these commands, the produced files already have the correct names, just change the PFX file name to your actual name, here I have certnew.pfx.
openssl pkcs12 -in certnew.pfx -nocerts -out site.key -nodes
openssl pkcs12 -in certnew.pfx -nokeys -out site.crt
Once you have the files move them to the user home folder on the server that will run the tunnel as you can see here with WinSCP
These commands are in the new SSH window, you should have another windows, with the setup in the other one!
Move certificate files 🦒
The path here is /etc/mstunnel however you will not have access to this, to get access
mv /home/crazyroot/site.key /etc/mstunnel/private/site.key
mv /home/crazyroot/site.crt /etc/mstunnel/certs/site.crt
cd /etc/mstunnel
-bash: cd: /etc/mstunnel: Permission denied
So use this command to connect as the root user, you will need the password of the root user to get this:
sudo su -
Then you need to move those files from the home folder to the correct directory with this
Then you need to move those files from the home folder to the correct directory with this
mv /home/crazyroot/site.crt /etc/mstunnel/certs/site.crt
Resume tunnel install ✅
Now the files are in place you can go back to your setup SSH session when you can press enter for it to continue:
Just before it complete you will get a request that says something like this:
Request ID = f8e18bd4-6a04-48ba-88e3-dfd79108c9bb
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code GVWA82BNP
This is where you need to load a browser and visit the website as above, and use the code you have been given in the box below:
You will need to be a Global Administrator or an Intune Administrator else you will get this error:
Error: You must be assigned to role 'Global administrator' or 'Intune administrator'
Shown below in all its glory:
Verify service is online and running
When this is complete the service is online and if you now run this command:
That will confirm its running and it will stay running.....
State: running
sudo mst-cli server status
That will confirm its running and it will stay running.....
State: running
Health: healthy
Then you can check this from the MDM portal and see the magic.
Monitoring for future downtime
Then you can check this from the MDM portal and see the magic.
Monitoring for future downtime
I personally would like some early warning when this platform does fail for whatever reason, Therefore, I was looking at the responses you can get from this if you’re actually not establishing a VPN tunnel, so if you visit the HTTPS - I have noticed you get a response to save I’m alive.
This means you can set up an external based monitoring system to look for this key phrase, and you can then be notified as there could be an indicative problem with the service which would need investigating.
The formula is the public name of the tunnel with a keyword check for "I am still alive!!" to be present which is case sensitive, as below:
This alert will monitor for the online message - and when it is not present, the monitor will fail, and you will be notified.