I have the same , but also have an IP ACL on the VPN ports / protocols (only on new connections). When I do need to Connect from random remote ip , I have a port knocking rule that adds ip for 5min to acl

I actually have the port knock action on a blank website w a unusual subdomain. So we need i it I pull up website and can then vpn in temporaryly