VPN Flapping Leads To Bogus Routing

Problem

Two computers, A and B, trying to connect across a site-to-site VPN to computers C and D.

Computer A cannot see C but can see D. Computer B can see both computers.

Doing a traceroute from A to C shows that the packet is going out the WAN port instead of the VPN. Traceroute from A to D shows the VPN being used. All traffic from B is using the VPN appropriately.

Clearing the session table of all sessions that involve A clear the problem for some period of time; the problem might or might not return, and when it does it might affect B's communication with D instead of A's communication.

Root cause

Site to site VPN is flapping.

When the VPN is down, there's no route to the remote network, so the default route out the WAN is used and a session is set up. Now that the session is set up, subsequent communication will be sent using this session, even when the VPN returns. The problem only affects computers attempting to communicate while the VPN is in the down state, thus the seemingly random distribution of problems.

Solution

Fix the site to site VPN.

Masking the problem

Put a extremely undesired blackhole route on the firewall for packets destined for the VPN network. When the VPN is down, the packets get blackholed and no session is set up. When the VPN returns, normal service resumes. This doesn't fix sessions getting dropped when the VPN flaps, but it does solve computers getting hung out to dry for long periods of time.

In general I'm adding black hole routes for the RFC-1918 networks to my firewalls now under the premise that I shouldn't be leaking those networks into the internet anyways. As a side-effect it means that you don't end up with bogus information in the session table.

config router static
     edit 0
         set status enable
         set dst 10.0.0.0 255.0.0.0
         set distance 254
         set weight 0
         set comment "Don't Leak RFC 1918"
         set blackhole enable
     next
     edit 0
         set status enable
         set dst 192.168.0.0 255.255.0.0
         set distance 254
         set weight 0
         set comment "Don't leak RFC 1918"
         set blackhole enable
     next
     edit 0
         set status enable
         set dst 172.16.0.0 255.240.0.0
         set distance 254
         set weight 0
         set comment "Don't leak RFC 1918"
         set blackhole enable
     next
     end