21

We recently replaced international MPLS with new ASA 5510s and site-to-site VPNs. However, when we deployed this we ran into a problem where each remote location has 2 ISPs for redundancy, but when enabling the VPN on both interfaces it flaps between the two and the tunnel is up and down as the tunnel gets torn down and moved between ISPs. Cisco has been working on this for 8 months now and we still don't have stable tunnels with multiple ISPs.

Remote Office: 

access-list RWS_TUNNEL remark Interesting traffic for IND-RWS tunnel
access-list RWS_TUNNEL extended permit ip object-group BNG_tunnel_NETS object-group CORP_tunnel_NETS
crypto map RWS_TUNNEL 1 match address RWS_TUNNEL
crypto map RWS_TUNNEL 1 set peer 216.xxx.102.2 
crypto map RWS_TUNNEL 1 set transform-set IND-RWS
tunnel-group 216.xxx.102.2 type ipsec-l2l
tunnel-group 216.xxx.102.2 ipsec-attributes
 pre-shared-key *****


route outside 0.0.0.0 0.0.0.0 216.xxx.206.1 1 track 2
route outside2 0.0.0.0 0.0.0.0 182.xxx.26.229 100
sla monitor 55
 type echo protocol ipIcmpEcho 63.251.61.142 interface outside
 num-packets 5
 timeout 1000
 frequency 10
sla monitor schedule 55 life forever start-time now
track 2 rtr 55 reachability

Central office: 

access-list BNG_TUNNEL remark Interesting traffic for IND-RWS tunnel
access-list BNG_TUNNEL extended permit ip object-group CORP_tunnel_NETS object-group BNG_tunnel_NETS 

route outside2 0.0.0.0 0.0.0.0 216.xxx.102.1
crypto map BNG_TUNNEL 1 match address BNG_TUNNEL
crypto map BNG_TUNNEL 1 set peer 182.xxx.26.230 216.xxx.206.4
crypto map BNG_TUNNEL 1 set transform-set L2L

tunnel-group 182.xxx.26.230 type ipsec-l2l
tunnel-group 182.xxx.26.230 ipsec-attributes
 pre-shared-key *****
tunnel-group 216.xxx.206.4 type ipsec-l2l
tunnel-group 216.xxx.206.4 ipsec-attributes
 pre-shared-key *****

So what I've found is that when ISAKMP is enabled on both outside interfaces (remote office) and both IPs are configured as peers (central office) the VPN comes up successfully on both interfaces, but at some point will begin flapping between IPs. This is true with or without the SLA Monitoring, so even if the routes are all static, behaviour still occurs.

Any insight is appreciated.

Mike Pennington
  • 29,876
  • 11
  • 78
  • 152
Scott Boultinghouse
  • 485
  • 1
  • 4
  • 8
  • To help diagnose the issue, try enabling the "crypto isakmp disconnect-notify" function and let us know what you find. I'm very curious to find out why those tunnels eventually start to flap. – skrumcd May 26 '13 at 21:34

5 Answers5

14

I have been migrating sites away from policy-based VPNs for just this reason. Policy-based VPNs are too unpredictable when it comes to failover behavior. I much prefer route-based IPsec tunnels, either point-to-point or DMVPN. Unfortunately, to my knowledge the ASA platform still doesn't support route-based tunnels.

Jeremy Stretch
  • 4,718
  • 1
  • 31
  • 42
9

I would recommend using a DMVPN solution to connect remote sites over L2L (Lan-to-Lan) IPSec tunnels between ASAs. The DMVPN solution is much easier, cleaner, and will allow spoke to spoke communication as well.

twidfeki
  • 506
  • 3
  • 6
  • Can you elaborate on the fundamental thoughts behind this and how this would be implemented? – SimonJGreen May 08 '13 at 06:46
  • With a DMVPN solution all of the configuration is done on the client side (spokes), you don't have to make any changes to the hubs after the initial setup. For the client, you can create a template to use over and over again. You can have multiple tunnels from the spokes to multiple hubs and use routing protocols to determine which tunnel to route traffic over. Also, you can configure the DMVPN to use multipoint GRE and the spokes can talk to each other directly without passing traffic through the hubs. – twidfeki May 08 '13 at 17:49
4

Could be:

CSCub92666

ASA:Old connections tear down IPSEC vpn tunnel on switchover

Symptom: In IPsec vpn tunnel fail over configuration on ASA,fail over from primary to backup link works. But after second fail over from backup to primary link vpn tunnel start flapping in few minutes and remains unstable. The behavior is observed because of old leftover connection still pointing to backup isp.

t0i
  • 41
  • 1
2

As a follow-up on this question I've been working with Cisco TAC on this for over a year. They've finally identified that this is a bug with the way the ASA platform handles connections. essentially it wasn't clearing connections from one interface when it moved the tunnel to the other interface and it would spiral out of control as it started to see entries in the connection table out both interfaces.

I've deployed IOS version 8.4.7 on the firewall with two ISPs and it actually appears to be behaving properly. Failover is happening when the primary interface goes down, and then moving back when that interface comes back AND remaining on that interface. We'll see.

Scott Boultinghouse
  • 485
  • 1
  • 4
  • 8
2

I agree with above statements. Go simple VTI based IPSEC or alternatively DMVPN. Just remember to run different routing procotol instances within and without the tunnels. Yeah you'll have to replace the ASAs with ISRs.

Are both ISPs going back to a single head office ASA or two? If two I find it hard to see (with the config available at least) how this behaviour could occur, but if its the same ASA (or same pair) it could be related.

wintermute000
  • 443
  • 3
  • 9
  • Yeah, we have a HA pair in the central location. BGP routing hides the multiple ISPs there, but for the remote offices the ISP terminates directly onto the ASA. – Scott Boultinghouse May 08 '13 at 16:59
  • I'd split off the headend so the other ISP connection is terminated on a different device or at least comes in on a different physical interface/IP on the ASA. That should help / trying a different termination device should be free / non disruptive, just use a spare ISR for now – wintermute000 May 09 '13 at 03:57