Cisco PBR with DMVPN and Dual ISP

Question

I have a DMVPN hub with 2 ISPs. I have two tunnels setup between the spokes, one for each ISP. I would like to dedicate one of the ISP links to voice traffic and failover. I'm having trouble achieving both. In my current config, voice travels where I want it, but I can't get failover working. The limitation seems to be because there's no verify-availability when using set interface under the route-map and the tunnel interface doesn't go down when connectivity is lost. Is there another way to accomplish what I'm looking for?

Here's the relevent config sections:

track 89 ip sla 89 reachability
!
track 245 ip sla 245 reachability

interface Tunnel0
 ip address 10.10.255.1 255.255.255.192
 ip nhrp map multicast dynamic
 ip nhrp network-id 1
 tunnel source GigabitEthernet0/1.245
 tunnel mode gre multipoint
 tunnel key 0
 tunnel route-via GigabitEthernet0/1.245 mandatory
 tunnel protection ipsec profile IPSec-Profile1
!
interface Tunnel1
 ip address 10.10.255.65 255.255.255.192
 ip nhrp map multicast dynamic
 ip nhrp network-id 2
 tunnel source GigabitEthernet0/1.89
 tunnel mode gre multipoint
 tunnel key 64
 tunnel route-via GigabitEthernet0/1.89 mandatory
 tunnel protection ipsec profile IPSec-Profile2

ip route 0.0.0.0 0.0.0.0 1.1.1.1 200 track 89
ip route 0.0.0.0 0.0.0.0 2.2.2.2 200 track 245

ip access-list extended VoIP
 permit ip any any dscp ef

ip sla 89
 icmp-echo 1.1.1.1 source-interface GigabitEthernet0/1.89
ip sla schedule 89 life forever start-time now
ip sla 245
 icmp-echo 2.2.2.2 source-interface GigabitEthernet0/1.245
ip sla schedule 245 life forever start-time now

route-map policyRouting permit 10
 match ip address VoIP
 set interface Tunnel1

UPDATE: Looks like the behavior changed in IOS. In older versions, if no corresponding routes in the routing table are found matching the set interface in the route map, the policy will reject the packet and proceed with normal forwarding. On new versions, it policy routes the packet to the next hop IP found in the routing table, but through the interface specified in the route-map. In my case, this results in the packet being routed through tunnel 1 to tunnel 0's IP. This causes it to fail.

Debug from the spoke - older IOS:

    *Jun  3 16:39:18.279: IP: route map policyRouting, item 10, permit
    *Jun  3 16:39:18.279: IP: s=10.10.17.2 (FastEthernet0/0), d=10.1.4.2 (Tunnel0), len 1228, policy rejected -- normal forwarding
    *Jun  3 16:39:18.279: IP: s=10.10.17.2 (FastEthernet0/0), d=10.1.4.2, len 1228, policy match
    *Jun  3 16:39:18.279: IP: route map policyRouting, item 10, permit
    *Jun  3 16:39:18.279: IP: s=10.10.17.2 (FastEthernet0/0), d=10.1.4.2 (Tunnel0), len 1228, policy rejected -- normal forwarding
    *Jun  3 16:39:18.279: IP: s=10.10.17.2 (FastEthernet0/0), d=10.1.4.2, len 1228, policy match
    *Jun  3 16:39:18.279: IP: route map policyRouting, item 10, permit
    *Jun  3 16:39:18.279: IP: s=10.10.17.2 (FastEthernet0/0), d=10.1.4.2 (Tunnel0), len 1228, policy rejected -- normal forwarding

Debug from the spoke - newer IOS:

    *Jun  3 16:10:15.807: IP: route map policyRouting, item 10, permit
    *Jun  3 16:10:15.807: IP: s=10.10.17.2 (FastEthernet0/0), d=10.1.4.2 (Tunnel1), len 1228, policy routed
    *Jun  3 16:10:15.807: IP: FastEthernet0/0 to Tunnel1 10.10.255.1
    *Jun  3 16:10:15.807: IP: s=10.10.17.2 (FastEthernet0/0), d=10.1.4.2, len 1228, policy match
    *Jun  3 16:10:15.807: IP: route map policyRouting, item 10, permit
    *Jun  3 16:10:15.807: IP: s=10.10.17.2 (FastEthernet0/0), d=10.1.4.2 (Tunnel1), len 1228, policy routed
    *Jun  3 16:10:15.807: IP: FastEthernet0/0 to Tunnel1 10.10.255.1
    *Jun  3 16:10:15.807: IP: s=10.10.17.2 (FastEthernet0/0), d=10.1.4.2, len 1228, policy match
    *Jun  3 16:10:15.807: IP: route map policyRouting, item 10, permit
    *Jun  3 16:10:15.807: IP: s=10.10.17.2 (FastEthernet0/0), d=10.1.4.2 (Tunnel1), len 1228, policy routed

I managed to get failover working using an event manager script. Ugly, but works. Curious if someone has a better solution. — Jeremy, May 29 '14 at 18:53
What is your failure scenario exactly? The tunnel drops but the internet connection out via that gig0/1.89 or gig0/1.245 interfaces is still up? It looks like you're pinging outside the tunnel. — knotseh, May 30 '14 at 09:14
What I'm trying to detect is ISP failure. Since the tunnel interfaces are tied to a specific ISP interface, the tunnels will fail when the ISP fails. Not a problem for traffic that's not routed through PBR since I'm running eigrp. If the tunnels lose connectivity, the routes are removed and traffic will flow through the other tunnel. However, traffic routing via PBR continues to send through the tunnel when the ISP is down. The router will be plugged into a FiOS ONT (ISP1) and a cable modem (ISP2). So the interface will not drop if I lose ISP connectivity. — Jeremy, May 30 '14 at 12:52
I'm pinging outside the tunnel because this is a hub and spoke scenario. It wouldn't make sense to ping a tunnel IP from the hub since multiple spokes will be connected through that tunnel interface. — Jeremy, May 30 '14 at 12:54
Oddly, I'm getting the behavior that I was looking for at the spoke. Route-map is exactly the same. However, when communication over tunnel 1 breaks, it routes traffic over tunnel 0, despite the tunnel 1 interface staying up. I can only assume that when specifying an interface under the route-map, it then references the routing table (or cache) to find a route via that interface. When the tunnel breaks, the EIGRP route through tunnel 1 is removed, and so it routes over tunnel 0. Exactly what I want. However, now I can't figure out why that is not happening at the hub with a similar config. — Jeremy, May 30 '14 at 17:24
ok thanks for the additional information - can you also let us know what Code you're running as well as router? — knotseh, May 30 '14 at 23:48
Hub is 15.1(4)M4. Spoke is 12.4(11)T1. I haven't been able to work on it this week yet. I have a few things I want to try out, including matching IOS on both ends to eliminate the possibility of it being a code difference. — Jeremy, Jun 03 '14 at 12:43
The behavior changed with IOS versions. On the older IOS, if the packet matches the route-map, but it can't find a route corresponding to the "set interface", the policy rejects it and proceeds with normal forwarding (this is what I want). On later versions, it policy routes it based on the next hop IP in the routing table. However, it will route it out the interface specified in the route-map "set interface". So in my case, it will route to the tunnel 0 IP address through the tunnel 1 interface, which fails. I'm adding the debug output demonstrating this to the original question. — Jeremy, Jun 03 '14 at 17:11

score 4 · Accepted Answer · answered May 29 '14 at 19:28

Here's what I came up with until I find a better way.

    event manager environment downInterface t1
    event manager environment upInterface t1
    event manager applet ShutInterface
     event track 89 state down
     action 0 cli command "enable"
     action 1 cli command "conf t"
     action 2 cli command "interface $downInterface"
     action 3 cli command "shut"
     action 4 cli command "end"
     action 5 syslog priority errors msg "Shutting down $downInterface"
    event manager applet NoshutInterface
     event track 89 state up
     action 0 cli command "enable"
     action 1 cli command "conf t"
     action 2 cli command "interface $upInterface"
     action 3 cli command "no shut"
     action 4 cli command "end"
     action 5 syslog priority informational msg "Enabling $upInterface"

EEM is another way to do it! Thanks for posting this! – knotseh Jun 03 '14 at 01:49 — knotseh, Jun 03 '14 at 01:49

score 2 · Answer 2 · answered Jun 03 '14 at 01:48

2

I think you need to add to your route-map.

set ip next-hop YOUR_IP_ADDRESS

Where YOUR_IP_ADDRESS is the next hop router you want to route to.

eg)

route-map policyRouting permit 10
 match ip address VoIP
 set ip next-hop x.y.z.w

You should be able to keep the set interface command however the router should automatically select the interface which matches its routing table. Without the set ip next-hop command, the router will forward the packet based on its routing table - regardless of the interface required.

For 12.2 based code - take a look at Cisco's documentation.

answered Jun 03 '14 at 01:48

knotseh

2,535
4
24
35

Problem is, the next hop IP is learned through EIGRP (assuming it should be a tunnel IP, not the ISP gateway). Also, since this a hub & spoke scenario, I would need a route-map for each site and I would need to expand the route map to match on destination IP. – Jeremy Jun 03 '14 at 11:58
Problem is, the behavior you described is not what I'm seeing (at the hub anyway). It's not forwarding based on the routing table. If tunnel 1 loses connectivity, the routes are removed from the routing table since EIGRP will lose it's neighbors. However, it still continues to send traffic through the tunnel interface despite there not being any routing table entries for tunnel 1. That's where my problem lies. Traffic that doesn't match the route-map works fine, since that does get routed based on the routing table. – Jeremy Jun 03 '14 at 12:03
I'm going to start over with the hub config to simplify it and to eliminate any other possible config factors since it's working as desired at the spoke end. I'll post back what I find, if anything, along with the full configs of both ends. – Jeremy Jun 03 '14 at 12:06
From that document (I saw this before as well): "The set interface command is supported only over point-to-point links, unless a route cache entry exists using the same interface specified in the set interface command in the route map.". If I'm interpreting that correctly, it looks like it references the route cache for a route entry through the interface specified. However, looking at the route cache when tunnel1 is down, there are no entries for tunnel 1 for the remote end network. Yet, it still attempts to route packets that match the route-map through Tunnel1. – Jeremy Jun 03 '14 at 12:51
hmm this is very interesting. I've only successfully done PBR on P2P interfaces or tunnels. I did try this over DMVPN one time and what you're finding is similar to what happened to me. In my case, the traffic didn't even acknowledge the route-map. I may lab this and see what happens. Let us know the solution if you find it! – knotseh Jun 05 '14 at 23:28

Cisco PBR with DMVPN and Dual ISP

2 Answers2