Cisco Performance Routing version 3 - PfRv3

Introduction

In my previous posts, DMVPN Part I – IV, I have described and built a dual homed DMVPN overlay network over MPLS and Internet WAN. In this post, I am going to enable an intelligent path control with the Performance Routing version 3 (PfRv3). Using a DMVPN together with a PfRv3, we can have a transport independent WAN, where forwarding decision is done based on the application requirements (e.q bandwidth, delay, jitter, and packet loss) instead of only relying on information received from routing protocols.

Figure 1 shows the example topology. There is one Master Controller (MC) S1R4 and two Border Routers (BR) S1R1 and S1R2 on the Hub site. On the Remote site, there is a router which has two roles, site Master Controller (MC) and site Border Router (BR). Remote site BR and Hub BRs are connected over MPLS and the Internet using a DMVPN overlay technology. A routing protocol for overlay network is BGP. The complete DMVPN configuration can be found in the document: “DMVPN Part III. BGP over DMVPN WAN”. On the Hub site, there is an internal router that has four loopback interfaces which represent LANs. There are two business critical networks, 172.16.2.0/24 – Voice and 172.16.3.0/24 – CRM. We will use these two networks in the PfRv3 demonstration. 

Figure 1: Example topology picture

I am going to build a PfRv3 solution step by step and during the building process, I am also going to explain the theory part. Complete device configuration can be found from the Appendix 1 at the end of the document.

Setting up the Hub site Master Controller

Figure 2 shows the component related to Global Domain MC.

Figure 2. Global Master Controller on HUB site

Step 1: Define the IWAN domain name.

IWAN domain name is common for all of the routers participating in the IWAN domain (MC/BR). All IWAN related configuration will be made under the domain configuration.

S1R4-MC(config)#domain WOMBAT

S1R4-MC(config-domain)#

Step 2: Under the domain configuration, define the VRF and set the role of the router to master hub.

Since we don’t have any vrf on MC we set the vrf to default. The router is hub master and it is identified based on the Loopback0 interface ip-address, which has to be reachable for other BR/MC. We are going to use password cisco.

S1R4-MC(config-domain)#vrf default 

S1R4-MC(config-domain-vrf)#master hub

S1R4-MC(config-domain-vrf-mc)#password cisco

S1R4-MC(config-domain-vrf-mc)#source-interface loopback 0

Step 3: Define local/enterprise networks.

Create an ip prefix-list that defines site-local networks. Assign the prefix-list to MC configuration. This is mandatory on the HUB MC, without it, the status of MC will be down. I have defined three entries, one for the LAN#1-2 (172.16.0.0/23) and dedicated entries for Voice 172.16.2.0/24 and CRM 172.16.3.0/24.

S1R4-MC(config)# ip prefix-list SITE-PREFIX-PFR seq 5 permit 172.16.0.0/23

S1R4-MC(config)# ip prefix-list SITE-PREFIX-PFR seq 10 permit 172.16.2.0/24

S1R4-MC(config)# ip prefix-list SITE-PREFIX-PFR seq 15 permit 172.16.3.0/24

!

S1R4-MC(config-domain-vrf-mc)#site-prefixe prefix-list SITE-PREFIX-PFR

We also need to define enterprise networks with prefix-list. This prefix-list should include all the network that needs to be controlled by PfRv3 in PfR Domain. I am going to use 172.16.0.0/16.

S1R4-MC(config)# ip prefix-list ENTERPRISE-PREFIX seq 5 permit 172.16.0.0/16

!

S1R4-MC(config-domain-vrf-mc)#enetrpise-prefixe prefix-list ENTERPRISE-PREFIX

Step 4: Hub Point of Presence (POP) Id and Site Id (not configurable).

Every IWAN site has site-Id that is derived from the loopback address of the site Local MC. POP Id is only used in Hub or Transit sites. For the Hub site, the POP Id is automatically set to 0.

So far we have done the following configuration to Hub MC

S1R4-MC#sh run | sec domain|prefix

domain WOMBAT

 vrf default

  master hub

   source-interface Loopback0
   enetrpise-prefixe prefix-list ENTERPRISE-PREFIX

   site-prefixes prefix-list SITE-PREFIX

   password cisco

ip prefix-list SITE-PREFIX seq 5 permit 172.16.0.0/22
ip prefix-list ENTERPRISE-PREFIX seq 5 permit 172.16.0.0/16

Setting up Border Routers (BR) of the Hub site

Steps 1-3: The configuration of PfR domain on BRs on the Hub site is quite equal with the MC configuration. First, we need to specify the domain name and VRF, then we set the role which is now border. Under the border configuration, we set the source interface and password. We also define the site MC address which is the Loopback 0 interface ip address (192.168.14.14) of Hub site MC. This configuration is equal in both border routers S1R1 and S1R2.

S1R1(config)#domain WOMBAT

S1R1(config-domain)#vrf default

S1R1(config-domain-vrf)#border

S1R1(config-domain-vrf-br)#source-interface Loopback0

S1R1(config-domain-vrf-br)#master 192.168.14.14

S1R1(config-domain-vrf-br)#password cisco

Steps 4: Define the path names and path-Id

Path name and ID identifies the underlying transport network. I am going to name the primary transport network to MPLS with path-Id 11 and secondary ones to INET with path-Id 12. These settings are done under the tunnel interfaces. Note that we also need to define the domain to which the tunnel interface belongs to. All of these parameters are defined within one configuration line.

S1R1(config)#int tunnel 11

S1R1(config-if)# domain WOMBAT path MPLS path-id 11

S1R2(config)#int tunnel 12

S1R2(config-if)# domain WOMBAT path INET path-id 12

Figure 3. Border Router on HUB site

Setting up the Branch site Master Controller (MC)

The configuration of the MC on Branch site follows the same principal than what we have done with Hub MC. First, we define the name of the IWAN domain to WOMBAT and then we assign it to the default vrf. Under the vrf configuration, we set the role to master branch and set both the password and source address. Since Branch MC needs to communicate with the Hub MC, we need to define the ip address of Hub MC. We could, but we do not have to define site prefixes on Branch MC since they are learned from egress traffic.

S2R1(config)# 

S2R1(config)#domain WOMBAT

S2R1(config-domain)#vrf default

S2R1(config-domain-vrf)#master branch

S2R1(config-domain-vrf-mc)#source-interface Loopback0

S2R1(config-domain-vrf-mc)#password cisco

S2R1(config-domain-vrf-mc)#hub 192.168.14.14

Figure 4.MC on the Remote site

Setting up the Branch site Border Router (BR)

There are a couple of differences between Hub BR and Branch BR configuration, first Branch BR will use its local MC instead of Hub Site MC, second, there is no need for path name/id configuration since those are learned automatically from the Hub MC.

S2R1(config)#domain WOMBAT

S2R1(config-domain)#vrf default

S2R1(config-domain-vrf)#border

S2R1(config-domain-vrf-br)#source-interface Loopback0

S2R1(config-domain-vrf-br)#master local

S2R1(config-domain-vrf-br)#password cisco

Figure 5. BR on the Remote site

Monitoring peering MC and BR peering in IWAN domain

Hub Site Master Controller

From the Hub site MC, we can see that the operational status of MC is UP, MC use Loopback 0 ip address 192.168.14.14 for communication with other systems inside IWAN domain, connection status to both Hub site Border Routers is CONNECTED. We can also see that the there is one external interface per BR, S1R1 has external interface Tunnel11 (MPLS) with Path-Id 11 and S1R2 has external interface Tunnel12 (INET) with Path-id 12. Both BR has also Tunnel 0 interface. It is an auto-tunnel which is used for traffic reroute, but I will explain that on next chapter.

S1R4-MC#sh domain WOMBAT master status | sec Type|Up|Loop|Borders

  Instance Type:    Hub

  Operational status:  Up

  Loopback IP Address: 192.168.14.14

  Borders:

    IP address: 192.168.11.11

    Version: 2

    Connection status: CONNECTED (Last Updated 05:43:15 ago )

    Interfaces configured:

      Name: Tunnel11 | type: external | Service Provider: MPLS path-id:11 | Status: UP | Zero-SLA: NO | Path of Last Resort: Disabled

          Number of default Channels: 0

    Tunnel if: Tunnel0

    IP address: 192.168.12.12

    Version: 2

    Connection status: CONNECTED (Last Updated 05:39:30 ago )

    Interfaces configured:

      Name: Tunnel12 | type: external | Service Provider: INET path-id:12 | Status: UP | Zero-SLA: NO | Path of Last Resort: Disabled

          Number of default Channels: 0

         

    Tunnel if: Tunnel0

Figure 6. MC on HUB site

Hub Site Border Routers

As can be seen from both Border routers in Hub site, connection to MC is ok and the external WAN interfaces are up.

S1R1#sh domain WOMBAT border status                  

<snipped>

Instance Status: UP

Present status last updated: 05:58:01 ago

Loopback: Configured Loopback0 UP (192.168.11.11)

Master: 192.168.14.14

Master version: 2

Connection Status with Master: UP

MC connection info: CONNECTION SUCCESSFUL

<snipped>

External Wan interfaces:

     Name: Tunnel11 Interface Index: 13 SNMP Index: 8 SP: MPLS path-id: 11 Status: UP Zero-SLA: NO Path of Last Resort: Disabled

Auto Tunnel information:

   Name:Tunnel0 if_index: 14

   Virtual Template: Not Configured

   Borders reachable via this tunnel:  192.168.12.12

S1R2#sh domain WOMBAT border status

<snipped>

Instance Status: UP

Present status last updated: 05:56:50 ago

Loopback: Configured Loopback0 UP (192.168.12.12)

Master: 192.168.14.14

Master version: 2

Connection Status with Master: UP

MC connection info: CONNECTION SUCCESSFUL

<snipped>

External Wan interfaces:

     Name: Tunnel12 Interface Index: 13 SNMP Index: 8 SP: INET path-id: 12 Status: UP Zero-SLA: NO Path of Last Resort: Disabled

Auto Tunnel information:

   Name:Tunnel0 if_index: 14

   Virtual Template: Not Configured

   Borders reachable via this tunnel:  192.168.11.11

Figure 7. BR on HUB site

Auto Tunnel

Auto-tunnel is an mGRE tunnel that is used in a situation where the external WAN link of BR does not full fill the requirements of the specific class/application. In this situation, the MC instructs the BR to reroute traffic stream over the Auto-tunnel to the other BR, which then forward the traffic stream over its’ external WAN interface to the destination. Auto-tunnel has generated automatically and does not need any manual configuration. 

BR S1R1 has Auto-tunnel Tunnel0 and it can reach the BR S1R2 (192.168.12.12) through it.

S1R1#sh domain WOMBAT border status | beg Auto Tunnel

Auto Tunnel information:

   Name:Tunnel0 if_index: 14

   Virtual Template: Not Configured

   Borders reachable via this tunnel:  192.168.12.12

And BR S1R1 also has a route to the target.

S1R1#sh ip route | i 192.168.12.12 

O        192.168.12.12 [110/2] via 10.11.12.12, 09:45:58, GigabitEthernet3

BR S1R1 use Loopback 0 as a source address and the mode of the tunnel GRE multipoint (mGRE). Note that there is a “derived-config” option in show run command since it also shows the auto-generated configurations.

S1R1#sh derived-config int tunn 0

Building configuration...

Derived configuration : 186 bytes

!

interface Tunnel0

 description PFR auto-tunnel for VRF default

 ip unnumbered Loopback0

 no ip redirects

 tunnel source Loopback0

 tunnel mode gre multipoint

 tunnel key 43434343

end

Here we can see the information about Tunnel0 interface.

S1R1#sh interface tunn 0

Tunnel0 is up, line protocol is up

  Hardware is Tunnel

  Description: PFR auto-tunnel for VRF default

  Interface is unnumbered. Using address of Loopback0 (192.168.11.11)

  MTU 9960 bytes, BW 10000 Kbit/sec, DLY 50000 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation TUNNEL, loopback not set

  Keepalive not set

  Tunnel linestate evaluation up

  Tunnel source 192.168.11.11 (Loopback0)

   Tunnel Subblocks:

      src-track:

         Tunnel0 source tracking subblock associated with Loopback0

          Set of tunnels with source Loopback0, 1 member (includes iterators), on interface <OK>

  Tunnel protocol/transport multi-GRE/IP

    Key 0x296C167, sequencing disabled

    Checksumming of packets disabled

  Tunnel TTL 255, Fast tunneling enabled

  Tunnel transport MTU 1460 bytes

  Tunnel transmit bandwidth 8000 (kbps)

  Tunnel receive bandwidth 8000 (kbps)

  Last input never, output never, output hang never

  Last clearing of "show interface" counters 07:49:52

  Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/0 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

     0 packets input, 0 bytes, 0 no buffer

     Received 0 broadcasts (0 IP multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

     0 packets output, 0 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 output buffer failures, 0 output buffers swapped out

S1R1#                                                                       

In figure 8 Hub site BR S1R1 receives traffic destined to the Remote site. It forwards data over the MPLS path. BR S2R1 on Remote site monitors ingress traffic from the MPLS link. It notices that the link is congested and it does not full fill requirements for application-1. It notifies local MC about the event which in turns sends a Threshold  Crossing Alert (TCA) message to Hub site MC over both WAN links (for reliability).

Figure 8. Threshold Crossing Alert

After receiving TCA from remote BR, the MC of Hub site instructs the BR S1R1 to reroute traffic over the Auto-tunnel to BR S1R2. When BR S1R1 receives the next packet from the same flow, it encapsulates it with the GRE header and forwards it over the Auto-tunnel to the BR S1R2, which in turn removes the tunnel header and forwards the packet over the INET link. 

Figure 9. Reaction to TCA

This was simplified description how the Auto-tunnel is used, but I will explain the whole process in greater detail on becoming chapters.

Policies

Now we are going to specify requirements for each traffic class. We have two traffic classes with following requirements:

-Voice: Path MPLS, fall-back INET: One-way-delay 800ms, Packet loss 1%, monitor interval 5 sec.

-Critical: Path MPLS, fall-back INET: One-way-delay 700ms, Packet loss 5%, monitor interval 5 sec.

Note! I am using Cisco Virtual Internet Routing Lab (VIRL) in my lab and delays are quite high. That’s why the one-way delays are set up with high values.

Domain policy configuration

Policy configuration is done on the Hub MC.

Step 1: Define the traffic classes. Traffic Classes are configured under the MC configuration. We have two classes; VOICE and CRITICAL.

S1R4-MC#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

S1R4-MC(config)#domain WOMBAT

S1R4-MC(config-domain)#vrf default

S1R4-MC(config-domain-vrf)#master hub

S1R4-MC(config-domain-vrf-mc)#class VOICE sequence 10

S1R4-MC(config-domain-vrf-mc)#class CRITICAL sequence 20

Step 2: Define Policies, monitor interval and path preference for each class

Move to MC configuration and set the monitor interval. Normal interval is 30 seconds but we want faster reaction time for critical traffic so we set the monitor interval to 5 seconds.

S1R4-MC(config)#domain WOMBAT

S1R4-MC(config-domain)#vrf default

S1R4-MC(config-domain-vrf)#master hub

S1R4-MC(config-domain-vrf)#monitor-interval 5 dscp af21

S1R4-MC(config-domain-vrf)#monitor-interval 5 dscp ef

Define policies and path preference for traffic class VOICE

S1R4-MC(config-domain-vrf-mc)#class VOICE sequence 10

S1R4-MC(config-domain-vrf-mc-class)#match dscp ef policy custom

S1R4-MC(config-domain-vrf-mc-class-type)#priority 2 loss threshold 1

S1R4-MC(config-domain-vrf-mc-class-type)#priority 1 one-way-delay threshold 800       

S1R4-MC(config-domain-vrf-mc-class-type)#path-preference MPLS fallback INET

Define policies and path preference for traffic class CRITICAL

S1R4-MC(config-domain-vrf-mc-class)#class CRITICAL sequence 20

S1R4-MC(config-domain-vrf-mc-class)#match dscp af21 policy custom

S1R4-MC(config-domain-vrf-mc-class-type)#priority 2 loss threshold 5

S1R4-MC(config-domain-vrf-mc-class-type)#priority 1 one-way-delay threshold 700       

S1R4-MC(config-domain-vrf-mc-class-type)#path-preference MPLS fallback INET

Now all the basics PfRv3 configuration are done. Here is the complete domain WOMBAT configuration for Hub Site MC + related prefix-lists.

S1R4-MC#sh run | sec domain W|-PREFIX   

domain WOMBAT

 vrf default

  master hub

   source-interface Loopback0

   site-prefixes prefix-list SITE-PREFIX-PFR

   password cisco

   monitor-interval 5 dscp af21

   monitor-interval 5 dscp ef

   enterprise-prefix  prefix-list ENTERPRISE-PREFIX

   class VOICE sequence 10

    match dscp ef policy custom

      priority 2 loss threshold 1

      priority 1 one-way-delay threshold 800     

     path-preference MPLS fallback INET

   class CRITICAL sequence 20

    match dscp af21 policy custom

      priority 2 loss threshold 5

      priority 1 one-way-delay threshold 700

     path-preference MPLS fallback INET

ip prefix-list ENTERPRISE-PREFIX seq 5 permit 172.16.0.0/16

ip prefix-list SITE-PREFIX-PFR seq 5 permit 172.16.0.0/23

ip prefix-list SITE-PREFIX-PFR seq 10 permit 172.16.2.0/24

ip prefix-list SITE-PREFIX-PFR seq 15 permit 172.16.3.0/24

Figure 10. Domain Policies

Monitoring

Site Prefix Database

Configurations are ready, next we are going to check how the system actually works. In Hub MC configuration, we defined the Hub site local network (prefix-list SITE-PREFIX-PFR) and enterprise level networks (ip prefix-list ENTERPRISE-PREFIX). We did not configure any prefix-list for local networks on the Branch MC since local networks are learned dynamically from outgoing traffic.

As can be seen from the following outputs, Hub MC and BRs as well as Branch MC/BR, they all have common prefix database. The difference is how the prefixes are learned. Let’s take a look at the network 172.16.3.0/24. There are two flags attach to entry, C and M on the Hub MC. The C-Flag is unambiguous, it tells that this particular prefix is included in site prefix-prefix definition in MC configuration. M tells that this entry is shared to other MCs or BRs. But how it is shared?

S1R4-MC#sh domain WOMBAT master site-prefix

  Change will be published between 5-60 seconds

  Next Publish 01:20:04 later

  Prefix DB Origin: 192.168.14.14

  Last publish Status : Peering Success

  Total publish errors : 0

  Total learned prefix discards: 0

  Prefix Flag: S-From SAF; L-Learned; T-Top Level; C-Configured; M-shared

Site-id              Site-prefix          Last Updated         DC Bitmap  Flag     

--------------------------------------------------------------------------------

192.168.21.21         10.11.21.21/32       01:28:49 ago         0x0         S

192.168.21.21         10.12.21.21/32       01:28:49 ago         0x0         S

192.168.14.14         172.16.0.0/23        03:36:33 ago         0x1         C,M

192.168.14.14         172.16.2.0/24        03:36:33 ago         0x1         C,M

192.168.14.14         172.16.3.0/24        03:36:33 ago         0x1         C,M

192.168.21.21         172.16.21.21/32      01:28:49 ago         0x0         S

255.255.255.255      *172.16.0.0/16        02:40:08 ago         0x0         T

192.168.14.14         192.168.14.14/32     03:36:33 ago         0x1         L

192.168.21.21         192.168.21.21/32     01:28:49 ago         0x0         S

From the Hub site BR S1R1 and from the Branch site MC/BR we can see that an entry 172.16.3.0/24 has also S-flag attached to it. This means that this information is received from the MC via EIGRP SAF. The SAF stand for “Service Advertisement Framework” and in PfRv3 it is used among the other things for publishing/subscribing information about prefixes, but it is not used for routing advertisement. We can see that prefix 172.16.3.0/24 is located on the site that has a site-id 192.168.14.14 (Hub MC Loopback0), but there is no information about how this site is reachable.

S1R1#sh domain WOMBAT border site-prefix

  Prefix Flag: S-From SAF; L-Learned; T-Top Level; C-Configured; M-shared

Site-id              Site-prefix          Last Updated         DC Bitmap  Flag     

--------------------------------------------------------------------------------

192.168.21.21         10.11.21.21/32       01:31:09 ago         0x0         S

192.168.21.21         10.12.21.21/32       01:31:09 ago         0x0         S

192.168.14.14         172.16.0.0/23        00:42:16 ago         0x1         S,C,M

192.168.14.14         172.16.2.0/24        00:42:16 ago         0x1         S,C,M

192.168.14.14         172.16.3.0/24        00:42:16 ago         0x1         S,C,M

192.168.21.21         172.16.21.21/32      01:31:09 ago         0x0         S

255.255.255.255      *172.16.0.0/16        02:42:17 ago         0x0         S,T

192.168.14.14         192.168.14.14/32     00:42:16 ago         0x1         S

192.168.21.21         192.168.21.21/32     01:31:09 ago         0x0         S

Same flags can be seen from the branch site MC…

S2R1#sh domain WOMBAT master site-prefix

  Change will be published between 5-60 seconds

  Next Publish 00:16:18 later

  Prefix DB Origin: 192.168.21.21

  Last publish Status : Peering Success

  Total publish errors : 0

  Total learned prefix discards: 0

  Prefix Flag: S-From SAF; L-Learned; T-Top Level; C-Configured; M-shared

Site-id              Site-prefix          Last Updated         DC Bitmap  Flag     

--------------------------------------------------------------------------------

192.168.21.21         10.11.21.21/32       03:18:54 ago         0x0         L

192.168.21.21         10.12.21.21/32       03:18:54 ago         0x0         L

192.168.14.14         172.16.0.0/23        00:54:50 ago         0x1         S,C,M

192.168.14.14         172.16.2.0/24        00:54:50 ago         0x1         S,C,M

192.168.14.14         172.16.3.0/24        00:54:50 ago         0x1         S,C,M

192.168.21.21         172.16.21.21/32      01:52:21 ago         0x0         L

255.255.255.255      *172.16.0.0/16        02:54:50 ago         0x0         S,T

192.168.14.14         192.168.14.14/32     00:54:50 ago         0x1         S

192.168.21.21         192.168.21.21/32     03:18:54 ago         0x0         L

…and from branch site BR.

S2R1#show domain WOMBAT border site-pre

  Change will be published between 5-60 seconds

  Prefix DB Origin: 192.168.21.21

  Last publish Status :

  Total publish errors : 0

  Prefix Flag: S-From SAF; L-Learned; T-Top Level; C-Configured; M-shared

Site-id              Site-prefix          Last Updated         DC Bitmap  Flag     

--------------------------------------------------------------------------------

192.168.21.21         10.11.21.21/32       01:56:21 ago         0x0         S

192.168.21.21         10.12.21.21/32       01:56:21 ago         0x0         S

192.168.14.14         172.16.0.0/23        01:07:28 ago         0x1         S,C,M

192.168.14.14         172.16.2.0/24        01:07:28 ago         0x1         S,C,M

192.168.14.14         172.16.3.0/24        01:07:28 ago         0x1         S,C,M

192.168.21.21         172.16.21.21/32      01:56:21 ago         0x0         S

255.255.255.255      *172.16.0.0/16        03:07:29 ago         0x0         S,T

192.168.14.14         192.168.14.14/32     01:07:28 ago         0x1         S

192.168.21.21         192.168.21.21/32     01:56:21 ago         0x0         S

Lets’ take a closer look at the EIGRP SAF operation. We can see from the Hub MC that it has three EIGRP-SFv4 neighbors 192.168.11.11 (BR S1R1), 192.168.12.12 (BR S1R2) and 192.168.21.21 (BR S2R1).

S1R4-MC#sh eigrp service-family ipv4 neighbors

EIGRP-SFv4 VR(#AUTOCFG#) Service-Family Neighbors for AS(59501)

H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq

                                                   (sec)         (ms)       Cnt Num

2   192.168.21.21           Lo0                     535 05:38:05    7   100  0  165

1   192.168.12.12           Lo0                     547 05:38:16    2   100  0  10

0   192.168.11.11           Lo0                     542 05:38:16    6   100  0  5

From the Hub MC, we see that the EIGRP-SFv4 (AS 59501) is enabled on interface Loopback 0. It also listens to an unicast-based peering subscriptions from local BRs and from remote MCs. Even not shown in configuration example, the only interface where EIGRP is enabled is Loopback 0 which means that it only listens to EIGRP-SF peering request destined to its’ Loopback0 interface.

S1R4-MC#sh derived-config | sec eigrp

router eigrp #AUTOCFG# (API-generated auto-configuration, not user configurable)

 !

 service-family ipv4 autonomous-system 59501

<snipped>

  !

  sf-interface Loopback0

   hello-interval 120

   hold-time 600

  exit-sf-interface

  !

<snipped>

  !

  topology base

  exit-sf-topology

  remote-neighbors source Loopback0 unicast-listen

 exit-service-family

From the Hub BR S1R1, we can see that it peers with both Hub site MC and other BR.

S1R1#sh eigrp service-family ipv4 neighbors

EIGRP-SFv4 VR(#AUTOCFG#) Service-Family Neighbors for AS(59501)

H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq

                                                   (sec)         (ms)       Cnt Num

4   192.168.14.14           Lo0                     533 05:49:05 1748  5000  0  33

3   10.11.12.12             Gi3                     519 05:49:05    1  3000  0  11

Unlike on the Hub MC, BR S1R1 has EIGRP enabled on all of the GigabitEthernet interfaces (not shown in configuration) and on the Auto-tunnel interface (Tunnel 0) but not on the DMVPN tunnel interface (Tunnel 11). This way it is able to send an EIGRP peering subscriptions out of all site internal interfaces. BR takes the MC address from the domain WOMBAT configuration where we defined the master controller address and it uses its’ own Loopback 0 interface ip address as a source address. Note that BRs are EIGRP Stub routers.

S1R1#sh derived-config | sec eigrp

router eigrp #AUTOCFG# (API-generated auto-configuration, not user configurable)

 !

 service-family ipv4 autonomous-system 59501

  eigrp stub connected

  !

<snipped>

  !

  sf-interface Loopback0

   hello-interval 120

   hold-time 600

  exit-sf-interface

  !

  sf-interface Tunnel0

   hello-interval 120

   hold-time 600

  exit-sf-interface

  !

  sf-interface Tunnel11

   shutdown

   hello-interval 120

   hold-time 600

  exit-sf-interface

  !

<snipped>

  !

  topology base

  exit-sf-topology

  neighbor 192.168.14.14 Loopback0 remote 100

 exit-service-family

On the Remote site, MC S2R1 has established peering only with Hub site MC, not with Hub site BRs.

S2R1#sh eigrp service-family IPv4 neighbors

EIGRP-SFv4 VR(#AUTOCFG#) Service-Family Neighbors for AS(59501)

H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq

                                                   (sec)         (ms)       Cnt Num

1   192.168.14.14           Lo0                     488 06:28:36  223  1338  0  36

Since S2R1 both Local MC and BR, it listens to unicast messages in the same way that the Hub site MC and send EIGRP peering request as a BR. There is also two neighbors shown in an EIGRP-SF configuration, one for the Branch MC to Hub MC and one for the internal communication between local MC and local BR. EIGRP is enabled in all GigabitEthernet interface but not in DMVPN tunnel interface.

S2R1#sh derived-config | sec eigrp 

router eigrp #AUTOCFG# (API-generated auto-configuration, not user configurable)

 !

 service-family ipv4 autonomous-system 59501

  eigrp stub connected leak-map

  !

  sf-interface GigabitEthernet1

   hello-interval 120

   hold-time 600

  exit-sf-interface

  !

  sf-interface LI-Null0

   hello-interval 120

   hold-time 600

  exit-sf-interface

  !

  sf-interface Loopback0

   hello-interval 120

   hold-time 600

  exit-sf-interface

  !

  sf-interface Loopback21

   hello-interval 120

   hold-time 600

  exit-sf-interface

  !      

  sf-interface Tunnel11

   shutdown

   hello-interval 120

   hold-time 600

  exit-sf-interface

  !

  sf-interface Tunnel12

   shutdown

   hello-interval 120

   hold-time 600

  exit-sf-interface

  !

  sf-interface VoIP-Null0

   hello-interval 120

   hold-time 600

  exit-sf-interface

  !

  sf-interface Tunnel0

   hello-interval 120

   hold-time 600

  exit-sf-interface

  !

  topology base

  exit-sf-topology

  remote-neighbors source Loopback0 unicast-listen

  neighbor 192.168.14.14 Loopback0 remote 100

  neighbor 192.168.21.21 Loopback0 remote 100

 exit-service-family

Figure 11. EIGRP SAF

Hub site BRs has EIGRP-SF peering with each other via interface Gi3. It is a direct link between BRs. EIGRP-SF has enabled on all BR interface automatically and neighbors discovery relies on link-local multicast address 224.0.0.10. Even though BRs are EIGRP-SF neighbors, they only communicate with MC. That’s why I am going to shut down the interface Gi3 on both BRs.

S1R1#sh eigrp service-family ipv4 neighbors

EIGRP-SFv4 VR(#AUTOCFG#) Service-Family Neighbors for AS(59501)

H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq

                                                   (sec)         (ms)       Cnt Num

4   192.168.14.14           Lo0                     499 06:51:54 1748  5000  0  36

3   10.11.12.12             Gi3                     528 06:51:54    1  3000  0  11

S1R2#sh eigrp service-family ipv4 neighbors

EIGRP-SFv4 VR(#AUTOCFG#) Service-Family Neighbors for AS(59501)

H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq

                                                   (sec)         (ms)       Cnt Num

4   192.168.14.14           Lo0                     540 06:58:39 1745  5000  0  37

3   10.11.12.11             Gi3                     524 06:58:39  227  1362  0  4

Figure 12. EIGRP-SF between BRs

Figure 13 shows the complete EIGRP-SF peering. These are now the channels which prefix information are learned.

Figure 13. EIGRP-SF complete peering.

Now we have to check how the Hub MC publish site-local prefixes. On the branch MC router S2R1 there are no prefix-lists since it learns local prefixes from the outgoing traffic.  If we take a look at the first site-prefix 172.16.21.21/32 which is the Loopback21 interface, we can see that it has L-flag attach to it. This means that the prefix is learned locally from the egress traffic

S2R1#sh domain WOMBAT master site-prefix 192.168.21.21

Site-id              Site-prefix          Last Updated         DC Bitmap  Flag     

--------------------------------------------------------------------------------

192.168.21.21         192.168.21.21/32     06:56:32 ago         0x0         L

192.168.21.21         10.11.21.21/32       06:56:32 ago         0x0         L

192.168.21.21         10.12.21.21/32       06:56:32 ago         0x0         L

192.168.21.21         172.16.21.21/32      05:29:58 ago         0x0         L

From the Hub MC, we can see that the same prefix is learned from site 192.168.21.21 (Branch site) via EIGRP SAF.

S1R4-MC#sh domain WOMBAT master site-prefix 192.168.21.21

Site-id              Site-prefix          Last Updated         DC Bitmap  Flag     

--------------------------------------------------------------------------------

192.168.21.21         192.168.21.21/32     01:21:48 ago         0x0         S

192.168.21.21         10.11.21.21/32       01:21:48 ago         0x0         S

192.168.21.21         10.12.21.21/32       01:21:48 ago         0x0         S

192.168.21.21         172.16.21.21/32      01:21:48 ago         0x0         S

--------------------------------------------------------------------------------

PfR Policies

EIGRP SF is also used for publishing PfRv3 policies. In previous configuration examples, I have configured following policies on Hub MC.

S1R4-MC#sh domain WOMBAT master policy

 No Policy publish pending

 Last publish Status : Peering Success

 Total publish errors : 0

--------------------------------------------------------------------------------

  class VOICE sequence 10

    path-preference MPLS fallback INET

    class type: Dscp Based

      match dscp ef policy custom

        priority 2 packet-loss-rate threshold 1.0 percent

        priority 1 one-way-delay threshold 800 msec

        priority 2 byte-loss-rate threshold 1.0 percent

  class CRITICAL sequence 20

    path-preference MPLS fallback INET

    class type: Dscp Based

      match dscp af21 policy custom

        priority 2 packet-loss-rate threshold 5.0 percent

        priority 1 one-way-delay threshold 700 msec

        priority 2 byte-loss-rate threshold 5.0 percent

And they are published to Branch MC S2R1.

S2R1#sh domain WOMBAT master policy

--------------------------------------------------------------------------------

  class VOICE sequence 10

    path-preference MPLS fallback INET

    class type: Dscp Based

      match dscp ef policy custom

        priority 2 packet-loss-rate threshold 1.0 percent

        priority 1 one-way-delay threshold 800 msec

        priority 2 byte-loss-rate threshold 1.0 percent

  class CRITICAL sequence 20

    path-preference MPLS fallback INET

    class type: Dscp Based

      match dscp af21 policy custom

        priority 2 packet-loss-rate threshold 5.0 percent

        priority 1 one-way-delay threshold 700 msec

        priority 2 byte-loss-rate threshold 5.0 percent

--------------------------------------------------------------------------------

Policies are also published to BRs as shown from the output taken from BR S1R1(only quick monitor is shown for brevity)

S1R1#sh domain WOMBAT border pmi | sec quick 

PMI[Ingress-per-DSCP-quick ]-FLOW MONITOR[MON-Ingress-per-DSCP-quick -0-48-10]

    monitor-interval:5

    key-list:

      pfr site source id ipv4

      pfr site destination id ipv4

      ip dscp

      interface input

      policy performance-monitor classification hierarchy

      pfr label identifier

    Non-key-list:

      transport packets lost rate

      transport bytes lost rate

      pfr one-way-delay

      network delay average

      transport rtp jitter inter arrival mean

      counter bytes long

      counter packets long

      timestamp absolute monitoring-interval start

    DSCP-list:

      ef-[class:CENT-Class-Ingress-DSCP-ef-0-13]

        packet-loss-rate:react_id[34]-priority[2]-threshold[1.0 percent]

        one-way-delay:react_id[46]-priority[1]-threshold[800 msec]

        network-delay-avg:react_id[47]-priority[1]-threshold[1600 msec]

        byte-loss-rate:react_id[37]-priority[2]-threshold[1.0 percent]

      af21-[class:CENT-Class-Ingress-DSCP-af21-0-14]

        packet-loss-rate:react_id[38]-priority[2]-threshold[5.0 percent]

        one-way-delay:react_id[44]-priority[1]-threshold[700 msec]

        network-delay-avg:react_id[45]-priority[1]-threshold[1400 msec]

        byte-loss-rate:react_id[41]-priority[2]-threshold[5.0 percent]

Channels

Channels are logical entities which are used for measuring the performance of the path for specific application/dscp flows between sites. What we are going to do is to send icmp packets from S2R1 (src 172.16.21.21) to S1R3 (dst 172.16.3.13) with DSCP EF.

Before starting the ping, there are no channels for DSCP EF in either Hub MC or Branch MC.

S2R1#sh domain WOMBAT master channels dscp ef

   Legend: * (Value obtained from Network delay:)

S1R4-MC#sh domain WOMBAT master channels dscp ef

   Legend: * (Value obtained from Network delay:)

Figure 15. Channels: ping from 172.16.21.21 to 172.16.3.13 with DSCP EF.

Now we take an extended ping with ToS 184 (DSCP EF).

S2R1#ping

Protocol [ip]:

Target IP address: 172.16.3.13

Repeat count [5]:

Datagram size [100]:

Timeout in seconds [2]:

Extended commands [n]: y

Ingress ping [n]:

Source address or interface: 172.16.21.21

Type of service [0]: 184

Set DF bit in IP header? [no]:

Validate reply data? [no]:

Data pattern [0x0000ABCD]:

Loose, Strict, Record, Timestamp, Verbose[none]:

Sweep range of sizes [n]:

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.3.13, timeout is 2 seconds:

Packet sent with a source address of 172.16.21.21

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 4/7/15 ms

Now there are two channels from Branch site to Hub site for traffic marked with DSCP EF, channel 38 over the link INET and Channel 37 over the link MPLS. Based on destination site-id (IP address of the Hub site MC), we know that both channels go to Hub site. First part of the pfr-Label field 0:12 (ch38 INET) and 0:11 (ch37 MPLS) shows both source site POP-ID (0) and PATH-IDs 12 and 11. The second part of the pfr-Label field shows the same information about the destination site (these are set to zeros). TCA stands for Threshold Crossing Alert <loss/delay/jitter>.

S2R1#show domain WOMBAT master channels summary | i Ch-ID|46

Ch-ID - Channel ID, SP - Service Provider

Ch-ID Dst-Site-ID     DSCP        SP    pfr-Label             Status TCA     

38    192.168.14.14   ef[46]      INET  0:12 | 0:0 [0xC0000]  A      1/0/0

37    192.168.14.14   ef[46]      MPLS  0:11 | 0:0 [0xB0000]  A      2/1/0

From the HUB MC perspective, there are two channels which terminate to Branch site (dst-ch-Id is the ip address of Branch MC).

S1R4-MC#show domain WOMBAT master channels summary | i Ch-ID|46

Ch-ID - Channel ID, SP - Service Provider

Ch-ID Dst-Site-ID     DSCP        SP    pfr-Label             Status TCA     

31    192.168.21.21   ef[46]      MPLS  0:0 | 0:11 [0xB]      A      4/2/0

32    192.168.21.21   ef[46]      INET  0:0 | 0:12 [0xC]      A      0/0/0

We can see that MPLS link is currently the primary link for traffic-class DSCP EF.

S2R1#show domain WOMBAT master traffic-classes dscp ef

 Dst-Site-Prefix: 172.16.3.0/24       DSCP: ef [46] Traffic class id:22

  Clock Time:                 00:43:43 (UTC) 03/24/2017

  TC Learned:                 00:02:12 ago

  Present State:              CONTROLLED

  Current Performance Status: in-policy

  Current Service Provider:   MPLS since 00:00:01 (hold until 88 sec)

  Previous Service Provider:  INET pfr-label: 0:12 | 0:0 [0xC0000] for 100 sec

  BW Used:                    0 Kbps

  Present WAN interface:      Tunnel11 in Border 192.168.21.21

  Present Channel (primary):  37 MPLS pfr-label:0:11 | 0:0 [0xB0000]

  Backup Channel:             38 INET pfr-label:0:12 | 0:0 [0xC0000]

  Destination Site ID bitmap: 1

  Destination Site ID:        192.168.14.14 (Active)

  Class-Sequence in use:      10

  Class Name:                 VOICE using policy User-defined

    priority 2 packet-loss-rate threshold 1.0 percent

    priority 1 one-way-delay threshold 800 msec

    priority 2 byte-loss-rate threshold 1.0 percent

  BW Updated:                 - ago

  <snipped>

Figure 16. Channels

The primary channel is created based on the actual traffic flow. Since MPLS link currently full fills requirements of traffic-class VOICE (DSCP EF) the path is selected based on the routing protocol information. Every time when the new channel is created, also the backup channel is created. PfRv3 uses Parent Route Lookup for finding a backup path(s). PfRv3 is able to look parent routes from both EIGRP and BGP tables through the direct API. This is one of the reasons, why EIGRP and BGP are preferred routing protocols over DMVPN in Cisco IWAN solution. We have two entries for network 172.16.0.0 in Branch BGP table and the route with next-hop 10.11.21.11 is selected to best route based on better a Local Preference value, and it is installed into RIB.

S2R1#sh ip bgp 

<snipped>

     Network          Next Hop            Metric LocPrf Weight Path

<snipped>

 * i  172.16.0.0/22    10.12.21.12              0   1000      0 i

 *>i                   10.11.21.11              0  10000      0 i

<snipped>

S2R1#sh ip route | i 172.16.0.0/22

B        172.16.0.0/22 [19/0] via 10.11.21.11, 20:29:33

PfRv3 searches parent routes from NHRP cache, BGP table, EIGRP table, static routes, and RIB in this order. Couple words about NHRP cache and Parent Route Lookup, when a host in one branch wants to communicate with a host on another branch site over DMVPN (Phase3) infrastructure, the automatic tunnel between site will be created and the shortcut route will appear in RIB. As soon as this happens, pfr will pick up the shortcut route and start using that. The shortcut route is described in my previous DMVPN articles.

Path Monitoring and Control

Performance Monitor (PM), which is integrated part of Cisco IOS (Figure 17), passively collects statistics on router performance. These statistics includes information about packet loss, one-way delay, jitter, IP source/destination and DSCP TC among the other things. PfR makes path decision based on this information. Site BR routers activate three Performance Monitor Instances (PMIs) on every WAN interface.

PMI 1: This PMI monitors and learns site local networks from the outgoing traffic. Because of this PMI, there is no need for statics definition about site local networks on Branch sites.

PMI 2: This PMI monitors bandwidth usage per traffic class from ingress traffic.

PMI 3: This instance collects performance metrics per traffic class (ingress direction). This PMI contains dedicated monitor for both critical (quick monitor) and noncritical traffic. The default monitor interval is 30 seconds but it can be manually reduced on the Hub MC as we have done in our example (command “monitor-interval” under domain>vrf>master hub configuration). There can only be one quick monitor for predefined DSCP. In our lab, I have set the monitor interval for both DSCP EF and AF to be 5 seconds.

Since we have minimum two paths between sites in IWAN solution, PfR can collect statistics about DSCP marked user traffic only from the active path (default class traffic can be load balanced over both links). To get also metrics from the standby path, the source MC instructs the local BR to generate smart probes over the standby path. Destination BR will measure metrics from the incoming smart probe traffic and then drops the packets. This way the system knows the quality of standby line before it reroutes traffic over it.

Figure 17. Performance monitoring

Reactive Control Plane Operation

In figure 18, we send an ICMP Query message from the Hub site router S1R3 with a source address of 172.16.3.13 and DSCP EF to destination 172.16.21.21 on router S2R1 on Branch site.

1. The packet is routed to the BR S1R1 based on routing information received from OSPF.
2. The packet is routed over the MPLS WAN connection through the Tunnel11. Since packets are marked with the DSCP EF (class VOICE), data flow triggers proactive Control Plane operations. BR S1R1 makes the forwarding decision based on the RIB and selects the BGP learned route over the MPLS WAN as a primary path. It also does a parent route lookup (NHRP cache>BGP table, EIGRP table, static routes, and RIB) for finding standby path to the destination. It has a RIB entry learned via OSPF from the BR S1R2 so it is selected as the secondary path. BR S1R1 assign channel-Id 35 to primary path (path-id 11) and channel-Id 36 to the backup path (path-id 12).
3. BR S1R1 send an EIGRP SAF message to Hub MC to inform it about the new channel (a) which in turn informs the backup BR about new channels and instructs it to generate a smart probe over the backup channel.
4. BR S1R2 starts to generate smart probes over the backup channel, which are used for monitoring the status of standby path on destination site BR.
5. BR S2R1 on the Branch site receives the actual data flow from the Hub site over the MPLS WAN (tunnel 11). Since all MC/BR in PfR Domain WOMBAT has a same Traffic Class Policy Database, BR S2R1 notices that it has to monitor this flow. It checks the ingress flow that the quality of the path 11 does not violate the threshold values defined for DSCP EF marked traffic (packet/byte loss and delay). Since the Hub BR S1R2 also sends on smart probes, BR S2R1 is able to monitor also the backup path.

S1R4-MC#sh domain WOMBAT master traffic-classes detail

 Dst-Site-Prefix: 172.16.21.21/32     DSCP: ef [46] Traffic class id:16

  Clock Time:                 05:09:09 (UTC) 03/24/2017

  TC Learned:                 00:00:38 ago

  Present State:              CONTROLLED

  Current Performance Status: in-policy

  Current Service Provider:   MPLS path-id:11 since 00:00:07 (hold until 82 sec)

  Previous Service Provider:  Unknown

  BW Used:                    0 Kbps

  Present WAN interface:      Tunnel11 in Border 192.168.11.11

  Present Channel (primary):  35 MPLS pfr-label:0:0 | 0:11 [0xB]

  Backup Channel:             36 INET pfr-label:0:0 | 0:12 [0xC]

  Destination Site ID bitmap: 0

  Destination Site ID:        192.168.21.21

  Class-Sequence in use:      10

  Class Name:                 VOICE using policy User-defined

    priority 2 packet-loss-rate threshold 1.0 percent

    priority 1 one-way-delay threshold 800 msec

    priority 2 byte-loss-rate threshold 1.0 percent

  <snipped>

Figure 18. Proactive Control Plane operation in Hub Site.

6. BR S2R1 sends an ICMP reply over the MPLS WAN interface. This triggers the same events that we saw on Hub BR S1R1 when it forwards IP-packets marked with DSCP EF. All BRs also monitors outgoing data flows that belongs to the predefined Traffic Class and check the BW usage per TC. BRs also learns site-local prefixes from the outgoing flows. As in the case of Hub BR S1R2, the BR S2R1 also starts to generate smart probes over the backup path to BR S1R2. Instead of sending smart probes to the actual destination, they are sent to the BR of the destination site, over the backup path which in turn does the monitoring and then silently drops the packets.

S2R1#sh domain WOMBAT master traffic-classes detail

 Dst-Site-Prefix: 172.16.3.0/24       DSCP: ef [46] Traffic class id:24

  Clock Time:                 05:10:12 (UTC) 03/24/2017

  TC Learned:                 00:01:41 ago

  Present State:              CONTROLLED

  Current Performance Status: in-policy

  Current Service Provider:   MPLS since 00:01:10 (hold until 19 sec)

  Previous Service Provider:  Unknown

  BW Used:                    0 Kbps

  Present WAN interface:      Tunnel11 in Border 192.168.21.21

  Present Channel (primary):  41 MPLS pfr-label:0:11 | 0:0 [0xB0000]

  Backup Channel:             42 INET pfr-label:0:12 | 0:0 [0xC0000]

  Destination Site ID bitmap: 1

  Destination Site ID:        192.168.14.14 (Active)

  Class-Sequence in use:      10

  Class Name:                 VOICE using policy User-defined

    priority 2 packet-loss-rate threshold 1.0 percent

    priority 1 one-way-delay threshold 800 msec

    priority 2 byte-loss-rate threshold 1.0 percent

Figure 19. Proactive Control Plane operation in Branch Site.

Threshold Crossing Alert - TCA

So what happens when the path quality violates to given threshold for specific Traffic Class. In figure 20, the Policy VOICE is violated since the one-way delay is higher than the defined threshold 800ms. For the reaction to this event, BR S2R1 generates the Threshold Crossing Alert and notifies local MC which in turns sends TCA messages with EIGRP SAF to the Hub MC. For redundancy, the TCA message is sent over both paths.

As can be seen from the S1R4 syslog messages, Hub MC receives TCA from the Branch MC (TCA-origin) regarding the policy violation (one-way-delay) of the class VOICE from the site 192.168.14.14 (Hub) to the site 192.168.21.21 (Branch). We can also see that the current exit point MPLS.

Note! The syslog text is divided into different rows to facilitate reading.

S1R4-MC#

%DOMAIN-5-TCA: TCA Received.

Details:

Instance id=0:

VRF=default:

Source Site ID=192.168.14.14:

Destination Site ID=192.168.21.21:

TCA-ID=19:

TCA-Origin=192.168.21.21(R):

Exit=[CHAN-ID=35, BR-IP=192.168.11.11, DSCP=ef[46], Interface=Tunnel11, Path=MPLS[label=0:0 | 0:11 [0xB]]]:

Policy Violated=VOICE:

thresholds(actual/config)=[P1=one-way-delay(847/800)]

Figure 20. Threshold Crossing Alert notification

For the reaction to this event, the Hub MC instructs the BR S1R1 to reroute data packets, belonging to this Traffic Class, over the path INET. Note that site internal routing protocol OSPF still chooses the MPLS tunnel as the best path. So the PfR overrides the routing decision based on routing protocols.

%DOMAIN-5-TC_PATH_CHG: Traffic class Path Changed.

Details:

Instance=0:

VRF=default:

Source Site ID=192.168.14.14:

Destination Site ID=192.168.21.21:

Reason=Delay:

TCA-ID=20:

Policy Violated=VOICE:

TC=[Site id=192.168.21.21, TC ID=16, Site prefix=172.16.21.21/32, DSCP=ef(46), App ID=0]:

Original Exit=[CHAN-ID=35, BR-IP=192.168.11.11, DSCP=ef[46], Interface=Tunnel11, Path=MPLS[label=0:0 | 0:11 [0xB]]]:

New Exit=[CHAN-ID=36, BR-IP=192.168.12.12, DSCP=ef[46], Interface=Tunnel12, Path=INET[label=0:0 | 0:12 [0xC]]]

Figure 21. Threshold Crossing Alert reaction.

Now we can see that Traffic Class for DSCF EF uses INET as an Exit Path.

S1R4-MC#sh domain WOMBAT master traffic-classes detail

 Dst-Site-Prefix: 172.16.21.21/32     DSCP: ef [46] Traffic class id:16

  Clock Time:                 05:12:08 (UTC) 03/24/2017

  TC Learned:                 00:03:38 ago

  Present State:              CONTROLLED

  Current Performance Status: in-policy

  Current Service Provider:   INET path-id:12 since 00:00:15 (hold until 74 sec)

  Previous Service Provider:  MPLS pfr-label: 0:0 | 0:11 [0xB] for 172 sec

  (A fallback provider. Primary provider will be re-evaluated 00:02:45 later)

  BW Used:                    0 Kbps

  Present WAN interface:      Tunnel12 in Border 192.168.12.12

  Present Channel (primary):  36 INET pfr-label:0:0 | 0:12 [0xC]

  Backup Channel:             35 MPLS pfr-label:0:0 | 0:11 [0xB]

  Destination Site ID bitmap: 0

  Destination Site ID:        192.168.21.21

  Class-Sequence in use:      10

  Class Name:                 VOICE using policy User-defined

    priority 2 packet-loss-rate threshold 1.0 percent

    priority 1 one-way-delay threshold 800 msec

    priority 2 byte-loss-rate threshold 1.0 percent

One last thing to do. Figure 22 represents the main components and their relationships in the PfRv3 domain.

Figure 22. PfR Components and relationships.

And that's it.

Reference:

Cisco Intelligent WAN (iWAN) – ISBN-10: 1-58714-463-8

Brad Edgeworth, Jean Marc Barozet, David Prall, Anthony Lockhart, Nir Ben-Dvora

Appendix 1.
Device complete configurations

************************************************** ********* COMPLET CONFIGURATION OF S1R1 ********** ************************************************** S1R1#sh run Building configuration... Current configuration : 7257 bytes ! ! Last configuration change at 08:27:46 UTC Fri Mar 24 2017 ! version 16.4 service config service timestamps debug datetime msec service timestamps log datetime msec no platform punt-keepalive disable-kernel-core platform console serial ! hostname S1R1 ! boot-start-marker boot-end-marker ! ! vrf definition MPLS  !  address-family ipv4  exit-address-family ! enable secret 5 $1$bxgA$LvvAIBTilA3StgE2aTIa71 ! no aaa new-model ! ! ! ! ! ! ! ! ! no ip domain lookup ! ! ! ! ! ! ! !         ! ! subscriber templating ! ! ! multilink bundle-name authenticated ! domain WOMBAT  vrf default   border    source-interface Loopback0    master 192.168.14.14    password cisco ! ! ! ! crypto pki trustpoint TP-self-signed-3203326600  enrollment selfsigned  subject-name cn=IOS-Self-Signed-Certificate-3203326600  revocation-check none  rsakeypair TP-self-signed-3203326600 ! crypto pki trustpoint TP-self-signed-2311614653  enrollment selfsigned  subject-name cn=IOS-Self-Signed-Certificate-2311614653  revocation-check none  rsakeypair TP-self-signed-2311614653 ! ! crypto pki certificate chain TP-self-signed-3203326600  certificate self-signed 01   30820330 30820218 A0030201 02020101 300D0609 2A864886 F70D0101 05050030   31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274   69666963 6174652D 33323033 33323636 3030301E 170D3137 30333233 30323331   32305A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649   4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D33 32303333   32363630 30308201 22300D06 092A8648 86F70D01 01010500 0382010F 00308201   0A028201 0100CA8A 10460F45 A557C915 68BF060D 4E6E0903 FB9E1A18 54F962FB   18F3D9DC F945967A 7190D63A 6123C93F C3857785 8174EE23 1D3FF374 DF4EA0B7   8D444B16 76CD353D 56DFD09E EEDB1E81 2CDAEB90 02F48FA6 633368D1 730E0F24   8AD1B3B0 9FC81028 78FCC0B6 F185ACC4 FD815573 2A3C7A88 A3772AEF 1AD5D93F   81E8D1CD A0D21350 01367E44 FFD79ADA A02D10CB 0C6DF8BE 366A4483 10B2A1E4   1A92925C 4D1D995A 3983167C 1296CC84 52496838 2C638465 EE7F9F2C 7D36F899   01F5A4FE 4C1E640D FAFB4C44 4119AF1B 87BA15AB 2C56734B 7DD03F93 0419ED57   C9A911AB 8CD15E6F 5659C45E B1800449 AF66D344 2ADD8146 50BA42AC 86093C02   04D26279 F3BF0203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF   301F0603 551D2304 18301680 1492338E 433CF52C 103443CB 9E9371A4 C29CA6A4   22301D06 03551D0E 04160414 92338E43 3CF52C10 3443CB9E 9371A4C2 9CA6A422   300D0609 2A864886 F70D0101 05050003 82010100 5895E33A 06152C61 2A66A02D   D13FB097 538FF34E 1283A896 77AB84D1 44702D43 3E66FA7C 2E3B75D9 9E40E5B0   CB803D3A 598E8B28 1E01784A F4A5549C 06F8775E 382860A1 1C217FD1 0E111A7D   F7731FAC 1B636DA1 FB282130 53CA7E8B B9B6E346 3106C2BF 98A93DDF 34BD603F   E90B20E7 F2B36F2C CD95924E 3A40E731 45935E07 5E24D528 5BEA34C8 18B6E54F   3601C288 5913E229 B330272F 3B9F4B8D A3F66626 E82AA39F 39696668 643B027D   D39D5680 55305C54 19C5B8C1 BE114B60 102DDFCA C9F037A2 BF2A08EF DB8D1A3E   C9EDEA4D 10ADB033 A4DF93C2 35FE427C 942523FA 765D54D8 1CA04F79 93F0F997   3F862EC2 E3B6DC38 CD82915C 34D1A939 CD765B7B         quit crypto pki certificate chain TP-self-signed-2311614653 ! ! ! ! ! !         ! license udi pid CSR1000V sn 9CPWES24P7L diagnostic bootup level minimal ! spanning-tree extend system-id ! ! username cisco privilege 15 secret 5 $1$GMLK$ygnXRZxJuyfrqnY9syczR/ ! redundancy ! ! ! ! ! ! track 3 ip route 192.168.13.13 255.255.255.255 reachability ! ! ! ! ! !         ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! interface Loopback0  ip address 192.168.11.11 255.255.255.255  ip ospf network point-to-point  ip ospf 1 area 0 ! interface Tunnel11  ip address 10.11.21.11 255.255.255.0  no ip redirects  ip mtu 1400  ip nhrp network-id 100  ip nhrp redirect  ip tcp adjust-mss 1360  tunnel source GigabitEthernet2  tunnel mode gre multipoint  tunnel key 100  tunnel path-mtu-discovery  tunnel vrf MPLS  domain WOMBAT path MPLS path-id 11 ! interface GigabitEthernet1  ip address 10.255.0.63 255.255.0.0  negotiation auto  no mop enabled  no mop sysid ! interface GigabitEthernet2  vrf forwarding MPLS  ip address 10.11.111.11 255.255.255.0  negotiation auto  no mop enabled  no mop sysid ! interface GigabitEthernet3  ip address 10.11.12.11 255.255.255.0  ip ospf 1 area 0  negotiation auto  no mop enabled  no mop sysid ! interface GigabitEthernet4  ip address 10.11.13.11 255.255.255.0  ip ospf 1 area 0  negotiation auto  no mop enabled  no mop sysid ! router ospf 1  redistribute bgp 64501 subnets route-map BGP-to-OSPF ! router bgp 64501  bgp log-neighbor-changes  bgp listen range 10.11.21.0/24 peer-group SPOKE-ROUTERS  bgp listen limit 10  neighbor SPOKE-ROUTERS peer-group  neighbor SPOKE-ROUTERS remote-as 64501  neighbor SPOKE-ROUTERS timers 20 60  neighbor 192.168.12.12 remote-as 64501  neighbor 192.168.12.12 update-source Loopback0  !  address-family ipv4   bgp redistribute-internal   network 0.0.0.0   network 172.16.0.0 mask 255.255.252.0   network 192.168.14.14 mask 255.255.255.255   neighbor SPOKE-ROUTERS activate   neighbor SPOKE-ROUTERS send-community   neighbor SPOKE-ROUTERS route-reflector-client   neighbor SPOKE-ROUTERS next-hop-self all   neighbor SPOKE-ROUTERS soft-reconfiguration inbound   neighbor SPOKE-ROUTERS route-map FROM-MPLS-SPOKE in   neighbor SPOKE-ROUTERS route-map TO-MPLS-SPOKE out   neighbor 192.168.12.12 activate   neighbor 192.168.12.12 route-reflector-client   neighbor 192.168.12.12 next-hop-self   distance bgp 200 19 19  exit-address-family ! ! virtual-service csr_mgmt ! ip forward-protocol nd no ip http server ip http authentication local no ip http secure-server ! ip bgp-community new-format ip route 172.16.0.0 255.255.252.0 Null0 254 track 3 ip route vrf MPLS 10.21.111.21 255.255.255.255 10.11.111.111 ! ! ! ip prefix-list DEAFULT-ROUTE seq 10 permit 0.0.0.0/0 ! ip prefix-list DMVPN-T11 seq 10 permit 10.11.21.0/24 ! ip prefix-list DMVPN-T12 seq 10 permit 10.12.21.0/24 ! ip prefix-list SET-WEIGHT-50 seq 5 permit 192.168.21.21/32 !         ip prefix-list SITE1-INTERNAL-PREFIX seq 10 permit 172.16.0.0/22 ip prefix-list SITE1-INTERNAL-PREFIX seq 20 permit 192.168.14.14/32 no service-routing capabilities-manager ! ! route-map BGP-to-OSPF deny 10  match ip address prefix-list SITE1-INTERNAL-PREFIX ! route-map BGP-to-OSPF deny 30  match ip address prefix-list DEAFULT-ROUTE ! route-map BGP-to-OSPF permit 60  set metric 100  set metric-type type-1 ! route-map SET-WEIGHT-50 permit 20 ! route-map FROM-MPLS-SPOKE deny 10  match ip address prefix-list SITE1-INTERNAL-PREFIX ! route-map FROM-MPLS-SPOKE deny 30  match ip address prefix-list DEAFULT-ROUTE !         route-map FROM-MPLS-SPOKE deny 40  match ip address prefix-list DMVPN-T11 ! route-map FROM-MPLS-SPOKE deny 50  match ip address prefix-list DMVPN-T12 ! route-map FROM-MPLS-SPOKE permit 60 ! route-map TO-MPLS-SPOKE permit 10  match ip address prefix-list SITE1-INTERNAL-PREFIX  set local-preference 10000  set community 64501:11 ! route-map TO-MPLS-SPOKE permit 20  match ip address prefix-list DEAFULT-ROUTE  set local-preference 10000  set community 64501:111 ! route-map SET-WEIGHT-100 permit 10  match ip address prefix-list SET-WEIGHT-100  set weight 100 ! !         ! control-plane !  !  !  !  ! ! ! ! ! ! line con 0  stopbits 1 line vty 0  login line vty 1  login  length 0 line vty 2 4  login ! !         ! ! ! ! ! end S1R1# ************************************************** ********* COMPLET CONFIGURATION OF S1R2 ********** ************************************************** S1R2#sh run Building configuration... Current configuration : 7059 bytes ! ! Last configuration change at 08:25:38 UTC Fri Mar 24 2017 ! version 16.4 service config service timestamps debug datetime msec service timestamps log datetime msec no platform punt-keepalive disable-kernel-core platform console serial ! hostname S1R2 ! boot-start-marker boot-end-marker ! ! vrf definition INET  !  address-family ipv4  exit-address-family ! enable secret 5 $1$mzTo$ujtktGx4Jo9vgF7NBdCr6/ ! no aaa new-model ! ! ! ! ! ! ! ! ! no ip domain lookup ! ! ! ! ! ! ! !         ! ! subscriber templating ! ! ! multilink bundle-name authenticated ! domain WOMBAT  vrf default   border    source-interface Loopback0    master 192.168.14.14    password cisco ! ! ! ! crypto pki trustpoint TP-self-signed-1949394005  enrollment selfsigned  subject-name cn=IOS-Self-Signed-Certificate-1949394005  revocation-check none  rsakeypair TP-self-signed-1949394005 ! crypto pki trustpoint TP-self-signed-1886518991  enrollment selfsigned  subject-name cn=IOS-Self-Signed-Certificate-1886518991  revocation-check none  rsakeypair TP-self-signed-1886518991 ! ! crypto pki certificate chain TP-self-signed-1949394005  certificate self-signed 01   30820330 30820218 A0030201 02020101 300D0609 2A864886 F70D0101 05050030   31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274   69666963 6174652D 31393439 33393430 3035301E 170D3137 30333233 30323331   34385A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649   4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D31 39343933   39343030 35308201 22300D06 092A8648 86F70D01 01010500 0382010F 00308201   0A028201 0100A740 434B675A 15DFB2EB 3A1E696B C4699531 F7CFA7F5 841D8D39   E8018EF7 05CF343C 7B1DDC59 56BFF52C 41B25814 333CA591 A61C0EAE CE806840   EC95ACC5 0D7609AA 5164AA90 C44B1B72 30FC8AA9 10A2C9C5 6C93E7C2 2BF29D6A   077BB040 1624073E 463D1369 4D19F90B 158DD1C6 02E04662 115EB1E7 36CC7EDF   948F6BBB 5DA6575D 0657E263 06DB3769 145C299A 307FEEA8 327B4931 16E36E9D   11A3C532 FAB4DADC 0C1FE2BA E97262BB 507FDC8C DCD917A8 480AD651 56CF0B8E   DE588FB9 74A763DA 364C5A28 1650D9F4 ECD143FA 882A8B1F 991137D3 FB0E51F4   3374EE39 7C4BBB3F 45CE9B0B CAB3A12B D393C200 9C77C0A3 E1A4876E 7AD7A4E8   08606544 506F0203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF   301F0603 551D2304 18301680 1484ED6B 8986B469 7B4F5434 90894571 ECAEF525   5A301D06 03551D0E 04160414 84ED6B89 86B4697B 4F543490 894571EC AEF5255A   300D0609 2A864886 F70D0101 05050003 82010100 127EF969 E87A2FCA C1A41003   F0F5E1FB 687F5890 405323A3 35D92493 B3CB3238 B9F5028E 4E2669AD A0E286D2   88EE6FB5 2BAC8D78 B51F6B6E CE01910E F25C6AD8 6A309A3C 1458F5BC F8582586   440D60D3 FBE88BF9 AA11E51A 295B5861 45617B7F F80BBA98 2791D73D 335B7F01   804BD531 64304424 5B228856 A1EE4D13 D21B211F 215BCAF1 446CFC37 9A23DD66   1D86A678 6E2A62B7 BD6BD6C3 1F5CCAA0 CD150BCC DB12A2AC 983AE4A2 6BDA84F8   2D142780 1E939858 764AB820 427810F3 4D4B3DEF 49E8BE06 C2ED0A68 374DA36C   DAE38407 109133A0 F7DE18CC 2064530E DF7E2A2E E8D11AD5 AE0C518D F4286E41   B82FB68F FEEBFDBF D7392F1B 42303242 0DD903D3         quit crypto pki certificate chain TP-self-signed-1886518991 ! ! ! ! ! !         ! license udi pid CSR1000V sn 9FN5SB1JJZB diagnostic bootup level minimal ! spanning-tree extend system-id ! ! username cisco privilege 15 secret 5 $1$MVN0$1BU050LoS.iPDRMuWE/D7. ! redundancy ! ! ! ! ! ! track 3 ip route 192.168.13.13 255.255.255.255 reachability ! ! ! ! ! !         ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! interface Loopback0  ip address 192.168.12.12 255.255.255.255  ip ospf network point-to-point  ip ospf 1 area 0 ! interface Tunnel12  ip address 10.12.21.12 255.255.255.0  no ip redirects  ip mtu 1400  ip nhrp network-id 101  ip nhrp redirect  ip tcp adjust-mss 1360  tunnel source GigabitEthernet2  tunnel mode gre multipoint  tunnel key 101  tunnel path-mtu-discovery  tunnel vrf INET  domain WOMBAT path INET path-id 12 ! interface GigabitEthernet1  ip address 10.255.0.64 255.255.0.0  negotiation auto  no mop enabled  no mop sysid ! interface GigabitEthernet2  vrf forwarding INET  ip address 10.12.112.12 255.255.255.0  negotiation auto  no mop enabled  no mop sysid ! interface GigabitEthernet3  ip address 10.11.12.12 255.255.255.0  ip ospf 1 area 0  shutdown  negotiation auto  no mop enabled  no mop sysid ! interface GigabitEthernet4  ip address 10.12.13.12 255.255.255.0  ip ospf 1 area 0  negotiation auto  no mop enabled  no mop sysid ! router ospf 1  redistribute bgp 64501 subnets route-map BGP-to-OSPF ! router bgp 64501  bgp log-neighbor-changes  bgp listen range 10.12.21.0/24 peer-group SPOKE-ROUTERS  bgp listen limit 10  neighbor SPOKE-ROUTERS peer-group  neighbor SPOKE-ROUTERS remote-as 64501  neighbor SPOKE-ROUTERS timers 20 60  neighbor 192.168.11.11 remote-as 64501  neighbor 192.168.11.11 update-source Loopback0  !  address-family ipv4   bgp redistribute-internal   network 0.0.0.0   network 172.16.0.0 mask 255.255.252.0   network 192.168.14.14 mask 255.255.255.255   neighbor SPOKE-ROUTERS activate   neighbor SPOKE-ROUTERS send-community   neighbor SPOKE-ROUTERS route-reflector-client   neighbor SPOKE-ROUTERS next-hop-self all   neighbor SPOKE-ROUTERS soft-reconfiguration inbound   neighbor SPOKE-ROUTERS route-map FROM-INET-SPOKE in   neighbor SPOKE-ROUTERS route-map TO-INET-SPOKE out   neighbor 192.168.11.11 activate   neighbor 192.168.11.11 route-reflector-client   neighbor 192.168.11.11 next-hop-self   distance bgp 200 19 19  exit-address-family ! ! virtual-service csr_mgmt ! ip forward-protocol nd no ip http server ip http authentication local no ip http secure-server ! ip bgp-community new-format ip route 172.16.0.0 255.255.252.0 Null0 254 track 3 ip route vrf INET 10.21.112.21 255.255.255.255 10.12.112.112 ! ! ! ip prefix-list DEAFULT-ROUTE seq 10 permit 0.0.0.0/0 ! ip prefix-list DMVPN-T11 seq 10 permit 10.11.21.0/24 ! ip prefix-list DMVPN-T12 seq 10 permit 10.12.21.0/24 ! ip prefix-list SITE1-INTERNAL-PREFIX seq 10 permit 172.16.0.0/22 ip prefix-list SITE1-INTERNAL-PREFIX seq 20 permit 192.168.14.14/32 no service-routing capabilities-manager ! ! route-map FROM-INET-SPOKE deny 10  match ip address prefix-list SITE1-INTERNAL-PREFIX ! route-map FROM-INET-SPOKE deny 30  match ip address prefix-list DEAFULT-ROUTE ! route-map FROM-INET-SPOKE deny 40  match ip address prefix-list DMVPN-T11 ! route-map FROM-INET-SPOKE deny 50  match ip address prefix-list DMVPN-T12 ! route-map FROM-INET-SPOKE permit 60 ! route-map BGP-to-OSPF deny 10  match ip address prefix-list SITE1-INTERNAL-PREFIX ! route-map BGP-to-OSPF deny 30  match ip address prefix-list DEAFULT-ROUTE ! route-map BGP-to-OSPF permit 60  set metric 1000  set metric-type type-1 ! route-map TO-INET-SPOKE permit 10  match ip address prefix-list SITE1-INTERNAL-PREFIX  set local-preference 1000  set community 64501:12 ! route-map TO-INET-SPOKE permit 20  match ip address DEAFULT-ROUTE  set local-preference 1000  set community 64501:112 ! ! ! control-plane !  !  !  !  !        ! ! ! ! ! line con 0  stopbits 1 line vty 0  login line vty 1  login  length 0 line vty 2 4  login ! ! ! ! ! ! ! end           S1R2# ************************************************** ********* COMPLET CONFIGURATION OF S1R3 ********** ************************************************** S1R3#sh run Building configuration... Current configuration : 3654 bytes ! ! Last configuration change at 02:39:56 UTC Thu Mar 23 2017 ! version 15.6 service timestamps debug datetime msec service timestamps log datetime msec no service password-encryption ! hostname S1R3 ! boot-start-marker boot-end-marker ! ! ! no aaa new-model ethernet lmi ce ! ! ! mmi polling-interval 60 no mmi auto-configure no mmi pvc mmi snmp-timeout 180 ! ! ! ! ! ! ! ! ! ! ! ip cef no ipv6 cef ! multilink bundle-name authenticated ! ! ! ! ! redundancy ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! interface Loopback0  ip address 192.168.13.13 255.255.255.255  ip ospf network point-to-point  ip ospf 1 area 0 ! interface Loopback160  ip address 172.16.0.13 255.255.255.255  ip ospf network point-to-point  ip ospf 1 area 0 ! interface Loopback161  ip address 172.16.1.13 255.255.255.0  ip ospf 1 area 0 ! interface Loopback162  ip address 172.16.2.13 255.255.255.0  ip ospf 1 area 0 ! interface Loopback163  ip address 172.16.3.13 255.255.255.0  ip ospf 1 area 0 ! interface Loopback1000  ip address 10.28.158.1 255.255.255.0  ip ospf network point-to-point ! interface GigabitEthernet0/0  ip address 10.255.0.67 255.255.0.0  shutdown  duplex auto  speed auto  media-type rj45 ! interface GigabitEthernet0/1  ip address 10.11.13.13 255.255.255.0  ip ospf 1 area 0  duplex auto  speed auto  media-type rj45 ! interface GigabitEthernet0/2  ip address 10.12.13.13 255.255.255.0  ip ospf 1 area 0  duplex auto  speed auto  media-type rj45 ! interface GigabitEthernet0/3  ip address 10.14.13.13 255.255.255.0  ip ospf 1 area 0  duplex auto  speed auto  media-type rj45 !         router ospf 1  default-information originate always ! ip forward-protocol nd ! ! no ip http server no ip http secure-server ! ! ! ! control-plane ! banner exec ^C ************************************************************************** * IOSv is strictly limited to use for evaluation, demonstration and IOS  * * education. IOSv is provided as-is and is not supported by Cisco's      * * Technical Advisory Center. Any use or disclosure, in whole or in part, * * of the IOSv Software or Documentation to any third party for any       * * purposes is expressly prohibited except as otherwise authorized by     * * Cisco in writing.                                                      * **************************************************************************^C banner incoming ^C ************************************************************************** * IOSv is strictly limited to use for evaluation, demonstration and IOS  * * education. IOSv is provided as-is and is not supported by Cisco's      * * Technical Advisory Center. Any use or disclosure, in whole or in part, * * of the IOSv Software or Documentation to any third party for any       * * purposes is expressly prohibited except as otherwise authorized by     * * Cisco in writing.                                                      * **************************************************************************^C banner login ^C ************************************************************************** * IOSv is strictly limited to use for evaluation, demonstration and IOS  * * education. IOSv is provided as-is and is not supported by Cisco's      * * Technical Advisory Center. Any use or disclosure, in whole or in part, * * of the IOSv Software or Documentation to any third party for any       * * purposes is expressly prohibited except as otherwise authorized by     * * Cisco in writing.                                                      * **************************************************************************^C ! line con 0 line aux 0 line vty 0 4  login     transport input none ! no scheduler allocate ! end S1R3#    ************************************************** ********* COMPLET CONFIGURATION OF S1R4 ********** ************************************************** S1R4-MC#sh run Building configuration... Current configuration : 2209 bytes ! ! Last configuration change at 18:33:21 UTC Thu Mar 23 2017 ! version 16.4 service config service timestamps debug datetime msec service timestamps log datetime msec no platform punt-keepalive disable-kernel-core platform console serial ! hostname S1R4-MC ! boot-start-marker boot-end-marker ! ! ! no aaa new-model ! ! ! !         ! ! ! ! ! no ip domain lookup ! ! ! ! ! ! ! ! ! ! subscriber templating ! ! !         multilink bundle-name authenticated ! domain WOMBAT  vrf default   master hub    source-interface Loopback0    site-prefixes prefix-list SITE-PREFIX-PFR    password cisco    monitor-interval 5 dscp af21    monitor-interval 5 dscp ef    enterprise-prefix  prefix-list ENTERPRISE-PREFIX    class VOICE sequence 10     match dscp ef policy custom       priority 2 loss threshold 1       priority 1 one-way-delay threshold 800      path-preference MPLS fallback INET    class CRITICAL sequence 20     match dscp af21 policy custom       priority 2 loss threshold 5       priority 1 one-way-delay threshold 700      path-preference MPLS fallback INET ! !         ! ! ! ! ! ! ! ! ! ! license udi pid CSR1000V sn 9APRW3TTG40 diagnostic bootup level minimal ! spanning-tree extend system-id ! ! ! redundancy ! ! !         ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !         ! interface Loopback0  ip address 192.168.14.14 255.255.255.255  ip ospf network point-to-point  ip ospf 1 area 0 ! interface GigabitEthernet1  ip address dhcp  negotiation auto  no mop enabled  no mop sysid ! interface GigabitEthernet2  ip address 10.14.13.14 255.255.255.0  ip ospf 1 area 0  negotiation auto  no mop enabled  no mop sysid ! router ospf 1 ! ! virtual-service csr_mgmt ! ip forward-protocol nd no ip http server no ip http secure-server ! ! ! ! ip prefix-list ENTERPRISE-PREFIX seq 5 permit 172.16.0.0/16 ! ip prefix-list SITE-PREFIX seq 5 permit 172.16.0.0/22 ! ip prefix-list SITE-PREFIX-PFR seq 5 permit 172.16.0.0/23 ip prefix-list SITE-PREFIX-PFR seq 10 permit 172.16.2.0/24 ip prefix-list SITE-PREFIX-PFR seq 15 permit 172.16.3.0/24 no service-routing capabilities-manager ! ! ! ! control-plane !  !         !  !  ! ! ! ! ! ! line con 0  stopbits 1 line vty 0  login line vty 1  login  length 0 line vty 2 4  login ! ! ! ! ! !         end S1R4-MC# ************************************************** ********* COMPLET CONFIGURATION OF S2R1 ********** ************************************************** S2R1#sh run Building configuration... Current configuration : 9539 bytes ! ! Last configuration change at 06:54:21 UTC Fri Mar 24 2017 ! version 16.4 service config service timestamps debug datetime msec service timestamps log datetime msec no platform punt-keepalive disable-kernel-core platform console serial ! hostname S2R1 ! boot-start-marker boot-end-marker ! ! vrf definition INET  !  address-family ipv4  exit-address-family ! vrf definition MPLS  !  address-family ipv4  exit-address-family ! enable secret 5 $1$X/QH$/pzXcjSZ0YCld0.uDxVNw. ! no aaa new-model ! ! ! ! ! ! ! ! ! ! ! ! !         ! ! ! ! ! ! subscriber templating ! ! ! multilink bundle-name authenticated ! domain WOMBAT  vrf default   border    source-interface Loopback0    master local    password cisco   master branch    source-interface Loopback0    password cisco    hub 192.168.14.14 !         ! ! ! crypto pki trustpoint TP-self-signed-3397518362  enrollment selfsigned  subject-name cn=IOS-Self-Signed-Certificate-3397518362  revocation-check none  rsakeypair TP-self-signed-3397518362 ! crypto pki trustpoint TP-self-signed-4049959848  enrollment selfsigned  subject-name cn=IOS-Self-Signed-Certificate-4049959848  revocation-check none  rsakeypair TP-self-signed-4049959848 ! ! crypto pki certificate chain TP-self-signed-3397518362  certificate self-signed 01   30820330 30820218 A0030201 02020101 300D0609 2A864886 F70D0101 05050030   31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274   69666963 6174652D 33333937 35313833 3632301E 170D3137 30333233 30323331   34355A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649   4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D33 33393735   31383336 32308201 22300D06 092A8648 86F70D01 01010500 0382010F 00308201   0A028201 01009D06 36A53C91 5840A93D 1FA12DE1 6DD3788B 097D1EBB E55BBCE3   320B5F0D 305D66DE DB9AF8B6 C230DED4 E17DCC20 F9BB3455 28D6A958 1E27CDA3   D56981A5 D7FFB8AB 4B7C7D81 AE380169 00BF7B1B 9685C1EF F991A40C 14BFF10F   9F57A6A1 890360E7 E1CE8979 5F3446C4 35A36216 8C41206E BD3C4114 14A3A49E   130729E5 16084B4F A738E4BA D73E8ABC 95AB667C 9E836ECA 5833D004 A2AAF553   35C89E7A 0A52F80A F87EFE09 72B181FB 708954D5 41007F70 FBE6C5F6 AFAF32E2   84EDF2EE 6532004A 03950229 434677CE EBAFC0ED 4AF5BAB3 494B707E 16F7BC55   F31854F3 46433352 AE0A26E8 26A7CE34 45524537 5E05E0C3 914B4DE2 8601CEA1   D2AAD426 69FD0203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF   301F0603 551D2304 18301680 148FEE91 F0FF97D5 AC45AD3F E221CDBC 65F55100   55301D06 03551D0E 04160414 8FEE91F0 FF97D5AC 45AD3FE2 21CDBC65 F5510055   300D0609 2A864886 F70D0101 05050003 82010100 726B00D3 85C14ED9 49123108   BCD5210F 2169F9F7 3317F0EF B82C0E23 1CAD289C 2EA2066F 8B99B76C 70210B9B   BCE1E6F7 0EF510E9 A39FE2E6 F853F4CC 33B7A0DC 9EE780CB AC98DADE 561C241D   D106C8AC 5547ACD6 CF262B7C 01DAC9BC F47628D8 1540BDF4 30139C7E 1B36E8FE   E0B42079 3A983E6B 8A09EFF1 0E5D11D5 91AF2EE5 CF76D467 FA815E91 B0EE12F0   609B95BD 902BF076 685DE84D AF8A1E51 97112155 28C623B1 70CFEDD3 652E6CEA   E641EE96 1C0C5483 E7F2DE04 18CFA5C2 1633C0A4 A159941C 8BB6C9AD 8037EFA0   E36C3B58 49672145 63534B4D 271EF69B 4E6055C6 F4511C02 11F0153B 1E361891   C0C42F2C FF354A7C 698279DE AE9D1C9C 9FD83E2E         quit crypto pki certificate chain TP-self-signed-4049959848 ! ! ! ! ! ! ! license udi pid CSR1000V sn 98ZFG4S9SYF diagnostic bootup level minimal ! spanning-tree extend system-id ! ! username cisco privilege 15 secret 5 $1$w1WJ$U/JAhcqV2Y5UgD/.g/d5G. ! redundancy ! ! ! ! !         ! track 12 list threshold percentage  object 121  object 122  object 123  threshold percentage down 40 up 60  delay up 15 ! track 121 interface Tunnel12 line-protocol ! track 122 ip sla 122 reachability ! track 123 ip sla 123 reachability ! ! class-map type inspect match-any CM-INSPECT-ROUTER-OUT  match access-group name ACL-ROUTER-OUT class-map type inspect match-any CM-PASS-ROUTER-IN  match access-group name ACL-ESP-IN  match access-group name ACL-DHCP-IN  match access-group name ACL-GRE-IN class-map type inspect match-any CM-IN2OUT  match protocol tcp  match protocol udp  match protocol icmp class-map type inspect match-any CM-PASS-ROUTER-OUT  match access-group name ACL-ESP-OUT  match access-group name ACL-DHCP-OUT class-map type inspect match-any CM-INSPECT-ROUTER-IN  match access-group name ACL-ROUTER-IN ! policy-map type inspect PM-OUT-POLICY  class type inspect CM-INSPECT-ROUTER-OUT   inspect  class type inspect CM-PASS-ROUTER-OUT   pass  class class-default   drop policy-map type inspect PM-IN2OUT  class type inspect CM-IN2OUT   inspect  class class-default policy-map type inspect PM-IN-POLICY  class type inspect CM-INSPECT-ROUTER-IN   inspect  class type inspect CM-PASS-ROUTER-IN   pass  class class-default   drop ! zone security default  description ** Default Zone for all unassigned Interfaces ** zone security OUTSIDE  description ** Iternet Interface ** zone-pair security DEFAULT-TO-OUTSIDE source default destination OUTSIDE  service-policy type inspect PM-IN2OUT zone-pair security FROM-ROUTER source self destination OUTSIDE  service-policy type inspect PM-OUT-POLICY zone-pair security ZP-TO-ROUTER source OUTSIDE destination self  service-policy type inspect PM-IN-POLICY ! ! ! ! ! ! ! ! !         ! ! ! ! ! ! ! ! ! ! ! ! interface Loopback0  ip address 192.168.21.21 255.255.255.255 ! interface Loopback21  ip address 172.16.21.21 255.255.255.0  ip nat inside ! interface Tunnel11  ip address 10.11.21.21 255.255.255.0  no ip redirects  ip mtu 1400  ip nhrp network-id 100  ip nhrp nhs 10.11.21.11 nbma 10.11.111.11 multicast  ip tcp adjust-mss 1360  if-state nhrp  tunnel source GigabitEthernet2  tunnel mode gre multipoint  tunnel key 100  tunnel path-mtu-discovery  tunnel vrf MPLS ! interface Tunnel12  ip address 10.12.21.21 255.255.255.0  no ip redirects  ip mtu 1400  ip nhrp network-id 101  ip nhrp nhs 10.12.21.12 nbma 10.12.112.12 multicast  ip tcp adjust-mss 1360  no nhrp route-watch  if-state nhrp  tunnel source GigabitEthernet3  tunnel mode gre multipoint  tunnel key 101  tunnel path-mtu-discovery  tunnel vrf INET ! interface GigabitEthernet1  ip address 10.255.0.66 255.255.0.0  negotiation auto  no mop enabled  no mop sysid ! interface GigabitEthernet2  vrf forwarding MPLS  ip address 10.21.111.21 255.255.255.0  negotiation auto  no mop enabled  no mop sysid ! interface GigabitEthernet3  vrf forwarding INET  ip address dhcp  ip nat outside  zone-member security OUTSIDE  ip policy route-map RM-FROM-INET2LAN  negotiation auto  no mop enabled  no mop sysid ! router ospf 1 ! router bgp 64501  bgp log-neighbor-changes  neighbor HUB-MPLS peer-group  neighbor HUB-MPLS remote-as 64501  neighbor HUB-INET peer-group  neighbor HUB-INET remote-as 64501  neighbor 10.11.21.11 peer-group HUB-MPLS  neighbor 10.12.21.12 peer-group HUB-INET  !  address-family ipv4   redistribute connected route-map CONNECTED-TO-BGP   neighbor HUB-MPLS send-community   neighbor HUB-MPLS next-hop-self   neighbor HUB-MPLS soft-reconfiguration inbound   neighbor HUB-INET send-community   neighbor HUB-INET next-hop-self   neighbor HUB-INET soft-reconfiguration inbound   neighbor 10.11.21.11 activate   neighbor 10.12.21.12 activate   distance bgp 200 19 19  exit-address-family ! ! virtual-service csr_mgmt ! ip local policy route-map RM-FROM-INET2LAN ip nat inside source route-map RM-FOR-INET interface GigabitEthernet3 overload ip forward-protocol nd ip http server ip http authentication local ip http secure-server ! ip bgp-community new-format ip route 0.0.0.0 0.0.0.0 GigabitEthernet3 10.21.112.112 10 track 12 ip route 172.16.0.0 255.255.252.0 Null0 254 ip route vrf MPLS 10.11.111.11 255.255.255.255 10.21.111.111 ip route vrf INET 10.12.112.12 255.255.255.255 dhcp 10 ip route 10.21.112.112 255.255.255.255 GigabitEthernet3 dhcp 10 ! ip access-list extended ACL-DHCP-IN  permit udp any eq bootps any eq bootpc ip access-list extended ACL-DHCP-OUT  permit udp any eq bootpc any eq bootps ip access-list extended ACL-ESP-IN  permit esp any any ip access-list extended ACL-ESP-OUT  permit esp any any ip access-list extended ACL-FOR-NAT  permit ip 172.16.21.0 0.0.0.255 any ip access-list extended ACL-GRE-IN  permit gre any any ip access-list extended ACL-ROUTER-IN  permit udp any any eq non500-isakmp  permit udp any any eq isakmp  permit icmp any any echo  permit icmp any any echo-reply  permit icmp any any ttl-exceeded  permit icmp any any port-unreachable  permit udp any any range 33434 33463 ttl eq 1 ip access-list extended ACL-ROUTER-OUT  permit udp any any eq non500-isakmp  permit udp any any eq isakmp  permit icmp any any ! ip sla 122  icmp-echo 10.12.112.12 source-interface GigabitEthernet3  vrf INET  threshold 1000  timeout 3000 ip sla schedule 122 life forever start-time now ip sla 123  icmp-echo 10.9.2.11 source-interface GigabitEthernet3  vrf INET  threshold 1000  timeout 3000 ip sla schedule 123 life forever start-time now no service-routing capabilities-manager ! ! route-map CONNECTED-TO-BGP deny 10  description ** Do not advertise Tunnel interfaces to BGP **  match interface Tunnel11 Tunnel12 ! route-map CONNECTED-TO-BGP deny 20  description ** Do not advertise VIRL Internal mgmt address **  match interface GigabitEthernet1 ! route-map CONNECTED-TO-BGP permit 100 ! route-map RM-FOR-INET permit 10  match ip address ACL-FOR-NAT  match interface GigabitEthernet3 ! route-map RM-FROM-INET2LAN permit 10  match ip address ACL-FOR-NAT  set global ! ! ! control-plane !  !  !  !  ! ! ! ! ! ! line con 0  stopbits 1 line vty 0  login line vty 1  login  length 0 line vty 2 4  login ! ! ! ! ! ! ! end S2R1# ************************************************** ********* COMPLET CONFIGURATION OF MPLS ********** ************************************************** MPLS-111#sh run Building configuration... Current configuration : 2880 bytes ! ! Last configuration change at 02:26:56 UTC Thu Mar 23 2017 ! version 15.6 service timestamps debug datetime msec service timestamps log datetime msec no service password-encryption ! hostname MPLS-111 ! boot-start-marker boot-end-marker ! ! ! no aaa new-model ethernet lmi ce ! ! ! mmi polling-interval 60 no mmi auto-configure no mmi pvc mmi snmp-timeout 180 ! ! ! ! ! ! ! ! ! ! ! ip cef no ipv6 cef ! multilink bundle-name authenticated ! ! ! ! ! redundancy ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! interface GigabitEthernet0/0  ip address 10.255.0.62 255.255.0.0  shutdown  duplex auto  speed auto  media-type rj45 ! interface GigabitEthernet0/1  ip address 10.21.111.111 255.255.255.0  duplex auto  speed auto  media-type rj45 ! interface GigabitEthernet0/2  ip address 10.11.111.111 255.255.255.0  duplex auto  speed auto  media-type rj45 ! ip forward-protocol nd ! ! no ip http server no ip http secure-server ! ! ! ! control-plane ! banner exec ^C ************************************************************************** * IOSv is strictly limited to use for evaluation, demonstration and IOS  * * education. IOSv is provided as-is and is not supported by Cisco's      * * Technical Advisory Center. Any use or disclosure, in whole or in part, * * of the IOSv Software or Documentation to any third party for any       * * purposes is expressly prohibited except as otherwise authorized by     * * Cisco in writing.                                                      * **************************************************************************^C banner incoming ^C ************************************************************************** * IOSv is strictly limited to use for evaluation, demonstration and IOS  * * education. IOSv is provided as-is and is not supported by Cisco's      * * Technical Advisory Center. Any use or disclosure, in whole or in part, * * of the IOSv Software or Documentation to any third party for any       * * purposes is expressly prohibited except as otherwise authorized by     * * Cisco in writing.                                                      * **************************************************************************^C banner login ^C ************************************************************************** * IOSv is strictly limited to use for evaluation, demonstration and IOS  * * education. IOSv is provided as-is and is not supported by Cisco's      * * Technical Advisory Center. Any use or disclosure, in whole or in part, * * of the IOSv Software or Documentation to any third party for any       * * purposes is expressly prohibited except as otherwise authorized by     * * Cisco in writing.                                                      * **************************************************************************^C ! line con 0 line aux 0 line vty 0 4  login  transport input none ! no scheduler allocate ! end MPLS-111# ************************************************** ********* COMPLET CONFIGURATION OF INET ********** ************************************************** Internet#sh run Building configuration... Current configuration : 3178 bytes ! ! Last configuration change at 02:26:45 UTC Thu Mar 23 2017 ! version 15.6 service timestamps debug datetime msec service timestamps log datetime msec no service password-encryption ! hostname Internet ! boot-start-marker boot-end-marker ! ! enable password cisco ! no aaa new-model ethernet lmi ce ! ! ! mmi polling-interval 60 no mmi auto-configure no mmi pvc mmi snmp-timeout 180 ! ! ! ! ! ! ! ! ip dhcp excluded-address 10.21.112.1 10.21.112.20 ip dhcp excluded-address 10.21.112.22 10.21.112.112 ! ip dhcp pool WAN-address  network 10.21.112.0 255.255.255.0  default-router 10.21.112.112 ! ! ! ip cef no ipv6 cef !         multilink bundle-name authenticated ! ! ! ! ! redundancy ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! interface Loopback9211  ip address 10.9.2.11 255.255.255.255 ! interface GigabitEthernet0/0  ip address 10.255.0.61 255.255.0.0  shutdown  duplex auto  speed auto  media-type rj45 ! interface GigabitEthernet0/1  ip address 10.21.112.112 255.255.255.0  duplex auto  speed auto  media-type rj45 ! interface GigabitEthernet0/2  ip address 10.12.112.112 255.255.255.0  duplex auto  speed auto  media-type rj45 ! ip forward-protocol nd !         ! no ip http server no ip http secure-server ! ! ! ! control-plane ! banner exec ^C ************************************************************************** * IOSv is strictly limited to use for evaluation, demonstration and IOS  * * education. IOSv is provided as-is and is not supported by Cisco's      * * Technical Advisory Center. Any use or disclosure, in whole or in part, * * of the IOSv Software or Documentation to any third party for any       * * purposes is expressly prohibited except as otherwise authorized by     * * Cisco in writing.                                                      * **************************************************************************^C banner incoming ^C ************************************************************************** * IOSv is strictly limited to use for evaluation, demonstration and IOS  * * education. IOSv is provided as-is and is not supported by Cisco's      * * Technical Advisory Center. Any use or disclosure, in whole or in part, * * of the IOSv Software or Documentation to any third party for any       * * purposes is expressly prohibited except as otherwise authorized by     * * Cisco in writing.                                                      * **************************************************************************^C banner login ^C ************************************************************************** * IOSv is strictly limited to use for evaluation, demonstration and IOS  * * education. IOSv is provided as-is and is not supported by Cisco's      * * Technical Advisory Center. Any use or disclosure, in whole or in part, * * of the IOSv Software or Documentation to any third party for any       * * purposes is expressly prohibited except as otherwise authorized by     * * Cisco in writing.                                                      * **************************************************************************^C ! line con 0 line aux 0 line vty 0 4  password cisco  login  transport input telnet ! no scheduler allocate !         end Internet#