SD-WAN Part V: Hub and Spoke with Restrected Spoke Sites

 

 

Introduction

 

Cisco Viptela SD-WAN solution builds a full-mesh topology between vEdge devices by default when there are no Control Policies implemented. This means that vEdges tries to build an IPSec/GRE tunnel to every reachable TLOC public IP addresses no matter which site or color (transport network) TLOCs belong to. We have already change the default behavior by using the restrict option (chapter 2) under tunnel interfaces. In this way, tunnels are only established between TLOCs belonging to the same color. In this chapter, we are going to create a Hub and Spoke topology by implementing a Control Policy where the vSmart advertises TLOC/OMP routes from site 30 to sites 10 and 20 and TLOC/OMP routes from sites 10 and 20 to site 30. vSmart doesn’t advertise TLOC/OMP routes between sites 10 and 20. Site 10 and 20 will be our Branch/Remote sites and site 30 will be the Hub/DataCenter site.

 

Figure 5-1 recaps the operation of the Overlay Management Protocol (OMP). vEdge1 in site 10 advertises TLOC route advertisement to vSmart where it describes its System Id, transport color, and encapsulation method as well as Public/Private IP and restricts attributes (among several other attributes). vSmart forwards TLOC routes received from vEdge1 to both vEdge2 (site 20) and vEdge3 (site 30). vEdge1 also advertises OMP routes where it describes the reachability information about its local subnet 172.16.10.0/24 bound to VPN10.

Figure 5-1: TLOC Route advertisement.

 

When vEdge2 and vEdge3 receive TLOC route advertisement, they establish GRE tunnels between the same colors (restrict option) using the received public IP address as a tunnel destination. They also start the BFD session over the tunnel to monitor its health. When tunnels are up and running vEdges can send data using a public IP address as a destination IP address in the tunnel header. Figure 5-2 illustrates how vEdges establish color-to-color tunnels between themselves. vEdge1 has an mpls-to-mpls tunnel with both vEdge2 and vEdge3 as well as an internet-to-internet tunnel with both vEdges. vEdge2 and vEdge3 have also tunnels between themselves.

 

Figure 5-2: Full Mesh Tunnels Between vEdges.

 

Figure 5-3 shows the TLOC routes received by vEdge1. You can verify the real-time information of vEdge1 by selecting it from the device list in the Monitor/Network window on the vManage GUI. Select OMP Received TLOCS from the Device Option field in order to see received TLOC routes. As an example, we can see that vEdge1 has received two TLOC routes from vSmart (10.100.100.13) originated by vEdge2 10.10.100.102.

Figure 5-3: Received TLOCs from vEdge1 Perspective.

 

Figure 5-4 verifies that vEdge also has BFD sessions between vEdge2 and vEdge3 over both transport networks.

Figure 5-4: BFD Sessions of vEdge1.

 

Figure 5-5 shows that vEdge1 has received VPN10 specific OMP routes from both sites over both transport networks.

Figure 5-5: Received OMP Routes from vEdge1 Perspective.

  

We can also see that vEdge1 has installed VPN 10 specific OMP routes into VPN 10 FIB (Forwarding Information Base).

Figure 5-6: VPN10 related IP Routes from vEdge1 Perspective.

 

The example below verifies that we have IP connectivity from host 172.16.10.10 (site 10) to host 172.16.30.30 (site 30) and to host 172.16.20.20 (site 20).

 

PC1-10> ping 172.16.30.30

 

84 bytes from 172.16.30.30 icmp_seq=1 ttl=62 time=2.646 ms

84 bytes from 172.16.30.30 icmp_seq=2 ttl=62 time=4.439 ms

84 bytes from 172.16.30.30 icmp_seq=3 ttl=62 time=2.830 ms

84 bytes from 172.16.30.30 icmp_seq=4 ttl=62 time=2.663 ms

84 bytes from 172.16.30.30 icmp_seq=5 ttl=62 time=2.298 ms

 

PC1-10> ping 172.16.20.20

 

84 bytes from 172.16.20.20 icmp_seq=1 ttl=62 time=7.359 ms

84 bytes from 172.16.20.20 icmp_seq=2 ttl=62 time=5.029 ms

84 bytes from 172.16.20.20 icmp_seq=3 ttl=62 time=2.071 ms

84 bytes from 172.16.20.20 icmp_seq=4 ttl=62 time=2.637 ms

84 bytes from 172.16.20.20 icmp_seq=5 ttl=62 time=3.321 ms

Example 5-1: IP Connectivity Verification

vSmart - from CLI mode to vManaged mode

 

The default management mode for Viptela components is CLI mode. We can’t use NETCONF to push policies made in vMange to vSmart unless we change the mode to vMange mode. Figure 5-7 shows the failure notification when trying to activate the Centralized Policy when vSmart is in CLI mode.

Figure 5-7: Failure Notification.

 

We can verify the management mode either from the vSmart CLI or by navigating to configuration/devices and by selecting the Controllers tab from the vManage GUI. Example 5-2 shows that the vManged state is false and the Configuration Template is set to None.

 

vsmart# show system status | beg Pers

 

Personality:             vsmart

Model name:              vsmart

Services:                None

vManaged:                false

Commit pending:          false

Configuration template:  None

Policy template:         None

Policy template version: None

Chassis serial number:   None

Example 5-2: Verifying Manage Mode Used in vSmart by Using the Device CLI.

Figure 5-8 shows that vSmart mode is CLI and there is no assigned template.

Create CLI Template

 

In order to change the CLI mode to the vManaged mode navigate to the configuration/templates window on vManage GUI. Then select the Device tab and from there select CLI Template from the Create Template drop-down menu.

Figure 5-9: Changing vSmart to the vManaged Mode – Step#1.

 

Then select the vSmart from the Device Model drop-down menu and fill the Template Name and Description fields. You can either copy the vSmart configuration from the vSmart CLI or by selecting vSmart from the Load Running config from the reachable device drop-down menu. Click the Add button when done.

Figure 5-10: Changing vSmart to the vManaged Mode – Step#2.

 

Attach CLI Template to vSmart

 

Attach the CLI template to vSmart by selecting Attach Device from the template option menu […].

Figure 5-11: Changing vSmart to the vManaged Mode – Step#3.

 

Select vsmart from the Available Devices column and press the Arrow button to move it to Selected Devices columns. Then click the Attach button. 

Figure 5-12: Changing vSmart to the vManaged Mode – Step#4.

As the last step, select the Configure Devices button.

Figure 5-13: Changing vSmart to the vManaged Mode – Step#5.

 

Figure 5-14 shows the successful template installation.

Figure 5-14: Changing vSmart to the vManaged Mode – Verification: Success.

 

The failure shown in figure 5-15 may occur if you manually add the CLI template configuration.

Figure 5-15: Changing vSmart to the vManaged Mode – Verification: Failure.

 

We can verify successful changes from the CLI (example 5-3) or from the vManage GUI (figure 5-16).

 

vsmart# show system status | beg Pers

Personality:             vsmart

Model name:              vsmart

Services:                None

vManaged:                true

Commit pending:          true

Configuration template:  vSmart-Template-v1

Policy template:         None

Policy template version: None

Chassis serial number:   None

Example 5-3: Verifying Manage Mode Used in vSmart by Using the vSmart CLI.

Figure 5-16: Verifying Manage Mode Used in vSmart by Using the vManage GUI.

Policy Configuration

 

Figure 5-17 illustrates the SD-WAN Policy model. It has two main policy categories, a) Localized Policies, and b) Centralized Policies. The focus of this chapter is Centralized Policies. Centralized Policies has two sub-categories, a) Data Policies and Control/Topology policies. Our intent is to restrict traffic between site 10 and site 20 so we are going to build a Centralized Policy where we are using Control Policy to build a Hub and Spoke topology where sites 10 and 20 are restricted from each other by controlling TLOC/OMP route advertisements. 

Figure 5-17: SD-WAN Policy Model.

 

Figure 5-18 illustrates the TLOC/OMP route advertisement directions. The vSmart can advertise TLOC/OMP routes received from Branch Sites (site 10 and site 20) to DataCenter (site 30). Besides, vSmart can advertise TLOC/OMP routes from DataCenter to Branch Sites. However, vSmart doesn’t advertise TLOC/OMP routes received from one Branch Site to another. This way we create Hub and Spoke topology where Branch Sites are restricted from each other and no data can be sent between them. The whole Control Policy is based on routing control and we are not using any data traffic filtering.

Figure 5-18: TLOC/OMP Routes Advertisement Direction.

 

Example 5-4 shows the configuration we are going to build by using vManage GUI. It also shows the structure and configuration flow of the Centralized Policy. Note that the Centralized Policy doesn’t have any name, we can only use one Central Policy. In the first step, we create a set of lists where we define a Group of Interest. A list can include application, color, data prefix, policer, prefix, site, SLA class, TLOC, or VPN. We are going to use two site lists and a prefix list. One site list for DataCenter (site 30) and the other one for the Remote-Sites (sites 10 and 20). The prefix list matches all networks. In the second step, we create a Control Policy that has a set of sequences where we match TLOCs and OMP routes from site 30 listed under site list DataCenter and where we set the action Accept. At the end of the Control Policy, we have implicit reject. As the last step, we apply the Control Policy to sites listed in the site-list Remote-Sites as an outbound policy. By doing this, we allow vSmart to send TLOC and OMP routes originated from the DataCenter to Remote-Sites but not TLOC and OMP routes from site 10 to site 20 and the other way around. There is no policy applied towards site 30, so vSmart forwards all TLOC and OMP routes to site 30.

Policy                                   # Centrlazied Policy

 Lists                                   # Group of Interest  

  site-list DataCenter                   

   site-id 30

  !

  site-list Remote-Sites

   site-id 10

   site-id 20

  !

  prefix-list _AnyIpv4PrefixList

   ip-prefix 0.0.0.0/0 le 32

  !

 !

 control-policy HUB_AND_SPOKE_POLICY       # Control Policy

  sequence 1

   match tloc

    site-list DataCenter

   !

   action accept

   !

  !

  sequence 11

   match route

    prefix-list _AnyIpv4PrefixList

    site-list   DataCenter

   !

   action accept

   !

  !

  default-action reject

 !

!

apply-policy                       # Apply Policy

 site-list Remote-Sites

  control-policy HUB_AND_SPOKE_POLICY out

Example 5-4: Verifying Manage Mode Used in vSmart by Using the vSmart CLI.

Step-1: Create Site-List

 

Open the vMange GUI and navigate to configurations/policies and select the Centralized Policy tab. Click Add Policy button.

Figure 5-19: Add Centralized Policy.

 

Select Site from the left list pane. Click the New Site List button. I have named the first site list as Remote-Sites and added sites 10 and 20 to the list.

Figure 5-20: Adding Site Lists.

 

I have also created the DataCenter site-list in the same way. Figure 5-21 shows our two site lists. To move forward to Configure Topology and VPN Membership section, click the Next button at the bottom (not shown in the figure).

Figure 5-21: Site Lists DatCenter and Remote-Sites.

 

Example 5-5 shows the configuration related to site lists. The configuration is sent to vSmart as a part of the complete policy configuration when we activate the policy as the last step. Note that there is also an auto-generated prefix-list with prefix 0.0.0.0/0 le 32 which covers all subnets including 172.16.30.0/24.

 

policy

 lists

  site-list DataCenter

   site-id 30

  !

  site-list Remote-Sites

   site-id 10

   site-id 20

  !

  prefix-list _AnyIpv4PrefixList

   ip-prefix 0.0.0.0/0 le 32

Example 5-5: Site List Configuration.

 

Step-2: Create Control Policy

 

Open the Add Topology drop-down menu and select Custom Control (Route & TLOC) option.

 

Figure 5-22: Configure Centralized Policy Step 1: Control Policy TLOC.

Give the name and description and click the Sequence Type button. This will create the first sequence under Control Policy.

Figure 5-23: Configure Centralized Policy Step 2: Control Policy TLOC.

 

Select TLOC from the Add Control Policy pop-up window. By doing this the first sequence matches to TLOCs.

Figure 5-24: Configure Centralized Policy Step 3: Control Policy TLOC.

 

Click the Action selector and choose the Accept option.

Figure 5-25: Configure Centralized Policy Step 4: Control Policy TLOC.

 

Click the Match selector and open the Site List from the Match Conditions window. Select site list DataCenter.

Figure 5-26: Configure Centralized Policy Step 5: Control Policy TLOC.

Click Save Match and Actions button.

Figure 5-27: Configure Centralized Policy Step 6: Control Policy TLOC.

 

Figure 5-28 shows the first sequence TLOC. We created a sequence inside the Control Policy HUB_AND_SPOKE_POLICY where we match to TLOCs of DataCenter and set the action to permit. 

Figure 5-28: Configure Centralized Policy Step 6: Control Policy TLOC.

Create a new sequence under the same Control Policy. This sequence will match to OMP Routes. In addition, this sequence also matched the default IP prefix because we are not explicitly defined any IP prefix.

Figure 5-29: Configure Centralized Policy Step 7: Control Policy OMP Route.

 

The process of creating a route sequence is the same as what we did with the sequence related to TLOC. Set the Match and Action conditions.

Figure 5-30: Configure Centralized Policy Step 8: Control Policy OMP Route.

Figure 5-31 shows that the sequence related to TLOC is listed before the sequence related to OMP Route.

Figure 5-31: Configure Centralized Policy Step 9: Control Policy OMP Route.

 

Example 5-5 shows the configuration of the Control Policy. The auto-generated prefix-list with prefix 0.0.0.0/0 is added under the match route (OMP route) section because we did not specify the exact subnet.

 

control-policy HUB_AND_SPOKE_POLICY

  sequence 1

   match tloc

    site-list DataCenter

   !

   action accept

   !

  !

  sequence 11

   match route

    prefix-list _AnyIpv4PrefixList

    site-list   DataCenter

   !

   action accept

   !

  !

  default-action reject

Example 5-6: Configuration Related to TLOC/OMP Control-Policy Sequences.

By clicking the Next button, you will be forwarded to Configure Traffic Rules window.

Figure 5-31: Configure Centralized Policy Step 10: Control Policy.

 

We are not using any traffic rules so click the Next button to move to Apply Policy to Sites and VPN window.

Figure 5-32: Configure Centralized Policy Step 11: Control Policy.

Step-3: Apply Control Policy

 

Give the name to Centralized Policy and some description. Note that you can activate only one Centralized Policy at a time and that is why I am using version-definition with the name. Select site list Remote-Sites from the Outbound Site List field. By doing this we add a Control Policy Hub_and_Spoke_Policy toward sites 10 and 20 defined in the Remote-Sites site-list. Remember that the overall policy is a Centralized Policy and the Control Policy is the policy that is used for setting up the topology. Click the Add button.

Figure 5-33: Configure Centralized Policy Step 12: Apply Policy.

 

Example 5-7 shows the CLI configuration generated in Apply Policies to Sites and VPNs phase.

 

apply-policy

 site-list Remote-Sites

  control-policy HUB_AND_SPOKE_POLICY out

Example 5-7: Configure Centralized Policy Step 12 from the CLI: Apply Policy.

 

Figure 5-34 shows that Control Policy Hub_and_Spoke_Policy is attached to the site list Branch-Sites. You can check the configuration that has been created by clicking the Preview button. Click the Save Policy button.

Figure 5-34: Configure Centralized Policy Step 13: Activating Policy.

 

Step-4: Activate Centralized Policy

 

As the final step, we activate the Centralized Policy by selecting Activate from the option drop-down menu […]

Figure 5-35: Configure Centralized Policy Step 14: Activating Policy.

 

You will be asked to confirm the activation. Click the Activate button.

Figure 5-36: Configure Centralized Policy Step 14: Activating Policy.

 

Figure 5-37 shows that the Centralized Policy is now applied successfully to vSmart.

Figure 5-37: Configure Centralized Policy Step 14: Activating Policy.

Policy Verification

 

We can verify that our Centralized Policy is in effect from the vMange GUI by verifying which TLOC and OMP routes vEdge1 has received from the vSmart. Figure 5-38 shows that in addition to local TLOCs, vEdge1 has received only site 30 TLOCs (DataCenter).

Figure 5-38: Centralized Policy Verification – TLOC routes.

 

We can also see that vEdge1 has only received site 30 related OMP routes from the vSmart.

Figure 5-39: Centralized Policy Verification – OMP routes.

Figure 5-40 verifies that vEdge1 has installed OMP route 172.16.30.0/24 to VPN10 Routing Information Base (RIB). 

Figure 5-40: Centralized Policy Verification – IP routes.

 

We can also see that there is only two BFD session from vEdge1 (site 10: Branch-Site) to vEdge-3 (site 30: DataCenter), one over Public-Internet transport network and the other one over MPLS transport network.

Figure 5-41: Centralized Policy Verification – BFD Sessions.

Figure 5-42 illustrates the topology we created with Centralized Topology. There are no tunnels between vEdge1 and vEdge2.

Figure 5-42: Final Topology – Hub and Spoke.

 

Example 5-8 shows that we can’t anymore ping the end-point  172.16.20.20 but we still can ping end-point 172.16.30.30.

 

PC1-10> ping 172.16.20.20

 

*172.16.10.1 icmp_seq=1 ttl=64 time=0.140 ms (ICMP type:3, code:0, Destination network unreachable)

*172.16.10.1 icmp_seq=2 ttl=64 time=0.126 ms (ICMP type:3, code:0, Destination network unreachable)

*172.16.10.1 icmp_seq=3 ttl=64 time=0.131 ms (ICMP type:3, code:0, Destination network unreachable)

*172.16.10.1 icmp_seq=4 ttl=64 time=0.128 ms (ICMP type:3, code:0, Destination network unreachable)

*172.16.10.1 icmp_seq=5 ttl=64 time=0.210 ms (ICMP type:3, code:0, Destination network unreachable)

 

PC1-10> ping 172.16.30.30

 

84 bytes from 172.16.30.30 icmp_seq=1 ttl=62 time=2.646 ms

84 bytes from 172.16.30.30 icmp_seq=2 ttl=62 time=4.439 ms

84 bytes from 172.16.30.30 icmp_seq=3 ttl=62 time=2.830 ms

84 bytes from 172.16.30.30 icmp_seq=4 ttl=62 time=2.663 ms

84 bytes from 172.16.30.30 icmp_seq=5 ttl=62 time=2.298 ms

Example 5-8: Data Plane Testing.

Example 5-9 shows the complete vSmart configuration

 

vsmart# sh run

system

 host-name             vsmart

 system-ip             10.100.100.13

 site-id               100

 admin-tech-on-failure

 organization-name     nwkt

 vbond 10.100.0.11

 aaa

  auth-order local radius tacacs

  usergroup basic

   task system read write

   task interface read write

  !

  usergroup netadmin

  !

  usergroup operator

   task system read

   task interface read

   task policy read

   task routing read

   task security read

  !

  usergroup tenantadmin

  !

  user admin

   password $6$jKzSSqC2GCJveJV4$VxMCv59Qv2J.lDd2luqXXJ9dUuv3izVKXPEbE3b43AAry3n6                                                                                                       ptI7DqunO0y0TzxaUVRGAUZ7E/ySEiWdyt8/60

  !

  user ciscotacro

   description CiscoTACReadOnly

   group       operator

   status      enabled

  !

  user ciscotacrw

   description CiscoTACReadWrite

   group       netadmin

   status      enabled

  !

 !

 logging

  disk

   enable

  !

 !

 ntp

  server 10.100.0.14

   version 4

  exit

 !

!

omp

 no shutdown

 graceful-restart

!

vpn 0

 interface eth0

  ip address 10.100.0.13/24

  ipv6 dhcp-client

  tunnel-interface

   allow-service all

   allow-service dhcp

   allow-service dns

   allow-service icmp

   no allow-service sshd

   allow-service netconf

   no allow-service ntp

   no allow-service stun

  !

  no shutdown

 !

 ip route 0.0.0.0/0 10.100.0.1

!

vpn 512

 interface eth1

  ip dhcp-client

  no shutdown

 !

!

policy

 lists

  site-list DataCenter

   site-id 30

  !

  site-list Remote-Sites

   site-id 10

   site-id 20

  !

  prefix-list _AnyIpv4PrefixList

   ip-prefix 0.0.0.0/0 le 32

  !

 !

 control-policy HUB_AND_SPOKE_POLICY

  sequence 1

   match tloc

    site-list DataCenter

   !

   action accept

   !

  !

  sequence 11

   match route

    prefix-list _AnyIpv4PrefixList

    site-list   DataCenter

   !

   action accept

   !

  !

  default-action reject

 !

!

apply-policy

 site-list Remote-Sites

  control-policy HUB_AND_SPOKE_POLICY out

 !

!

vsmart#

Example 5-9: Complete vSmart Configuration.

Spoke-to-Spoke traffic

 

If we want to allow traffic flows between spoke sites without direct tunneling we can do the OMP route summarization in the Hub site. In our example, I have added a null route under VPN10 configuration for network 172.16.0.0/10. This static route is automatically redistributed into the OMP route advertisement process. Our Centralized Policy allows OMP Routes from site 30 to be advertised to sites 10 and 20. 

Figure 5-43: Spoke-to-Spoke: OMP Summary Route Advertisement.

Figure 5-44 shows that vEdge3 advertises the OMP Summary route 172.16.0.0/19 to vSmart.

Figure 5-44: Spoke-to-Spoke: OMP Summary Route Advertisement – vEdge3.

 

Figure 5-45 shows that vSmart in turn advertises the OMP Summary route 172.16.0.0/19 to both vEdge1 and vEdge2.

Figure 5-45: Spoke-to-Spoke: OMP Summary Route Advertisement – vSmart.

Figures 5-46 and 5-47 verifies that both vEdge1 and vEdge2 have received the OMP Summary route 172.16.0.0/19 from vSmart.

Figure 5-46: Spoke-to-Spoke: Received OMP Summary Route – vEdge1.

Figure 5-47: Spoke-to-Spoke: Received OMP Summary Route – vEdge2.

Figure 5-48 verifies that we have a data plane connection also between sites 10 and 20 though we don’t have a GRE tunnel or BFD session between sites (figures 5-49 and 5-50).

Figure 5-48: Spoke-to-Spoke: Data Plane Testing.

Figure 5-49: Spoke-to-Spoke: GRE Tunnels of vEdge2.

 

Figure 5-50: Spoke-to-Spoke: BFD Sessions of vEdge2.

Summary

 

This chapter started by explaining tasks required to change vSmart from CLI Managed to vManaged mode. Next, it explained how to construct a Centralized Policy that has a Control Policy which rejects TLOC and OMP routes received from site 10 to be advertised to site 20 and the other way around. As a result, there is no GRE tunnel between site 10 and site 20 and they are restricted from each other. The last section describes how we can allow traffic between site 10 and site 20 by using OMP route summarization in site 30.