SD-WAN Part IV: MPLS Network with IS-IS Segment Routing as SD-WAN Transport

 

Introduction

 

In order to have IP connectivity between hosts A and B over the underlay transport network, we need to build a tunnel (IPSec or GRE) between the Public  IP addresses of vEdge devices (TLOC Routes). Then we also need VPN-specific subnet routing information (OMP Routes) to be able to route traffic over the tunnel. This chapter discusses the role and operation of various protocols involved in Control Plane operations when an MPLS Transport network is used as an Underlay Network for SD-WAN solution. The first section introduces the Segment Routing solution for building a Label Switch Path (LSP) between PE routers over the MPLS backbone by using the IS-IS routing protocol for both routing and label distribution. The second section explains how to build L3VPN between vEdge Public IP addresses over the LSP. Figure 4-1 shows the high-level routing model used in this chapter.

Figure 4-1: Control Plane Model.

Building a Label Switch Path

 

In order to build a Labels Switch Path (LSP) between PE devices, we need a routing protocol for IP reachability and MPLS labels for subnet/label binding. This is because forwarding decision within MPLS network is based on labels, not IP addresses. Label/Subnet binding information in traditional MPLS networks is done by using Label Distribution Protocol (LDP). Having one protocol for routing and another for the label distribution, however, increases overall complexity. The Segment-Routing is developed for simplifying the solution. It uses a routing protocol for advertising both IP reachability information and MPLS label-related information. The reason why I use the term “IP reachability” instead of the term “route” is that the most common IGP protocols in MPLS Underlay are OSPF and

IS-IS and which don’t advertise routes but link-state information. Segment-Routing in turn doesn’t advertise label/subnet binding information, it advertises a label range and then the index number for each destination.

 

Segment Routing Global Block (SRGB)

 

Segment-Routing Global Block (SRGB) defines the label range where Link-State IGP allocates IGP Prefix-SID (Prefix Segment Identifier). In practice, IGP Prefix-SID is a node-specific identifier within the MPLS network. We are using the default SRGB 16000 – 23999 in our examples (figure 4-2). When the IS-IS Segment-Routing protocol extension is enabled on a router, it advertises the SRGB label range (8000) within SR Capability sub-TLV (Type-2) carried in the IS-IS Router Capability TLV (Type-242). The SRGB base label (16000) in turn is advertised by using the Prefix-SID/Label sub-TLV (Type-1) carried within the SR Capability sub-TLV.

 

Note: IS-IS uses Links State Protocol (LSP) Data Units for advertising information to its IS-IS neighbors. Information is encoded as Type/Length/Value (TLV) fields.

 

IGP Prefix Segment (Prefix-SID)

 

IS-IS Segment-Routing extension uses the Extended IP reachability TLV (Type-135) for advertising the node IP and its metric. Its Prefix-SID sub-TLV (Type-3) describes the index number/label value that is used by receiving IS-IS routers for calculating the label associated with the advertised node IP address. The Index/value is derived from the statically configured value, either as an absolute label value or as an index number. Note that the value has to be a unique, node-specific value. As an example, in PE-01 we are using absolute label value 16001. The index value is calculated by subtracting SRGB base value 16000 from the defined absolute label value 16001 which gives us an index value 1 (16001 – 16000 = 1). The receiving IS-IS router calculates the label value by adding the index value to the SRGB base value which gives the label 16001 (16000 + 1 = 16001).

 

The IS-IS Segment Routing configuration of PE-01, P-02, and PE-02 can be found at the end of this chapter.

 

 

Figure 4-2: MPLS Transport Underlay: IS-IS Link State Packet sent by PE-01.

 

Capture 4-1 shows the complete IS-IS LSP advertised by PE-01.

IEEE 802.3 Ethernet

    Destination: ISIS-all-level-1-IS's (01:80:c2:00:00:14)

    Source: 50:15:00:00:1b:08 (50:15:00:00:1b:08)

    Length: 134

Logical-Link Control

ISO 10589 ISIS InTRA Domain Routeing Information Exchange Protocol

ISO 10589 ISIS Link State Protocol Data Unit

    PDU length: 131

    Remaining lifetime: 1199

    LSP-ID: 0000.0000.0001.00-00

    Sequence number: 0x00000018

    Checksum: 0xac69 [correct]

    [Checksum Status: Good]

    Type block(0x01): Partition Repair:0, Attached bits:0, Overload bit:0, IS type:1

    Area address(es) (t=1, l=2)

    Protocols supported (t=129, l=1)

    Traffic Engineering Router ID (t=134, l=4)

    IP Interface address(es) (t=132, l=4)

        Type: 132

        Length: 4

        IPv4 interface address: 1.1.1.1

    Hostname (t=137, l=5)

    Extended IS reachability (t=22, l=30)

    Extended IP Reachability (t=135, l=26)

        Type: 135

        Length: 26

        Ext. IP Reachability: 10.1.2.0/24

            Metric: 40

            0... .... = Distribution: Up

            .0.. .... = Sub-TLV: No

            ..01 1000 = Prefix Length: 24

            IPv4 prefix: 10.1.2.0

            no sub-TLVs present

        Ext. IP Reachability: 1.1.1.1/32

            Metric: 1

            0... .... = Distribution: Up

            .1.. .... = Sub-TLV: Yes

            ..10 0000 = Prefix Length: 32

            IPv4 prefix: 1.1.1.1

            SubCLV Length: 8

            subTLV: Prefix-SID (t=3, l=6)

                Code: Prefix-SID (3)

                Length: 6

                Flags: 0x40, Node-SID

                Algorithm: Shortest Path First (SPF) (0)

                SID/Label/Index: 0x00000001

    Router Capability (t=242, l=16)

        Type: 242

        Length: 16

        Router ID: 0x01010101

        .... ...0 = S bit: False

        .... ..0. = D bit: False

        Segment Routing - Capability (t=2, l=9)

            1... .... = I flag: IPv4 support: True

            .0.. .... = V flag: IPv6 support: False

            Range: 8000

            SID/Label (t=1, l=3)

                Label: 16000

Capture 4-1: IS-IS Link State Packet sent by PE-01.

Example 4-1 shows that PE-03 knows the SRGB range as well as Prefix-SID/Index associated with IP 1.1.1.1/32.

 

PE-03# sh isis database detail PE-01.00-00

IS-IS Process: SR LSP database VRF: default

IS-IS Level-1 Link State Database

  LSPID                 Seq Number   Checksum  Lifetime   A/P/O/T

  PE-01.00-00           0x00000012   0x6AB1    968        0/0/0/1

    Instance      :  0x0000000D

    Area Address  :  49

    NLPID         :  0xCC

    Router ID     :  1.1.1.1

    IP Address    :  1.1.1.1

    Hostname      :  PE-01              Length : 5

    Extended IS   :  P-02.00            Metric : 40

      Interface IP Address :  10.1.2.1

      IP Neighbor Address :  10.1.2.2

      ADJ-SID         :  16                 Flags  : V/L,  Weight 1

    Extended IP   :         1.1.1.1/32  Metric : 1           (U)

      Prefix-SID  :                  1  Algo   : 0    Flags : N

    Extended IP   :        10.1.2.0/24  Metric : 40          (U)

    Capability    : Router-Id 1.1.1.1          Flags 0x0

      SR-Range        :  16000 - 23999 (8000) Flags I--

    Digest Offset :  0

 

IS-IS Level-2 Link State Database

  LSPID                 Seq Number   Checksum  Lifetime   A/P/O/T

Example 4-1: show isis database detail PE-01.00-00 on PE-03.

 

 

The IPv4 Forwarding Equivalency Class (FEC)  under VRF default in example 4-2 shows that PE-03 uses MPLS label 16001 in the outer MPLS header when sending data packets to PE-01 (1.1.1.1/32). Deaggregatin FEC is customer VRF related and there we can see the customer VPN Label advertised by MP-BGP. We will focus on that in the next section. The Adjacency SID (ADJ SID) describes the Inter-Router links.

 

 

PE-03#  sh mpls switching detail

 

VRF default

IPv4 FEC

 In-Label                       : 16001

 Out-Label stack                : 16001

 FEC                            : 1.1.1.1/32

 Out interface                  : Eth1/3

 Next hop                       : 10.2.3.2

 Input traffic statistics       : 0 packets, 0 bytes

 Output statistics per label    : label 16001, 0 packets, 0 bytes

 

Deaggregation FEC type

 In-Label                       : 492287

 VRF                            : Customer-77

 Address-Family                 : IPv4

 Input traffic statistics       : 0 packets 0 bytes

 

ADJ SID

 In-Label                       : 16

 Out-Label stack                : 3

 FEC                            : 10.2.3.2

 Out interface                  : Eth1/3

 Next hop                       : 10.2.3.2

 Input traffic statistics       : 0 packets, 0 bytes

 Output statistics per label    : label 3, 0 packets, 0 bytes

 

 

*Label statistics accurate as of 72 seconds ago

 

Block      Label-Range

1          16000 - 23999

Example 4-2: show isis database detail PE-01.00-00 on PE-03.

 

 

Example 4-3 shows that PE-03 is installed the information in its RIB. Whenever PE-03 has something to send to 1.1.1.1/32, it pushes the label 16001 as a top label for the packet. In case that traffic is received from the customer VRF, PE-03 also adds the VPN label as a bottom of stack label.

 

 

PE-03# show ip route 1.1.1.1 isis-SR detail

IP Route Table for VRF "default"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

 

1.1.1.1/32, ubest/mbest: 1/0

    *via 10.2.3.2, Eth1/3, [115/81], 00:05:37, isis-SR, L1 (mpls)

         MPLS[0]: Label=16001 E=0 TTL=255 S=0

         client-specific data: 41

Example 4-3: show ip route 1.1.1.1 isis-SR detail on PE-03.

 

Examples 4-4 and 4-5 show the same show commands from the PE-01 perspective.

 

PE-01# sh isis database detail PE-03.00-00

IS-IS Process: SR LSP database VRF: default

IS-IS Level-1 Link State Database

  LSPID                 Seq Number   Checksum  Lifetime   A/P/O/T

  PE-03.00-00           0x00000012   0xF5F7    1182       0/0/0/1

    Instance      :  0x0000000D

    Area Address  :  49

    NLPID         :  0xCC

    Router ID     :  3.3.3.3

    IP Address    :  3.3.3.3

    Hostname      :  PE-03              Length : 5

    Extended IS   :  P-02.00            Metric : 40

      Interface IP Address :  10.2.3.3

      IP Neighbor Address :  10.2.3.2

      ADJ-SID         :  16                 Flags  : V/L,  Weight 1

    Extended IP   :         3.3.3.3/32  Metric : 1           (U)

      Prefix-SID  :                  3  Algo   : 0    Flags : N

    Extended IP   :        10.2.3.0/24  Metric : 40          (U)

    Capability    : Router-Id 3.3.3.3          Flags 0x0

      SR-Range        :  16000 - 23999 (8000) Flags I--

    Digest Offset :  0

 

<snipped>

Example 4-4: show isis database detail PE-03.00-00 on PE-01.

 

 

PE-01# sh mpls switching detail

 

VRF default

IPv4 FEC

 In-Label                       : 16003

 Out-Label stack                : 16003

 FEC                            : 3.3.3.3/32

 Out interface                  : Eth1/2

 Next hop                       : 10.1.2.2

 Input traffic statistics       : 0 packets, 0 bytes

 Output statistics per label    : label 16003, 0 packets, 0 bytes

 

Deaggregation FEC type

 In-Label                       : 492287

 VRF                            : Customer-77

 Address-Family                 : IPv4

 Input traffic statistics       : 0 packets 0 bytes

 

ADJ SID

 In-Label                       : 16

 Out-Label stack                : 3

 FEC                            : 10.1.2.2

 Out interface                  : Eth1/2

 Next hop                       : 10.1.2.2

 Input traffic statistics       : 0 packets, 0 bytes

 Output statistics per label    : label 3, 0 packets, 0 bytes

<snipped>

Example 4-5: sh mpls switching detail on PE-01.

MP-BGP: Advertising Customer Routes

 

MP-BGP is used for advertising customer routes between PE devices. BGP uses IPv4/Labeled VPN Unicast afi/safi for advertising VPNv4 addresses (Route-Distinguisher:IPv4 prefix). BGP Update carries the IPv4 prefix, its associated RD, and VPN Label value. The VPN label value within Segment-Routing enabled devices is taken from the SR Dynamic range (default range 24000 - 1048575). Besides, MP-BGP Update carries Route-Target extended community which is used for BGP export/import policy. In our example, PE-01 advertises its customer-specific subnet 10.200.0.0/24 with RD 65077:77 and VPN Label 492287. The BGP Update message is label switched across the MPLS transport meaning P-02 forwards packet based on MPLS label 16003, which is the label used with the destination IP address 3.3.3.3/32.

Figure 4-3: MPLS Transport Overlay: MP-BGP Update by PE-01.

 

Capture 4-2 shows the complete MP-BGP packet sent by PE-01.

 

MultiProtocol Label Switching Header, Label: 16003, Exp: 6, S: 1, TTL: 64

Internet Protocol Version 4, Src: 1.1.1.1, Dst: 3.3.3.3

Transmission Control Protocol, Src Port: 17583, Dst Port: 179, Seq: 106, Ack: 228, Len: 132

Border Gateway Protocol - UPDATE Message

    Marker: ffffffffffffffffffffffffffffffff

    Length: 84

    Type: UPDATE Message (2)

    Withdrawn Routes Length: 0

    Total Path Attribute Length: 61

    Path attributes

        Path Attribute - MP_REACH_NLRI

            Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete

            Type Code: MP_REACH_NLRI (14)

            Length: 32

            Address family identifier (AFI): IPv4 (1)

            Subsequent address family identifier (SAFI): Labeled VPN Unicast (128)

            Next hop network address (12 bytes)

                Next Hop: Empty Label Stack RD=0:0 IPv4=1.1.1.1

            Number of Subnetwork points of attachment (SNPA): 0

            Network layer reachability information (15 bytes)

                BGP Prefix

                    Prefix Length: 112

                    Label Stack: 492287 (bottom)

                    Route Distinguisher: 65077:77

                    MP Reach NLRI IPv4 prefix: 10.200.0.0

        Path Attribute - ORIGIN: IGP

        Path Attribute - AS_PATH: empty

        Path Attribute - LOCAL_PREF: 100

        Path Attribute - EXTENDED_COMMUNITIES

            Flags: 0xc0, Optional, Transitive, Complete

            Type Code: EXTENDED_COMMUNITIES (16)

            Length: 8

            Carried extended communities: (1 community)

                Route Target: 65077:77 [Transitive 2-Octet AS-Specific]

Capture 4-2: MP-BGP update sent by PE-01.

 

 

Example 4-6 show the BGP table of PE-03 concerning subnet 10.200.0.0./24 that is attach to customer VRF in PE-01. We can see that VPN label 492287 is associated with network 10.200.0.0/24 with the next-hop 1.1.1.1.

 

PE-03# sh bgp vpnv4 unicast 10.200.0.0/24

BGP routing table information for VRF default, address family VPNv4 Unicast

Route Distinguisher: 65077:77    (VRF Customer-77)

BGP routing table entry for 10.200.0.0/24, version 7

Paths: (1 available, best #1)

Flags: (0x8008001a) (high32 00000000) on xmit-list, is in urib, is best urib route, is in HW

  vpn: version 11, (0x00000000100002) on xmit-list

 

  Advertised path-id 1, VPN AF advertised path-id 1

  Path type: internal, path is valid, imported same remote RD, is best path, in

rib

  AS-Path: NONE, path sourced internal to AS

    1.1.1.1 (metric 81) from 1.1.1.1 (1.1.1.1)

      Origin IGP, MED not set, localpref 100, weight 0

      Received label 492287

      Extcommunity: RT:65077:77

 

  VRF advertise information:

  Path-id 1 not advertised to any peer

 

  VPN AF advertise information:

  Path-id 1 not advertised to any peer

Example 4-6: sh bgp vpnv4 unicast 10.200.0.0/24 on PE-03.

 

 

The information is installed from the BGP table into the routing table. Examples 4-7 and 4-8 illustrate the recursive next-hop resolution and verify that  PE-03 uses MPLS label 16001 when forwarding customer traffic over the MPLS transport network.

 

PE-03# show ip route detail vrf Customer-77 | sec 10.200.0.0

10.200.0.0/24, ubest/mbest: 1/0

    *via 1.1.1.1%default, [200/0], 00:25:37, bgp-65077, internal, tag 65077 (mpls-vpn)

         MPLS[0]: Label=492287 E=0 TTL=0 S=0 (VPN)

         client-specific data: 2

         recursive next hop: 1.1.1.1/32%default

         extended route information: BGP origin AS 65077 BGP peer AS 65077

Example 4-7 show ip route detail vrf Customer-77 | sec 10.200.0.0 on PE-03.

 

 

PE-03#  show ip route 1.1.1.1 detail

IP Route Table for VRF "default"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

 

1.1.1.1/32, ubest/mbest: 1/0

    *via 10.2.3.2, Eth1/3, [115/81], 00:26:43, isis-SR, L1 (mpls)

         MPLS[0]: Label=16001 E=0 TTL=255 S=0

         client-specific data: 41

Example 4-8 show ip route 1.1.1.1 detail on PE-03.

 

 

When the MPLS Underlay Network routing and label binding process is done and the BGP Updates are sent, vEdges have IP connectivity and they can establish a tunnel between them and sent BFD messages by using it. At this phase the TLOC Route sent to vSmart is valid. Figure 4-4 shows that BFD messages are encapsulated with a label stack where the inner VPN label defines the customer VRF and the outer MPLS label is used for forwarding packets to the destination.

Figure 4-4: BFD over GRE.

 

 

 

Capture 4-3 shows the complete captured packet.

 

MultiProtocol Label Switching Header, Label: 16003, Exp: 6, S: 0, TTL: 62

MultiProtocol Label Switching Header, Label: 492287, Exp: 6, S: 1, TTL: 62

Internet Protocol Version 4, Src: 10.200.0.101, Dst: 10.200.1.103

Generic Routing Encapsulation (IP)

Internet Protocol Version 4, Src: 10.200.0.101, Dst: 10.200.1.103

User Datagram Protocol, Src Port: 3784, Dst Port: 3784

BFD Control message

Capture 4-3: BFD Message Sent by PE-01.

 

When the GRE tunnel is established between vEdge-1 and vEdge-3 user data can be routed over it. Just for the recap, VPN routes in vEdges are advertised to vSmart as OMP routes including all valid TLOCs (Public IP address, Colour, and Encapsulation) that can be used for routing packets towards the advertised VPN subnet. The TLOC is valid when BFD messages can be exchanged between vEdges attached to the same color. As can be seen from figure 4-5 and capture 4-4 there are three label values when data is sent across the MPLS transport network; one for the LSP between PE devices, one as an MPLS customer VRF identifier, and one for the client VPN identifier in vEdges.

Figure 4-5: ICMP from Host A to Host B.

 Capture 4-4 shows the complete captured packet.

MultiProtocol Label Switching Header, Label: 16003, Exp: 0, S: 0, TTL: 62

MultiProtocol Label Switching Header, Label: 492287, Exp: 0, S: 1, TTL: 62

Internet Protocol Version 4, Src: 10.200.0.101, Dst: 10.200.1.103

Generic Routing Encapsulation (MPLS label switched packet)

MultiProtocol Label Switching Header, Label: 1003, Exp: 0, S: 1, TTL: 64

Internet Protocol Version 4, Src: 172.16.10.10, Dst: 172.16.30.30

Internet Control Message Protocol

Capture 4-4: ICMP Request Sent by Host A to Host B.

Summary

 

When we are running SD-WAN over the MPLS transport network we need to understand what information is needed and how it is advertised. First, we need to build an LSP between PE devices. In our example, this was done by using the IS-IS Segment-Routing protocol extension. Then we need IP connectivity between vEdge devices. This is done by advertising networks associated with customer VRF, where vEdges are connected, by using MP-BGP. In addition, we need to advertise TLOC routes and OMP routes to vSmart and from there to other vEdges. It is crucial to understand the relationship between control plane protocols IS-IS, BGP, and OMP and this way to understand how the system works.

The reason why I wrote this chapter is that I wanted readers to understand the additional complexity coming with MPLS transport compared to Internet transport where we only rely on routing. My intent is not to say that don’t use MPLS transport.

MPLS device configurations

 

PE-01# sh run

hostname PE-01

install feature-set mpls

feature-set mpls

feature bgp

feature isis

feature mpls l3vpn

feature mpls segment-routing

feature mpls oam

feature mpls segment-routing traffic-engineering

 

segment-routing

  mpls

    connected-prefix-sid-map

      address-family ipv4

        1.1.1.1/32 absolute 16001

vrf context Customer-77

  rd 65077:77

  address-family ipv4 unicast

    route-target import 65077:77

    route-target export 65077:77

vrf context management

 

interface Ethernet1/1

  no switchport

  vrf member Customer-77

  ip address 10.200.0.1/24

  no shutdown

 

interface Ethernet1/2

  no switchport

  ip address 10.1.2.1/24

  isis network point-to-point

  ip router isis SR

  mpls ip forwarding

  no shutdown

!

interface loopback0

  ip address 1.1.1.1/32

  ip router isis SR

icam monitor scale

 

router isis SR

  net 49.0000.0000.0001.00

  is-type level-1

  address-family ipv4 unicast

    segment-routing mpls

router bgp 65077

  router-id 1.1.1.1

  address-family ipv4 unicast

  address-family vpnv4 unicast

  neighbor 3.3.3.3

    remote-as 65077

    update-source loopback0

    address-family ipv4 unicast

    address-family vpnv4 unicast

      send-community extended

  vrf Customer-77

    address-family ipv4 unicast

      network 10.200.0.0/24

 

 

PE-03# sh run

hostname PE-03

install feature-set mpls

feature-set mpls

feature bgp

feature isis

feature mpls l3vpn

feature mpls segment-routing

feature mpls oam

feature mpls segment-routing traffic-engineering

 

vlan 1

segment-routing

  mpls

    connected-prefix-sid-map

      address-family ipv4

        3.3.3.3/32 absolute 16003

 

vrf context Customer-77

  rd 65077:77

  address-family ipv4 unicast

    route-target import 65077:77

    route-target export 65077:77

vrf context management

 

interface Ethernet1/1

  no switchport

  vrf member Customer-77

  ip address 10.200.1.1/24

  no shutdown

 

interface Ethernet1/3

  no switchport

  ip address 10.2.3.3/24

  isis network point-to-point

  ip router isis SR

  mpls ip forwarding

  no shutdown

 

interface loopback0

  ip address 3.3.3.3/32

  ip router isis SR

icam monitor scale

 

line console

line vty

boot nxos bootflash:/nxos.9.3.5.bin sup-1

 

router isis SR

  net 49.0000.0000.0003.00

  is-type level-1

  address-family ipv4 unicast

    segment-routing mpls

router bgp 65077

  router-id 3.3.3.3

  address-family ipv4 unicast

  address-family vpnv4 unicast

  neighbor 1.1.1.1

    remote-as 65077

    update-source loopback0

    address-family vpnv4 unicast

      send-community extended

  vrf Customer-77

    address-family ipv4 unicast

      network 10.200.1.0/24

 

P-02# sh run

hostname P-02

install feature-set mpls

feature-set mpls

feature bgp

feature isis

feature mpls l3vpn

feature mpls segment-routing

feature mpls oam

feature mpls segment-routing traffic-engineering

 

vlan 1

segment-routing

  mpls

    connected-prefix-sid-map

      address-family ipv4

        2.2.2.2/32 absolute 16002

 

interface Ethernet1/2

  no switchport

  ip address 10.1.2.2/24

  isis network point-to-point

  ip router isis SR

  mpls ip forwarding

  no shutdown

 

interface Ethernet1/3

  no switchport

  ip address 10.2.3.2/24

  isis network point-to-point

  ip router isis SR

  mpls ip forwarding

  no shutdown

 

interface loopback0

  ip address 2.2.2.2/32

  ip router isis SR

 

router isis SR

  net 49.0000.0000.0002.00

  is-type level-1

  address-family ipv4 unicast

    segment-routing mpls

 

I have published these four books. You may find those useful.