This chapter is the first part of a series on Azure's highly available Network Virtual Appliance (NVA) solutions. It explains how we can use load balancers to achieve active/active NVA redundancy for connections initiated from the Internet.
In Figure 4-1, Virtual Machine (VM) vm-prod-1 uses the load balancer's Frontend IP address 20.240.9.27 to publish an application (SSH connection) to the Internet. Vm-prod-1 is located behind an active/active NVA FW cluster. Vm-prod-1 and NVAs have vNICs attached to the subnet 10.0.2.0/24.
Both NVAs have identical Pre- and Post-routing policies. If the ingress packet's destination IP address is 20.240.9.27 (load balancer's Frontend IP) and the transport layer protocol is TCP, the policy changes the destination IP address to 10.0.2.6 (vm-prod-1). Additionally, before routing the packet through the Ethernet 1 interface, the Post-routing policy replaces the original source IP with the IP address of the egress interface Eth1.
The second vNICs of the NVAs are connected to the subnet 10.0.1.0/24. We have associated these vNICs with the load balancer's backend pool. The Inbound rule binds the Frontend IP address to the Backend pool and defines the load-sharing policies. In our example, the packets of SSH connections from the remote host to the Frontend IP are distributed between NVA1 and NVA2. Moreover, an Inbound rule determines the Health Probe policy associated with the Inbound rule.
Note! Using a single VNet design eliminates the need to define static routes in the subnet-specific route table and the VM's Linux kernel. This solution is suitable for small-scale implementations. However, the Hub-and-Spoke VNet topology offers simplified network management, enhanced security, scalability, performance, and hybrid connectivity. I will explain how to achieve NVA redundancy in the Hub-and-Spoke VNet topology in upcoming chapters.
Load balancer Configuration using Azure Portal
Figure 4-2 depicts the configuration steps of Azure Loadbalancer building blocks and their relationships. After defining the basic information, such as resource group, name, and type, we associate a public IP address to Loadbalancer using Frontend IP configuration. We publish our service to the Internet with this Frontend IP (FIP). Next, we deploy a Backend pool where we attach our example VMs nva1 and nva2. Then we create a load balancing rule describing how data flows from the Internet is distributed between VMs. This step includes Frontend IP and Backend pool association. Besides, the load balancing rule states how VMs' health monitoring is done.
Figure 4-2: Load Balancer Configuration Steps.
Basic Information
Select the Create a resource from the Azure portal home view. Type the Load Balancer on the search field. Next, select Microsoft’s Load Balancer and click the Create button.
Figure 4-3: Start Load Balancer Deployment.
First, select the Subscription and resource group from the drop-down menu. Then fill in the Name field (lb-nva-pub) and choose a region from the drop-down menu. The SKU options for public Loadbalancers are Standard or Basic. Next, navigate to the Frontend IP Configuration tab.
Note! the Basic load balancer doesn’t support the following features: IP Based Backend, HTTPS health Probes, AZ redundancy, Diagnostic, HA Ports, Outbound Rules, TCP Reset on Idle, SLA, Global VNet Peering Support, Nat GW Support, Private Link Support, Global Tier.
Figure 4-4: Load Balancer Basics.
Frontend IP
We publish our service running on vm-prod-1 to the Internet using public IP address. Open the Frontend IP configuration window by clicking the Add a frontend IP configuration selector. Fill in the Name field (lb-nva-pub-fip-cfg). Select the IP version (IPv4) and Type (IP address). We don’t have any free public IP at this phase, so we create a new one by clicking the Create new hyperlink. Name the public IP (lb-nva-pub-fip) and leave the Routing preference to its default value. We are using Zone-redundancy in this example. Note that SKU (Standard), Tier (Regional), and Assignment (Static) are driven from the load balancer instance setting in the Basic tab and can’t be changed. In the figure, the Gateway Load balancer drop-down menu is left behind the Add a public IP Address window. Leave it to None since we are not chaining our load balancer with GLB at this phase (I’ll explain it in becoming chapters). When finished, click the Ok button.
Figure 4-5: Frontend IP Configuration.
The Frontend IP configuration lb-nva-pub-fip-cfg and the public IP address lb-nva-pub-fip are shown on the Frontend IP configuration window. To proceed, open the Backend pools tab.
Figure 4-6: Frontend IP Configuration Summary.
Figure 4-7 summarizes our Loadbalancer deployment process.
Figure 4-7: Frontend IP Deployment Progress Diagram.
Backend Pool
The backend pool (BEP) is a collection of backend resources, Virtual Machines (VM), or VM Scale Sets (VMSS) to which the load balancer distributes traffic. In order to create a BEP, navigate to the Backend pool settings window by clicking the Add a Backend pool in the Backend pool window.
Fill in the name field (lb-nva-pub-bep-in) and select the associated Virtual Network from the drop-down menu (vnet-hub).
You can select either a NIC-based or IP-based configuration. The primary difference between these two backend pool configurations lies in how the load balancer identifies backend resources.
IP-based backend pool: the load balancer identifies backend resources based on their IP addresses. Thus, any VM with an IP address in the backend pool will receive traffic from the load balancer.
NIC-based backend pool: The load balancer identifies backend resources based on their network interface cards. Hence, only VMs or VMSS instances with NICs explicitly added to the backend pool will receive traffic from the load balancer.
Figure 4-9: Backend Pool Configuration – Add New Backend Pool.
Select the Add selector for choosing the VMs you want to associate with the Backend pool. The Add IP configurations to the backend pool window show all VMs attached to Virtual Network with their Direct IP addresses (DIP) = Backend IP (BIP). We choose VMs connected to the 10.0.1.0/24 subnet.
Figure 4-10: Backend Pool Configuration – Adding VMs to Backend Pool.
Figure 4-11 shows that NVA1’s NIC nva1388 (10.0.1.4) and NVA2’s NIC nva2205 (10.0.1.5) are bind to Backend pool lb-nva-pub-bep-in. Besides, I have created a Backend pool named lb-nva-pub-bep-out for egress data flows (connection initiated by internal VMs) because outbound traffic may require different security and performance configurations than inbound traffic. For example, you may want to limit the types of traffic that can leave your network. However, we are not using the second backend pool in this chapter.
Figure 4-11: Backend Pool Configuration Summary.
Figure 4-12 summarizes our Loadbalancer deployment process.
Figure 4-12: Load Balancer Deployment Progress Diagram.
Inbound Rule
At this phase, we have created a Frontend IP (FIP) lb-nva-pub-fip. Besides, we have associated our nva's NICs attached to subnet 10.0.1.0/24 to the Backend pool (BEP) lb-nva-pub-bep-in. An Inbound rule binds FIP to BEP and defines the protocol (TCP or UDP) and the destination port. Besides, it specifies if we are using 5-tuple (source IP, source port, destination IP, destination port, and protocol), 3-tuple (source IP, destination IP, and protocol), or 2-tuple (source IP, and destination IP) hashing for ingress flow distribution. Among these things, Health Probe is associated with the Inbound rule for VM reachability checking. Loadbalancer distributes traffic only among healthy VMs.
Click the Add a load balancing rule selector to start the Inbound rule configuration.
Figure 4-13: Inbound Rule Configuration.
Start by filling in the Name field (lb-nva-pub-lb-rule-in) and select the IP Version (IPv4). Next, choose the Frontend IP address (lb-nva-pub-fib-cfg) and Backend pool (lb-nva-pub-bep-in) from the drop-down menus. Our service allows SSH connection from the internet to vm-prod-1, so we use TCP port 22. Similarly, in the Backend pool field, we will use port 22 since our example NVAs listen for SSH connection initiation on TCP port 22. If the application running on a VM uses a different port than the standard one, we can rewrite it when sending packets to the VM.
Figure 4-14: Inbound Rule Configuration.
Scroll down to Health Probe (HP) selection field. We don’t have an existing HP, so we create a new one. Name the HP and define the protocol, port, and polling interval. We use the same protocol and port in HP as in our Inbound rule distribution policy. Click the OK button to proceed.
Figure 4-15: Inbound Rule Configuration - Health Probe.
Idle timeout defines the times in minutes of how long the load balancer keeps the inactive connections in its flow table. The default value is 4 minutes. By enabling TCP reset, a load balancer sends a TCP Resets message to both connection endpoints after Idle time expiration. In our example, we have only one SSH application running on NVAs, so we don’t need to use Floating IPs. Leave the last option to its default, recommended selection for outbound access using the load balancer outbound rule. I will explain it in the upcoming chapter NVA Redundancy – Internet Connection.
Figure 4-16: Inbound Rule Configuration.
Figure 4-17 shows the Inbound rule lb-nva-pub-lb-rule-in configuration. It binds the Frontend IP to the Backend pool lba-nva-pub-bep-in. When Loadbalancer receives SSH connection initiation to Frontend IP, it uses a 5-tuple hash for selecting an NVA to which it forwards SSH packets without changing the destination port. We do not need to create an Outbound rule at this phase. I will explain it in the next chapter where we enable the Internet connection for vm-prod-1.
Figure 4-17: Inbound Rule Summary.
Figure 4-18 on the next page summarizes our Loadbalancer deployment process.
Figure 4-18: Load Balancer Deployment Progress Diagram.
NVA Configuration
Next, we enable IP Forwarding and create a NAT policy on both NVAs.
Enable IP Forwarding on NIC
To redirect or forward traffic that is not intended for the virtual machine (VM) itself, you need to enable IP Forwarding on the virtual network interface card (vNIC) and the Linux kernel. This configuration allows the VM to act as a forwarding device for network traffic.
az network nic update `
--name vm-nva-nic-west `
--resource-group rg-nwkt-demo `
--ip-forwarding true
Example 4-1: Enable IP Forwarding on vNIC.
Alternatively, you can use the Azure Portal for enabling IP forwarding on NIC.
Figure 4-19: Enabling the IP Forwarding on NIC Using an Azure Portal.
Enable IP Forwarding on Linux NVA
By changing the value of net.ipv4.ip_forwarding entry in the sysctl file, you can alter the IP Forwarding behavior on Linux. The example below shows that IP Forwarding is disabled by default.
azureuser@vm-nva-fw:~$ sudo sysctl -a |grep ip_forward
net.ipv4.ip_forward = 0
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0
Example 4-2: Verify the State of the “IP-Forward” on Linux IP.
For enabling IP Forwarding on Linux, use the CLI command “sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward". The echo command sets the IP Forward value 1 to the file located at /proc/sys/net/ipv4/ip_forward.
azureuser@vm-nva-fw:~$ sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
azureuser@vm-nva-fw:~$ sudo sysctl -a |grep ip_forward
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0
Example 4-3: Enable “IP-Forward” using Linux Shell.
Alternatively, you can use the Azure CLI for configuring Linux IP forwarding using the command “az vm extension set”. The example below shows how you can enable IP Forwarding.
az vm extension set `
--resource-group rg-nwkt-demo `
--vm-name vm-nva-fw `
--name customScript `
--publisher Microsoft.Azure.Extensions `
--settings '{"commandToExecute":"sudo sysctl -w net.ipv4.ip_forward=1"}'
Example 4-4: Enable “IP-Forward” using Azure CLI.
Configuring destination NAT on Linux
Next, we configure the NAT policy on the Linux kernel Firewall. The command iptables is a user-space utility program in Linux that we use to configure and manage the firewall ruleset. The “-t nat” option defines the table within iptables that you want to operate on. The example below specifies the NAT table (responsible for Network Address Translation functionality).
The -A PREROUTING option appends a new rule to the PREROUTING chain of the NAT table. The PREROUTING chain is used to modify packets as they arrive on the network interface before any routing decisions are made. The option -s defines the source IP, and -p defines the protocol. The -j DNAT option specifies the target IP address. We intend to redirect TCP messages received from the source IP 141.192.166.93 (Remote PC) to destination 10.0.2.6 (vm-prod-1).
sudo iptables -t nat -A PREROUTING -s 141.192.166.93 -p tcp -j DNAT --to-destination 10.0.2.6
Example 4-5: Destination NAT (DNAT) on Linux Kernel.
Configuring source NAT on Linux
After the destination NAT configuration, we set up the source NAT policy. The option -A POSTROUTING appends a new rule and adds a rule to the POSTROUTING chain of the NAT table. The POSTROUTING chain is used for modifying packets as they are about to be routed out of the system. The option -j MASQUERADE replaces the source IP address of outgoing packets with the IP address of the outbound interface (in this case, eth1), allowing the system to perform source NAT.
sudo iptables -t nat -A POSTROUTING -s 141.192.166. 93 -o eth1 -j MASQUERADE
Example 4-6: Source NAT (SNAT) on Linux Kernel.
Example 4-7 shows our DNAT and SNAT policies.
azureuser@nva1:~$ sudo iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DNAT tcp -- 141.192.166.93 0.0.0.0/0 to:10.0.2.6
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 141.192.166.93 0.0.0.0/0
Example 4-7: NAT Verification.
The tcpdump outputs in the following two examples verify that the NAT operation on NVA1 Firewall works as expected. The Firewall rule redirects packets with the destination IP 141.192.166.93 to 10.0.2.6 (DNAT and SNAT operation). Option -n disables the name resolution, and -t removes the time stamp from the output.
azureuser@nva1:~$ sudo tcpdump -i eth0 host 141.192.166.93 -n -t
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 141.192.166.93.58087 > 10.0.1.4.22: Flags [snipped for brevity]
IP 10.0.1.4.22 > 141.192.166.93.58087: Flags [snipped for brevity]
IP 141.192.166.93.58087 > 10.0.1.4.22: Flags [snipped for brevity]
Example 4-8: TCP Dump from the NVA1 (Eth0).
azureuser@nva1:~$ sudo tcpdump -i eth1 host 10.0.2.6 -n -t
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 10.0.2.4.58087 > 10.0.2.6.22: Flags [snipped for brevity]
IP 10.0.2.6.22 > 10.0.2.4.58087: Flags [snipped for brevity]
IP 10.0.2.4.58087 > 10.0.2.6.22: Flags [snipped for brevity]
Example 4-9: TCP Dump from the NVA1 (Eth1).
The figure below summarizes the complete configuration steps.
Figure 4-20: Complete Load Balancer Deployment and NVA Configuration Diagram.
Packet Walk
Figure 4-21 depicts an overview of the packet walk of a TCP SYN message when a Remote PC initiates an SSH connection with vm-prod-1.
Step 1: The remote PC sends the first IP packet with the source IP address 141.192.166.93 and the destination IP address 20.240.9.27 (Frontend IP). It uses a random TCP port number 49567 as a source port and 22 as a destination port.
Step 2 - Load balancer’s inbound packet processing: The packet arrives at the load balancer. It notices that the destination IP address is one of its Frontend IP (FIP) addresses and starts packet processing. Because this is the first packet of the flow, the load balancer doesn’t have related entries either in its Frontend IP to Backend IP mapping (FIP to BIP) table or Flow table. The FIP 20.240.9.27 is bound to the Backend pool to which we have attached our NVAs 1 & 2 (for simplicity, only NVA1 is shown in figure 4-21). The load balancer runs a hash algorithm from the 5-tuple flow information. As a result, it selects NVA1 (Backend IP 10.0.1.4) for the forwarding destination. Based on the FIP to BIP mapping result, it also creates a flow entry for connection: src-IP: 141.192.166.93, dst IP: 10.0.1.4, Protocol: TCP, src Port: 49567, dst Port: 22.
The high availability of Azure load balancer service relies on MUX pools consisting of several virtual load balancers which synchronize their distribution policies, flow tables, FIP-to-BIP mapping information, and other necessary information. Besides, all load balancers advertise the Frontend IP address ranges to their upstream Azure routers, which distribute reachability information to their Internet peers (with the proper summarization). Azure routers use BGP multi-path properties, so the next packet may enter another load balancer, depending on which router receives the ingress traffic and which path it selects. Due to synchronized data plane information, load balancers in a MUX can distribute the traffic.
Step 3 - Load balancer’s outbound packet processing: To create a tunnel header and forward the packet to NVA1, the load balancer needs to know the IP address of the physical server where NVA1 is running. Besides, it needs the know Virtual Network Identifier (VNI) for vnet-hub. The load balancer gets the required information from the Virtual Network controller. In our example, NVA1 is running on a physical server Host-A. The VNI for vnet-hub is 10000.
The load balancer creates tunnel headers using the destination IP address of Host-A and the VNI 1000. Then it encapsulates the original message and sends it to the upstream router. I have used VNet icons in the encapsulation/decapsulation processes because the Virtual Network in Azure is an on-demand tunnel scheme using VXLAN or NVGRE tunneling.
Step 4 - The packet processing on vm NVA1: The packet arrives at Host-A with its destination IP on the tunnel header belonging to Host-A. As it is the first packet of the flow, there is no related flow entry in the Generic Flow Table (GFT). Consequently, the VFP layers process the packet before forwarding it to interface Eth0 of NVA1.
During the Post-routing process, the original destination IP address 20.240.9.27 is replaced with 10.0.2.6 (vm-prod-1). After resolving the destination IP address, the routing process identifies the egress interface, which is Eth1.
Next, the Post-routing policy deploys a Source NAT by changing the source IP address 141.192.166.93 to the IP address 10.0.2.4 configured on the interface Eth1. Finally, NVA1 routes the packet out of its interface Eth1.
The ingress packet generated an entry in the UFT (User Flow Table), allowing the egress packet to be switched based on the flow entry. The tunneling process follows the same principles as with load balancers.
Steps 5 & 6 The packet processing on vm-prod-1: The packet processing is the same as I described with NVA1, excluding the NAT operations. From the vm-prod-1 perspective, the SSH connection initiator is NVA1 and sends a TCP SYN-ACK message to it. It uses its backend IP address 10.0.2.6 as a source and 10.0.2.4 as a destination in the message. Then, based on the flow-specific entry in the GFT table, the packet is encapsulated with the tunnel headers and forwarded to NVA1.
Step 7 - The TCP-SYN packet processing on vm NVA1: When the TCP-SYN packet arrives at NVA1 via Interface Eth1, it deploys both Pre-routing and Post-routing policies, i.e., source and destination NAT operations. The difference between the TCP SYN and TCP SYN-ACK message processing is that the TCP-SYN message to Remote PC is forwarded to Azure’s Internet edge bypassing the load balancer and without tunnel encapsulation. This Direct Server Return (DSR) operation is used when the packet has a public IP address as a destination.
The TCP ACK message may arrive at a different load balancer than the TCP SYN message. Due to the load balancer synchronization process, all load balancers in a MUX pool can forward packets based on their flow table. Note that the destination for this flow is always NVA1.
References
[1] Azure Load Balancer portal settings
https://learn.microsoft.com/en-us/azure/load-balancer/manage
[2] Azure Load Balancer components
https://learn.microsoft.com/en-us/azure/load-balancer/components
[3] Backend pool management
https://learn.microsoft.com/en-us/azure/load-balancer/backend-pool-management
[4] Configure the distribution mode for Azure Load Balancer
https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-distribution-mode?tabs=azure-portal
[5] Outbound rules Azure Load Balancer
https://learn.microsoft.com/en-us/azure/load-balancer/outbound-rules
[6] Azure Load Balancer SKUs
https://learn.microsoft.com/en-us/azure/load-balancer/skus
[7] Load balancer limits
https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#load-balancer
[8] Azure Load Balancer health probes
https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview#probe-down-behavior
[9] Load Balancer TCP Reset and Idle Timeout
https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-tcp-reset
[10] Multiple frontends for Azure Load Balancer
https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-multivip-overview
[11] Azure Load Balancer Floating IP configuration
https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-floating-ip
[12] Use Source Network Address Translation (SNAT) for outbound connections
https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-connections#outboundrules
[13] Default outbound access in Azure
https://learn.microsoft.com/en-us/azure/virtual-network/ip-services/default-outbound-access
