VMware Virtual SAN Layer 2 and Layer 3 Network Topologies, Summaries of Communication

Download VMware Virtual SAN Layer 2 and Layer 3 Network Topologies and more Summaries Communication in PDF only on Docsity!

VMware Virtual SAN

Layer 2 and Layer 3

Network Topologies

Deployments

T E C H N I C A L W H I T E P A P E R

Network Topologies Deployments

• Introduction Table of Contents
• Network and vSphere Technologies
  • Networking Related Technologies
    • IP Multicast.......................................................................................................................................................
    • Internet Group Management Protocol (IGMP)
    • Protocol-Independent Multicast (PIM)
  • vSphere Related Technologies
    • vSphere Virtual Switch
    • VMkernel Network Interface
    • Static Routes...................................................................................................................................................
    • Hosts Profiles
• Supported Network Topologies
  • Layer 2 Network Topologies
  • Layer 2 Physical Network Configuration
    • Cisco Hardware Devices
    • Brocade Hardware Devices.....................................................................................................................
  • Layer 3 Network Topologies
  • Layer 3 Physical Network Configuration
• Virtual Network Configuration
  • Creating vSphere Distributed Switch
  • Creating vSphere Distributed Port Groups
  • Creating VMkernel Network Interface for Virtual SAN.................................................................
  • Host Configuration Information................................................................................................................
  • Adding Host Static Routes
  • Enable and Configure Virtual SAN
• Validating Virtual SAN Configuration and Health
• Summary
• Acknowledgments
• Author

Network Topologies Deployments

Multicast Groups on the Layer 3 segment where the receiver is connected. This allows the switch to keep a table of the individual receivers that need a copy of the Multicast Group traffic.

The participating hosts in a Virtual SAN cluster will negotiate for IGMP version 3. If the network does not support IGMP version 3, the hosts will fall back to IGMP version 2. VMware recommends that the same version of IGMP be used in all Layer 3 segments.

Protocol-Independent Multicast (PIM)

Protocol-Independent Multicast (PIM) is a family of Layer 3 multicast routing protocols that provide different communication techniques for IP Multicast traffic to reach receivers that are in different Layer 3 segments from the Multicast Groups sources. There are different versions of PIM, each of which is best suited for different IP Multicast topologies. The main four versions of PIM are these:

• PIM Dense Mode (PIM-DM) – Dense Mode works by building a unidirectional shortest-path tree from each Multicast Groups source to the Multicast Groups receivers, by flooding multicast traffic over the entire Layer 3 Network and then pruning back branches of the tree where no receivers are present. Dense Mode is straightforward to implement and it is best suited for small Multicast deployments of one-to-many.
• PIM Sparse Mode (PIM-SM) – Sparse Mode avoids the flooding issues of Dense Mode by assigning a root entity for the unidirectional Multicast Groups shortest-path tree called a rendezvous point (RP). The rendezvous point is selected in a per Multicast Group basis.

Figure 1: Layer 3 Network PIM Sparse Mode Communication Flow

Sparse Mode scales fairly well for larger Layer 3 Networks and is best suited for one-to-many Multicast topologies. If the network only supports IGMP version 2, VMware recommends the use of PIM-SM for Virtual SAN deployments over Layer 3.

• Bidirectional PIM (Bi-PIM) – Bidirectional PIM assumes that there are many MGs that have many sources and many receivers (many-to-many). Whereas Sparse Mode can manage many-to-many Multicast topologies, Bidirectional PIM does it by reducing the load on the Multicast routers as compared to Sparse Mode. Bidirectional PIM does not build a shortest-path tree, so MG data paths may have longer end-to-end delays than Sparse Mode, however Bidirectional PIM allows for a Multicast Group traffic to flow both ways over the same data path.
• PIM Source-Specific Multicast (PIM-SSM) – Source Specific Multicast is similar to Sparse Mode but it carries information about the IP of the source. Receivers join Multicast Groups based on the source of the Multicast Groups.

Network Topologies Deployments

Figure 2: Layer 3 Network PIM Source Specific Mode Communication Flow

With Source Specific Multicast, shortest-path trees are built and are rooted in just one source, offering a more secure and scalable model for a limited amount of applications (mostly broadcasting of content).

If the networks are configured with IGMP version 3, then Source Specific Multicast requires the receivers to support IGMP version 3.

vSphere Related Technologies

vSphere Virtual Switch

VMware Virtual SAN supports the use of both the vSphere Standard Switch and vSphere Distributed Switch. However, VMware recommends the use of the vSphere Distributed Switch to take advantage of its centralized management capabilities as well as advanced network features.

Figure 3: VMware Distributed Switched - QoS with Network I/O Control

The Virtual SAN network configuration can be implemented with vSphere standard or distributed switches. In either case, the networking configuration requirements and behavior remain relatively the same.

vSphere Distributed switches provide several advantages around management, advanced network features, and scalability capabilities that are all conducive the benefits and values of VMware Virtual SAN.

vSphere Distributed Switches facilitate large scale deployments with the support of up to 500 hosts per switch. They also provide access to advanced network features such as Network I/O Control and IP Multicast Filtering.

For scenarios where different network traffic services share physical network adapters, VMware recommends the

Network Topologies Deployments

At this point, the hosts will establish their method of communication by joining the Virtual SAN default Multicast Group addresses, 224.1.2.3 and default Multicast Group Agent addresses 224.2.3.4.

In order to avoid unnecessary IP multicast floods within the Layer 2 segments, VMware recommends configuring IGMP snooping with an IGMP Querier in order to control the number of physical ports on the switches that will receive IP multicast frames.

For optimal network communication and efficiency, Virtual SAN multicast frames should be exclusively forwarded to the ports that are associated with the uplinks of the VMkernel network interfaces that are configured to carry the Virtual SAN traffic.

Figure 5: Multiple Virtual SAN Clusters

In scenarios with multiple Virtual SAN clusters, VMware recommends changing the default Multicast Group address and the default Multicast Group Agent address when the different clusters will share the same Layer 2 network segment.

This will prevent the clusters from receiving unnecessary multicast frames from one another.

In scenarios where members of a cluster have been deployed across different network segments (Layer 3), VMware recommends changing the default Multicast Group address and default Multicast Group Agent address.

VMware recommends the use of the Multicast Address range of 239.0.0.0/8 when changing the default addresses. Also, consult with members of the network team in order to identify the adequate Multicast Group addresses to use in order to comply with any potential Multicast Addressing policies that may exist.

For detailed instruction on how to change the default multicast address for Virtual SAN, please refer to the VMware Knowledge Base article 2075451.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=

TCP/IP Stacks

vSphere 6.0 introduced a new TCP/IP Stack architecture where multiple TPC/IP stacks can be utilized to manage different VMkernel network interfaces and their associated traffic.

As a result, the new architecture provides the ability to configure traffic services such vMotion, Management, Fault Tolerance, etc. on completely isolated TCP/IP stacks with the ability to use multiple default gateways.

For network traffic isolation and security requirements, VMware recommends deploying the different traffic services onto different network segments in an order to prevent the different traffic services from traversing through the same default gateway.

Network Topologies Deployments

Figure 6: vSphere Multiple TCP/IP Stack Architecture

In order to configure the traffic services onto separate TCP/IP stacks, each traffic service type needs to be deployed onto their own network segments.

The network segments will be accessed through a physical network adapter with VLAN segmentation and individually mapped to dissimilar VMkernel network interfaces with the respective traffic services (Virtual SAN, vMotion, Management, etc.) enabled.

Built-in TCP/IP stacks available in vSphere:

• Default^ TCP/IP Stack^ – multi-purpose stack that can be used to manage any of the host related traffic services. Shares a single default gateway between all configured network services.
• vMotion TCP/IP Stack – utilized to isolate vMotion traffic onto its own stack. The use of this stack completely removes or disable vMotion traffic from the default TCP/IP stack.
• Provisioning TCP/IP Stack – utilized to isolate some virtual machine related operations such as cold migrations, cloning, snapshot, NFC related traffic.

It is assumed that environments with isolated network requirements for the vSphere traffic services will not be able to use the same default gateway to direct traffic.

The use of the different TCP/IP stacks facilitates the management for traffic isolation with the ability to use different default gateways.

Currently, vSphere 6.0 does not include a dedicated TCP/IP stack for the Virtual SAN traffic service nor the supportability for the creation of custom Virtual SAN TCP/IP stack.

To ensure Virtual SAN traffic in Layer 3 network topologies leaves over the Virtual SAN VMkernel network interface, add the Virtual SAN VMkernel network interface to the Default TCP/IP Stack and define static routes for all of the Virtual SAN cluster members.

Static Routes

The use of static routes is required by traffic services for which vSphere does not provide a non-Default TCP/IP stack.

In the VMware recommended deployment scenario where the Management and Virtual SAN traffic services are configured to use different Layer 3 network segments, they will share the Default TCP/IP Stack but be configured in different Layer 2 domains.

The default route for the Default TCP/IP Stack should remain with the Management VMkernel network interface. Static routes will be added for the Virtual SAN traffic to egress of the Virtual SAN VMkernel network interface.

Network Topologies Deployments

Static routes are stored within the Hosts Profiles as part of their catalog parameters. Host Profiles can be applied to either individual hosts or a cluster; applying a Host Profile to a cluster will affect all hosts in the cluster and result in a consistent configuration across all hosts in that cluster.

Host Profiles can also be used to validate the system configuration by checking compliance for any host or cluster against an associated standardized Host Profile.

Supported Network Topologies

This section covers the different supported network topologies and the impact they introduce to the overall deployment and management of Virtual SAN in different network scenarios.

Layer 2 Network Topologies

Layer 2 network topologies are defined as networking architectures that are composed of devices that operate at the Data Link layer (Layer 2) of the OSI model.

This network topology is responsible for forwarding packets through intermediate Layer 2 devices such as hosts, bridge, or switches.

It is required that all of the hosts participating in a Virtual SAN cluster are able to establish communication through the VMkernel interface connected to a common Layer 2 network segment.

The Layer 2 network topology offers the least complex implementation and management of the IP Multicast requirements for Virtual SAN while constraining the radius of the cluster.

All cluster members will send IGMP join requests over the VMkernel network interfaces that are used for the Virtual SAN traffic service.

By default, the hosts will negotiate their communication for IGMP version 3 and failback to IGMP version 2 whenever the physical network device does not support IGMP version 3.

For maximum Layer 2 traffic efficiency, VMware recommends the use and configuration of IGMP Snooping in all the switches configured in the Layer 2 network segment where Virtual SAN is present.

IGMP Snooping allows physical network devices to forward Multicast frames to only the interfaces where IGMP Join requests are being observed.

Layer 2 Physical Network Configuration

This section covers the physical network configuration procedures to enable IP Multicast for Virtual SAN. The configuration is focused on IGMP snooping and IGMP snooping Querier.

We will assume all members of the cluster are in the same Layer 2 network segment, represented by VLAN 10. In this scenario the role of IGMP Querier will be performed by a physical switches and not a router.

Network Topologies Deployments

Figure 9: Virtual SAN Layer 2 Network Topology

For this scenario we will provide two different configuration examples that will be based on two different vendor platforms. The first example is based on the configuration of Cisco Nexus switch devices operating with the Cisco Nexus platform OS with IGMP version 3.

The second example is based on the configuration of Brocade VDX switch devices with IGMP version 2. Currently, Brocade VDX switch devices do not support IGMP version 3 and therefore the configuration will be based on IGMP version 2.

The configuration procedures for IP Multicast varies between different vendors and their respective network devices. Consult the network device vendor documentation for in-depth details and specific advanced procedures that go beyond the scope of this document.

Network Topologies Deployments

Layer 3 Network Topologies

Layer 3 network topologies are defined as networking architectures that are composed of devices that are capable of operating at the network layer (Layer 3) of the OSI model.

This network topology is responsible for routing packets through intermediate Layer 3 capable devices such as routers and Layer 3 capable switches.

All Virtual SAN cluster members are required to join the cluster’s Multicast Group by sending IGMP Join requests over the VMkernel network interfaces that are being used for the Virtual SAN traffic service.

Whenever hosts are deployed across different Layer 3 network segments, the result is a routed network topology.

Figure 10: Virtual SAN Over a Layer 3 Network Topology

However, since there is a need for those requests to be sent by each Layer 3 segment Default Gateway, the IGMP Querier has to be the Default Gateway itself.

The Default Gateway will use the Multicast Group memberships from the IGMP Joins to update the PIM protocol running.

In Layer 3 Network topologies, VMware recommends the use and configuration of IGMP Snooping in all the switches configured in the Layer 2 domains where hosts participating in the Virtual SAN cluster will be present.

Network Topologies Deployments

Layer 3 Physical Network Configuration

This section covers the configuration procedures for IGMP snooping, IGMP Querier and PIM. We will assume that there are three Layer 2 domains, each with its own Layer 3 segment. The Layer 2 domains will be represented by VLANs 10, 172 and 192, as shown in the figure below.

Two configuration examples are provided: one based on the Cisco Nexus platform (with IGMP version 3 and Source Specific Multicast) and the Brocade VDX (with IGMP version 2 and Sparse Mode).

Configuration procedures are typically different based on hardware vendor’s implementation. Consult the hardware vendor documentation for in-depth and specific procedures that are beyond the scope of this document.

Figure 11: Layer 3 Network Logical Design

N E T W O R K S S U B N E T S V L A N R O U T E R L O 0 M G M A G M R P VSAN1 172.16.10.0/24 172 R1 1.1.1.1 224.1.2.3 224.2.3.4 2.2.2. VSAN2 192.16.10.0/24 192 R2 2.2.2.2 224.1.2.3 224.2.3.4 2.2.2. VSAN3 10.16.10.0/24 10 R3 3.3.3.3 224.1.2.3 224.2.3.4 2.2.2.

Table 1: Network Information Configuration Table

MGM - Master Group Multicast Lo0 - Loopback Interface 0 AGM - Agent Group Multicast RP - Rendezvous Point

Network Topologies Deployments

Cisco Router 3 (R3)

configure terminal feature pim ip pim rp-address 2.2.2.2 group-list 224.1.2.3/ ip pim rp-address 2.2.2.2 group-list 224.2.3.4/

interface vlan 203 description Network Uplink ip address 20.1.3.1/ ip pim sparse-mode

interface vlan 10 ip address 10.16.10.253/ ip router ospf 9 area 0.0.0. ip igmp snooping ip igmp snooping querier 10.16.10.

interface Loopback 3 ip address 3.3.3.3/ ip router ospf 9 area 0.0.0.

Network Topologies Deployments

Brocade Hardware Devices

The following a sample configuration of IGMP version 2 and Sparse Mode in VDX 6740s running NOS 7.0.0.

Brocade Switch 1 (S1)

configure terminal ip igmp snooping enable interface vlan 172 ip igmp snooping enable

Brocade Switch 2 (S2)

configure terminal ip igmp snooping enable interface vlan 192 ip igmp snooping enable

Brocade Switch 3 (S3)

configure terminal ip igmp snooping enable interface vlan 10 ip igmp snooping enable

Brocade Router 1 (R1)

configure terminal interface vlan 201 interface vlan 172 ip igmp snooping enable ip igmp snooping querier enable

rbridge-id 101 router pim rp-address 2.2.2.

router ospf area 0.0.0.

interface loopback 1 ip address 1.1.1.1/ ip ospf area 0.0.0. no shutdown

interface ve 201 description Network Uplink ip address 20.1.1.1/ ip ospf area 0.0.0. ip pim-sparse no shutdown

interface ve 172 ip address 172.16.10.1/ ip ospf area 0.0.0. no shutdown

Network Topologies Deployments

Virtual Network Configuration

This section details the configuration procedures for the virtual network components and features such as vSphere Distributed Switch, vSphere Distributed Port Groups, VMkernel Network Interfaces, Virtual SAN Traffic service, and hosts static routes.

Creating vSphere Distributed Switch

Create a vSphere distributed switch on a data center to manage the networking configuration of multiple hosts at a time from a central place.

• From the vSphere Web Client, navigate to a data center.
• In the navigator, right-click the data center and select Distributed Switch > New Distributed Switch.
• In Name and Location, type a name for the new distributed switch and click Next.
• Select version, select the compatible with ESXi 6.0 and later and click Next
• In Edit Settings configure the distributed switch settings according to environment requirements. Click next, then Finish.

Creating vSphere Distributed Port Groups

Add a distributed port group to a vSphere Distributed Switch to create a distributed switch network to associate with VMkernel adapters.

• From the vSphere Web Client, navigate to the distributed switch.
• Right-click the distributed switch and select Distributed port group > New distributed port group.
• In the Select name and location section, type the name of the new distributed port group, VSAN1, and click Next.
• In the Configure settings section, configure VLAN (172), and Failover Order. Set one uplink to active, and the other to standby, then keep the default settings beyond that and click Next, and then Finish.

Creating VMkernel Network Interface for Virtual SAN

Create a VMkernel adapter on a host that is associated with a distributed switch to provide network connectivity to the host and to handle the traffic for Virtual SAN. Dedicate a single distributed port group per VMkernel adapter. For better isolation, you should configure one VMkernel adapter with one traffic type.

• From the vSphere Web Client, navigate to the host
• Under Manage, select Networking and then select VMkernel adapters.
• Click Add host networking.
• On the Select connection type page, select VMkernel Network Adapter and click Next.
• From the Select an existing network option, select a distributed port group and click Next.
• On the Port properties page, configure the settings for the VMkernel adapter based on the network information listed on table 2. Enable the Virtual SAN traffic service, then click Next, then Finish.

Network Topologies Deployments

Host Configuration Information

N E T W O R K S H O S T S V S A N V M K I P S U B N E T S V L A N VSAN1 octo.vsan.a.01 172.16.10.9/24 172.16.10.0/24 172 VSAN1 octo.vsan.a.02 172.16.10.10/24 172.16.10.0/24 172 VSAN1 octo.vsan.a.03 172.16.10.11/24 172.16.10.0/24 172 VSAN1 octo.vsan.a.04 172.16.10.12/24 172.16.10.0/24 172 VSAN2 octo.vsan.b.01 192.16.10.9/24 192.16.10.0/24 192 VSAN2 octo.vsan.b.02 192.16.10.10/24 192.16.10.0/24 192 VSAN2 octo.vsan.b.03 192.16.10.11/24 192.16.10.0/24 192 VSAN2 octo.vsan.b.04 192.16.10.12/24 192.16.10.0/24 192 VSAN3 octo.vsan.c.01 10.16.10.9/24 10.16.10.0/24 10 VSAN3 octo.vsan.c.02 10.16.10.10/24 10.16.10.0/24 10 VSAN3 octo.vsan.c.03 10.16.10.11/24 10.16.10.0/24 10 VSAN3 octo.vsan.c.04 10.16.10.12/24 10.16.10.0/24 10

Table 2: Host Network Information Configuration Table

Adding Host Static Routes

Static routes are used to instruct the Default TCP/IP Stack to use a different default gateway to direct the Virtual SAN traffic through the necessary paths to reach the remote Virtual SAN networks.

Static routes are required by all the hosts between all the different individual Virtual SAN networks.

N E T W O R K S S U B N E T S G A T E W A Y S V L A N S R O U T E R S VSAN1 172.16.10.0/24 172.16.10.253 172 R VSAN2 192.16.10.0/24 192.16.10.253 192 R VSAN3 10.16.10.0/24 10.16.10.253 10 R

Table 3: Virtual SAN Network Addresses

• Static Routes for hosts on VSAN 1 Network: esxcli network ip route ipv4 add –g 172.16.10.253 -n 192.168.10.0/ esxcli network ip route ipv4 add –g 172.16.10.253 -n 10.16.10.0/
• Static Routes for hosts on VSAN 2 Network: esxcli network ip route ipv4 add –g 192.168.10.253 -n 172.16.10.0/ esxcli network ip route ipv4 add –g 192.168.10.253 -n 10.16.10.0/
• Static Routes for hosts on VSAN 3 Network: esxcli network ip route ipv4 add –g 10.16.10.253 -n 172.16.10.0/ esxcli network ip route ipv4 add –g 10.16.10.253 -n 10.16.10.0/

After adding the static routes, the Virtual SAN traffic connectivity should be available across all networks.

Use the vmkping command test and confirm communication between the different networks by pinging the different default gateway from all three networks.

• Test connectivity to remote hosts from VSAN 1 Network: vmkping –I vmk3 192.168.10. vmkping –I vmk3 10.16.10.
• Test connectivity to remote hosts from VSAN 2 Network: vmkping –I vmk3 172.16.10. vmkping –I vmk3 10.16.10.