Designlet: Understanding IPv6 in VMware Cloud on AWS

Introduction

VMware Cloud on AWS release 1.22 enables IPv6 workloads in a SDDC with an IPv4 management stack as well as provides the foundation for IPv6 traffic into and out of the SDDC via Direct Connect (DX) or VMware Transit Connect (vTGW). This designlet will provide the comprehensive baseline of configuration options available for customers that want to deploy IPv6. 

Summary and Considerations

Implementation Internal Connectivity

Every VMware Cloud on AWS SDDC shares a common initial topology consisting of a NSX Edge router which provides connectivity between external and internal resources. Two NSX routers are used to provide management connectivity, called the Management Gateway (MGW), and a router to provide connectivity to customer created workloads, called the Compute Gateway (CGW).  The figure below depicts a default SDDC topology.

In VMware Cloud on AWS version 1.22, the ability to enable IPv6 for workload communication at a segment level as well as North/South communications using Direct Connect (DX), VMware Transit Connect or IPsec route-based VPN were added. This designlet will illustrate the steps required to enable IPv6 in the SDDC, and configure segments for both dual stack with IPv4 and IPv6 addresses and IPv6 only operation. Then, it will also review the steps required to configure IPv6 for North/South communication via DX and VMware Transit Connect and IPsec route-based VPNs.

The first step required to use IPv6 in a SDDC is to enable it by using the Actions menu in the main VMC User Interface.

A pop-up window will appear asking to confirm the enablement.

When this is done, a series of tasks are performed to enable IPv6 in the underlying AWS infrastructure the SDDC is built upon. The NSX infrastructure is prepared to support IPv6 including transit links, route tables and API and User Interface are updated. These tasks take up to 5 minutes to complete but once done, are not needed for the remainder of the SDDC’s lifecycle.

With IPv6 enabled, the next step is to configure additional routed Compute Gateways, also referred to as Tier-1 or T1 routers. Compute Gateways in NAT or Isolated mode can support IPv6 workloads amongst segments connected to the same Compute Gateway. The support for IPv6 NAT configurations like NAT6 or NAT64 are not available in the SDDC. Isolated Compute Gateways may support IPv6 North/South communications if the Customer Managed Appliance (CMA) used to connect to the routed networks in the SDDC support it. Refer to the Tech Zone article “VMware Cloud on AWS Static Routing on Multiple CGWs (T1s)” for details on CMA considerations. When at least one additional Tier-1 router has been created, new segments can be created with the Connected Gateway being a non-default Compute Gateway. Segment creation with IPv6 addressing is shown below.

NOTE: IPv6 cannot be configured on a segment connected to the Default Compute Gateway. If attempting to connect a segment with IPv6 already configured is attempted, a message is displayed highlighting the incompatibility as shown below.

Once the segment is created, it is important to change the default IP discovery profile from “default-ip-discovery-profile” to the “vmc-adv-ipdiscovery-profile.” The advanced IP discovery profile enable IPv6 Neighbor Discovery snooping, DHCPv6 Snooping and IPv6 support for VMware Tools. Without selecting the advanced IP discovery profile, there may be intermittent IPv6 connectivity on the segment.   Selecting the advanced IP discovery profile is shown below.

Segments connected to non-default Compute Gateways can support IPv4 and IPv6 or IPv4 or IPv6 only as shown below.

Implementing External Connectivity - Transit Connect

With segments to host workloads using IPv6 defined the next step is to make those resources available outside of the SDDC. In VMC version 1.22, three external connectivity paths are supported. They are Direct Connect (DX), VMware Transit Connect and IPsec route-based VPNs. VMware Transit Connect supports SDDC to SDCC, SDDC to AWS VPC, and SDDC to on-premises through Direct Connect Gateway topologies. Starting with Direct Connect and Transit Connect the use of route aggregations will be required because the segments are connected to non-default Compute Gateways. Route aggregations are discussed in detail in this Tech Zone designlet “Understanding Route Aggregation in VMC on AWS.” For the purposes of IPv6, an IPv6 prefix list  must be configured and applied to the INTRANET Connectivity Point in the same manner as is done for IPv4.

In VMC on AWS 1.22, the Route Aggregation settings have been moved under the Global Configuration page in the NSX Manager GUI. In the Route Aggregation tab, create an Aggregation Prefix List as shown below and ensure the IPv6 address family is selected.

The image below shows both the IPv4 and IPv6 prefix lists defined.

Once the Aggregation Prefix List is created, it can be configured as a route aggregation and associated with the desired Connectivity Endpoint.  The image below shows the newly created IPv6 Route Aggregation Prefix List being added to a Route Configuration for the INTRANET Connectivity Endpoint which is shared by both Direct Connect and VMware Transit Connect.

With this Route Aggregation being applied to the INTRANET Connectivity Endpoint, the aggregations will be advertised to the Direct Connect and VMware Transit Connect. The aggregations can be observed in the SDDC’s Transit Connect pane as shown below.

The aggregations can also be viewed in the Transit Connect routing table as shown in the figure below. Note that the SDDC ID is the Target and that the Aggregation label is not observed. The aggregation label is locally significant to the SDDC where the aggregation is configured and is not seen outside of the SDDC.

Implementing External Connectivity - VPN

The third North/South option for IPv6 traffic is via IPsec route-based VPN. VMC on AWS version 1.22 supports IPv6 being carried over an IPv4 VPN Endpoint. This enables customers to utilize any non-IPv6 enabled intermediary transport to interconnect IPv6 enabled networks. Configuration of the VPN is exactly the same as an IPv4-only VPN with two changes. When configuring the BGP addresses for the local prefix and the remote IP, an IPv6 network needs to be added, as shown below.

IPv6 over IPv4 VPNs is also supported for T1 route-based VPNs using static routes.  Refer to the Understanding VPN to Customer Created NSX T1s in VMC on AWS article for details on T1 VPN configuration. 

In addition to configuring the IPv6 transit network for the BGP session, Compute Gateway Firewall rules need to be configured permitting the source and destination IPv6 addresses through for the BGP protocol, bidirectionally. The rules permitting IPv4 BGP are automatically included while for IPv6 they must be manually configured. The image below reflects these rules where the Virtual Tunnel Interface IPv6 addresses are the source and destinations for the respective rules and the policy is applied to the VPN Tunnel Interface.

IPv6 Operations

Operationally, tools such as Traceflow and Live Traffic Analysis support IPv6 as well. When using Traceflow, select IPv6 as the IP type and then select the source and destination. Traceflow is constrained to collecting detailed path analytics information inside the SDDC.

When the trace is complete, the results will be displayed showing detailed hop-by-hop information as shown below.

The detailed observations can also be reviewed showing not only the network path but the firewall and distributed firewall policies in place. This is reflected in the image below.

Summary

With VMC on AWS 1.22, IPv6 has been enabled so that customers can take advantage of it for workloads inside the SDDC and have support for three external North South pathways using Direct Connect, VMware Transit Connect and IPsec route-based VPNs.