VMware Cloud on AWS Outposts - deep dive

Introduction

In today's world, cloud computing has become an integral part of the IT industry. VMware is one of the most popular cloud computing platforms, widely used for its powerful virtualization capabilities and the unique true Multi Cloud capabilities. With the advent of VMware Cloud on AWS Outposts, the boundaries between cloud and on-premises infrastructure are blurring. This hybrid cloud solution provides a seamless way to consume the VMware Cloud managed service in your private data center. One of the key aspects of this hybrid cloud solution is networking, which is crucial for ensuring efficient communication between your on-premises infrastructure and the VMware Cloud on AWS Outposts Software-Defined Data Center (SDDC). In this blog, we will take a deep dive into the networking aspects of VMware Cloud on AWS Outposts, discussing its architecture, components, and how it all works together. 

VMware Cloud on AWS Outposts – Pre-requirements

Site Readiness

To ensure a successful deployment of VMware Cloud on AWS Outposts, AWS provides comprehensive support during the site survey, site readiness, and network validation phases. 

Site Survey: AWS provides a Site Survey Guide that provides detailed instructions on how to prepare for the site survey, including a checklist of prerequisites such as power, and cooling. AWS also provides a list of requirements regarding the rack's connectivity to your existing network, including cabling and protocol requirements. 

Site Readiness: Once the site survey is complete, AWS provides a Site Readiness Checklist that outlines the necessary steps to prepare the site for the installation of VMware Cloud on AWS Outposts. The checklist includes requirements for network connectivity, network device configuration and physical space requirements. 

Network Validation and Site visit: Once the site survey and site readiness checklists are completed, AWS will visit the final location and perform a validation that ensures the network infrastructure is configured correctly to support VMware Cloud on AWS Outposts. The Network Validation steps include a review of the network topology, configuration, and security posture to identify any potential issues or areas for optimization. The Site visit also includes a series of tests to validate the network connectivity and performance between the on-premises environment and the cloud. 

Overall, VMware and AWS provides a comprehensive support program to ensure the successful deployment of VMware Cloud on AWS Outposts. The site survey, site readiness, and network validation are designed to identify and resolve any potential issues before deployment, ensuring a smooth and seamless experience for the customer. 

Physical Connection

The physical connection from VMware Cloud on AWS Outposts to the router or switch of your choice is an important aspect to consider when setting up your hybrid cloud solution. There are several connectivity options available, depending on your bandwidth requirements. The following connections are available. 

1Gbps, up to 8 uplinks	10 Gbps, up to 16 uplinks
Single-Mode  1000Base-LX  Multi-Mode  1000Base-SX	Single-Mode  10GBase-IR and LR  Multi-Mode  10GBase-SR
40 Gbps, up to 4 uplinks	100 Gbps, up to 4 uplinks
Single-Mode  40GBase-IR4 (LR4L)  40GBase-LR4  Multi-Mode  40GBase-SR  40GBase-ESR	Single-Mode  100GBase-CWDM4  100GBase-LR4  100G PSM4 MSA  Multi-Mode  100GBase-SR4

It is important to choose the right connectivity option based on your needs. 

VMware Cloud on AWS Outposts leverages the Link Aggregation Control Protocol (LACP) to establish two Link Aggregation Group (LAG) connections, one from each Outpost network device to each local network device. This allows for multiple links to be aggregated into a single logical link for improved bandwidth and redundancy. LACP is used with standard fast timers to ensure quick failover in the event of a link failure. It is important to note that LAGs cannot be configured to use slow timers. 

By utilizing LACP and LAGs, VMware Cloud on AWS Outposts provides a reliable and high-performance network connection between your network and VMware Cloud on AWS Outposts. 

Logical Connections

In addition to the physical connection, VMware Cloud on AWS Outposts establishes two logical connections to enable communication between your on-premises infrastructure and VMware Cloud on AWS Outposts. These connections are the Local Gateway and the Service Link. Both logical connections use the same physical connection of the VMware Cloud on AWS Outposts.  

The first logical connection is the Service Link. The Service Link connects the VMware Cloud on AWS Outposts Rack with the AWS Region, used by VMware and AWS, to provide the managed service. The Service Link uses AWS Direct Connect (private or public VIF), or a secure link over the public Internet. Additionally, this is used to provide connectivity to the connected customer VPC which is required for the SDDC deployment. There is a connection between NSX-T and a pre-deployed elastic network interface in the customer VPC. It offers a connection to AWS native services, such as Amazon S3, Amazon DynamoDB, Amazon EC2 and others. 

The second logical connection is the Local Gateway. The Local Gateway provides the connection between the SDDC and your network. You will have full control of which segments will be advertised to VMware Cloud on AWS Outposts. 

Network Layer Connectivity

To establish a BGP session between Outpost network devices and customer local network devices, each Outpost network device requires an IP address on each VLAN. To ensure logical point-to-point connectivity, it is recommended to use a dedicated subnet with a /30 or /31 CIDR for each VLAN. Alternatively, a Link Local address (169.254.0.0/16) can be used for the point-to-point connection.  

There are two paths that must be established: the Service Link path and the Local Gateway path. For the Service Link path, a VLAN subnet with a range of /30 or /31 and an IP address should be specified for the Service Link VLAN on the Outpost network device. Similarly, for the Local Gateway path, a VLAN subnet with a range of /30 or /31 and an IP address should be specified for the Local Gateway VLAN on the Outpost network device. 

The diagram below illustrates the connections between each Outpost network device and customer local network devices for both paths. In this example, there are four VLANs: 

• VLAN A represents the Service Link path that connects Outpost network device 1 with customer local network device 1. 

• VLAN B represents the Local Gateway path that connects Outpost network device 1 with customer local network device 1. 

• VLAN C represents the Service Link path that connects Outpost network device 2 with customer local network device 2. 

• VLAN D represents the Local Gateway path that connects Outpost network device 2 with customer local network device 2. 

By following these recommendations and establishing the required paths, a secure and efficient connection will be established between the Outpost network devices and customer local network devices. 

The following table shows example values for the subnets that connect the Outpost Network Device 1 (denoted as OND in the table below) with the customer local network device 1. 

VLAN	Subnet	Customer Device 1 IP	AWS OND 1 IP
A	10.0.0.0/30	10.0.0.2	10.0.0.1
B	172.16.0.0/30	172.16.0.2	172.16.0.1

The following table shows example values for the subnets that connect the Outpost Network Device 2 with the customer local network device 2. 

VLAN	Subnet	Customer Device 2 IP	AWS OND 2 IP
C	10.0.0.4/30	10.0.0.6	10.0.0.5
D	172.16.0.4/30	172.16.0.6	172.16.0.5

Example Switch configuration:

interface vlan600
 description l2-interface-to-rax-aws-outposts-tor1-local-gateway
 no shutdown
 ip address 10.142.2.13/30

interface vlan601
 description l2-interface-to-rax-aws-outposts-tor1-service-link
 no shutdown
 ip address 10.142.2.5/30

interface ethernet1/1/37:1
 description connection-to-rax-aws-outposts-tor1
 no shutdown
 channel-group 22 mode active
 no switchport
 flowcontrol receive off

interface ethernet1/1/38:1
 description connection-to-rax-aws-outposts-tor1
 no shutdown
 channel-group 22 mode active
 no switchport
 flowcontrol receive off

interface port-channel22
 description port-channel-to-rax-aws-outposts-tor1
 no shutdown
 switchport mode trunk
 switchport access vlan 1
 switchport trunk allowed vlan 600-601

 

Border Gateway Protocol (BGP) Connectivity

The AWS Outpost establishes external BGP peering sessions between each Outpost network device and the local network device for Local Gateway connectivity. This allows the Outpost network devices to communicate with customer local network devices using a private Autonomous System Number (ASN) that the customer assigns during setup. 

Each Outpost network device has a single external BGP peering session to a local network device through its Local Gateway VLAN. The peering session is established using the /30 or /31 IPs that you provided during network connectivity setup, creating a point-to-point connectivity between the Outpost network devices and customer local network devices. 

Service Link requirements: 

• The infrastructure CIDR. This must be a /26 CIDR per rack. 

• You provide the customer local network device 1 Service Link BGP peer IP address. (Network Layer Connectivity) 

• You provide the customer local network device 1 Service Link BGP peer ASN. The valid values are 1-4294967294. 

• You provide the customer local network device 2 Service Link BGP peer IP address. (Network Layer Connectivity) 

• You provide the customer local network device 2 Service Link BGP peer ASN. The valid values are 1-4294967294. 

Local Gateway requirements: 

• AWS provides the Local Gateway BGP ASN. 2-byte (16-bit) or 4-byte (32-bit). The valid values are 64512-65535 or 4200000000-4294967294. 

• (Optional) You provide the customer owned CIDR that gets advertised (public or private, /26 minimum). 

• You provide the customer local network device 1 Local Gateway BGP peer IP address. (Network Layer Connectivity) 

• You provide the customer local network device 1 Local Gateway BGP peer ASN. The valid values are 1-4294967294. 

• You provide the customer local network device 2 Local Gateway BGP peer IP address. (Network Layer Connectivity) 

• You provide the customer local network device 2 Local Gateway BGP peer ASN. The valid values are 1-4294967294 

Example switch configuration:

neighbor 10.142.2.14
  advertisement-interval 1
  advertisement-start 1
  description rax-aws-outposts-tor1-lgw
  fall-over
  remote-as 64601
  timers 3 9
  no shutdown

​

 

Service Link

The service link serves as an essential bridge linking your Outposts to your selected AWS Region (or home Region), enabling efficient management of the Outposts and facilitating the transfer of traffic to and from the AWS Region. This pivotal link operates through a sophisticated network of encrypted VPN connections, ensuring secure and reliable communication channels with the home Region.

The configuration extends the Amazon VPC from the AWS Region to the Outposts. An AWS Direct Connect public virtual interface is the service link connection. The following traffic goes over the service link and the AWS Direct Connect connection:

Software-defined Data Center

Deployment

As soon as the VMware Cloud on AWS Outposts is at its final destination, connected, powered, and validated by AWS, it will be handed over to VMware. After a final verification and validation by VMware, it will appear in the VMware Service portal under Infrastructure as CONNECTED.  

At this stage you are ready to deploy your Software-defined Data Center on the Outposts Infrastructure by simply clicking on ‘DEPLOY SDDC,’ which will start a familiar provisioning process. 

​ 

• Select the associated AWS Region which relates to your VMware Cloud on AWS Outposts Rack 
• Select your Outposts ID 
• Choose the number of Hosts 
• Name your SDDC 
• Connect your AWS account by running the CloudFormation 
• Select a VPC and Subnet*
• Enter the Management Subnet 
• Confirm and deploy SDDC 

 

*IMPORTANT: The selected Subnet has to be in the same AZ as the Outposts is linked

​

Network with NSX-T

VMware Cloud on AWS Outposts is remarkably similar to VMware Cloud on AWS and leverages VMware NSX-T which is a crucial part to connect and secure the environment. VMware Cloud NSX provides features like:  

• Gateway Firewall (Management Gateway and Compute Gateway)  
• Route- and policy-based VPN configuration  
• Distributed Firewall for east-west traffic  
• Distributed IDS/IPS  
• and more  

As soon as the SDDC is deployed and NSX is up and running, it will setup a new BGP-Connection between the local Gateway and NSX-T. This ensures a seamless connectivity to the customer network and will automatically advertise routes from VMware Cloud on AWS Outposts to the customer network. NSX-T is learning a 0/0 route from the Local Gateway. This ensures all traffic will be sent to the customer network. It gives full control over the traffic to the customer. 

Gateway Firewall

In VMware Cloud on AWS Outposts, we have two logical domains – one for “Management Resources” (where the ESXi hosts, vCenter, NSX Manager and NSX Controllers are deployed) and one for “Compute Resources”, where customer workload VMs are deployed. 

First, let's start with the Management Resources which are behind the Management Gateway: The Management Gateway provides secure and controlled access to the management components of a VMware Cloud on AWS Outposts SDDC. It acts as a bridge between the SDDC managed components and other networks, allowing administrators to access vCenter Server and other management components using secure protocols like HTTPS. 

The second default gateway is the Compute Gateway. The Compute Gateway provides network connectivity to the virtual machines (VMs) running in a VMware Cloud on AWS Outposts SDDC. It acts as a gateway between the SDDC and the internet or external networks, allowing virtual machines to access resources outside the SDDC. It is possible to create additional gateways for workloads. 

In summary, the Management Gateway and Compute Gateway are two critical components of a VMware Cloud on AWS SDDC. The Management Gateway provides secure and controlled access to the management components of the SDDC, while the Compute Gateway provides network connectivity to the virtual machines running in the SDDC. Together, these gateways help ensure that VMware Cloud on AWS Outposts customers have a secure, and flexible hybrid cloud solution. 

Distributed Firewall

The Distributed Firewall is a core component of NSX-T and provides a distributed and stateful firewalling capability that is built into the hypervisor kernel. It is designed to be deployed at the virtual machine level and provides granular traffic filtering and security policies that can be enforced across the entire data center. 

The Distributed Firewall works by creating security policies that define the allowed or denied traffic flows between VMs or between VMs and the outside world. These policies are based on a set of rules that specify the traffic source and destination, protocol, port, and other parameters. 

The following are some of the key features and benefits of the Distributed Firewall with NSX-T: 

1. Granular security policies: the Distributed Firewall provides granular security policies that can be applied at the VM level, allowing administrators to create fine-grained rules that define the allowed or denied traffic flows. This enables a Zero Trust security model where access is denied by default, and only authorized traffic flows are allowed. 
2. Distributed and stateful firewalling: the Distributed Firewall is built into the hypervisor kernel and provides a distributed and stateful firewalling capability. This means that firewall rules are enforced at the VM level, and traffic flows are inspected and tracked in real-time. This enables the Distributed Firewall to provide a high-performance and scalable security solution that can handle the traffic volumes of modern data centers. 
3. Automation and Orchestration: the Distributed Firewall can be automated and orchestrated using APIs and NSX-T's policy-driven approach. This enables administrators to create, manage, and enforce security policies across the data center using a single interface and consistent policy language. 
4. Compliance and Auditing: the Distributed Firewall provides a rich set of compliance and auditing capabilities that enable administrators to monitor and report on security policy violations, including detailed traffic flows and events. This enables organizations to demonstrate compliance with regulatory requirements and internal security policies. 

In summary, the Distributed Firewall with NSX-T is a powerful and flexible security solution that provides granular traffic filtering and security policies that can be enforced across the entire data center. It offers a distributed and stateful firewalling capability that is tightly integrated with other NSX-T services and can be automated and orchestrated using APIs and a policy-driven approach. 

Connected VPC

VMware Cloud on AWS Outposts is created within an AWS account and VPC dedicated to your organization and managed by VMware. During the SDDC deployment you connect the SDDC to an AWS account that belongs to you. This VPC is referred to as the customer AWS account. By connecting your VMC SDDC to your customer AWS account, you can take advantage of a wide range of AWS services directly via the Service Link and integrate them seamlessly with your vSphere-based workloads running within the SDDC. This allows you to build hybrid cloud solutions that are flexible, scalable, and easy to manage, all while taking advantage of the best of both worlds: the power and flexibility of VMware, and the vast array of services available on AWS. 

References

For additional resources for VMware Cloud on AWS Outposts, please refer to the Service Description and documentation linked below.

Introduction VMware Cloud on AWS Outposts 

VMware Cloud on AWS Outposts Launchpad

VMware Cloud on AWS Outposts Service Description

Shared Responsibility Model - VMware Cloud on AWS Outposts