Connecting Equinix ExpressRoute to Microsoft Azure VMware Solution

 

Summary

This article will guide you through the steps to connect Equinix to Microsoft Azure VMware Solution (AVS) over a private connection using ExpressRoute.  Paraphrasing Microsoft’s explanation of ExpressRoute, it is a private connection between the Microsoft Cloud and your network.  It offers the flexibility of dynamic routing, built in redundancy, uptime SLAs, and it does not traverse the public Internet.  Since ExpressRoute is a direct connection into the Microsoft Cloud, it allows the direct consumption of solutions inside Microsoft Azure, allowing you to expand capacity on-demand.  You are now able to take advantage of Microsoft Azure as a hyperscaler and quickly deploy a new AVS cluster to expand your VMware capacity, deploy cloud-native apps, or programmatically scale up & down your deployment, and connect to those deployments over a secure private connection. 

Architecture

This guide focuses specifically on connecting a colocation environment hosted in Equinix to Microsoft Azure and subsequently into AVS.  There are a couple of different types of ExpressRoute connectivity models, but this article will utilize the cloud exchange model connecting through a service provider.  The design integrates components from the colocation environment, leveraging native Equinix Fabric solutions, and builds a private connection directly into Microsoft’s network as if the colocation environment is directly adjacent to the cloud infrastructure.  Azure’s ExpressRoute circuit makes a connection from the colocation environment to a pair of active/active Microsoft Enterprise Edges (MSEE).  The deployment of an AVS private cloud creates dedicated Microsoft Enterprise Edges (D-MSEE) and connects the two different edge types by automatically creating an ExpressRoute to allow connectivity between native Azure solutions and the AVS private cloud.  ExpressRoute Global Reach is then enabled to stitch the two ExpressRoutes together and share routing information.  The diagram below shows a high level of what the end-to-end solution will look like.

Diagram

Description automatically generated

Requirements

This article assumes a fully functional VMware SDDC is deployed in an Equinix datacenter with top of rack switches (ToRs) configured and managed by the customer, and those switches are connected to one or more ports in Equinix Fabric.  Since network configurations vary by vendor, the steps to configure the ToRs is considered out of scope for this article.  On the Microsoft Azure side, an AVS cluster is fully deployed.  If connectivity to native Azure is desired, an Azure Virtual Network (vNet) with the required gateway subnet specifically named “GatewaySubnet” should also be deployed.  Configuring the ExpressRoute connection to the Virtual Network Gateway to allow communication across the ExpressRoute from the colocation environment to native Azure solutions is also considered out of scope for this article.  At a high level, an ExpressRoute circuit needs to be created in Azure and configured to allow communication between the two sites leveraging BGP for dynamic route sharing.  This article will cover the steps to peer Azure to Equinix, but assumes the reader has the knowledge to configure the required components on the Equinix side in the ToRs.  Equinix is one of the cloud exchange providers, thus no third party vendors are needed to create this private connection.

First, gather these requirements:

  • Two /30 private IP subnets
    • transit networks for both Primary & Secondary virtual connections
      • Both are needed during the Azure ExpressRoute creation, even if you’re doing a single non-redundant link in Equinix Fabric
    • These subnets should not overlap any other subnets in the environment
  • Origin VLAN ID
    • For transit network on the colo ToRs
  • Destination VLAN ID
    • For the Azure side
  • BGP ASN for colo ToRs
    • 16 bit and 32 bit ASNs are supported
    • Microsoft reserves 65515 to 65520, do not use those
  • Microsoft Edge router BGP ASN is 12076

Workflow

After the requirements have been gathered, proceed with the following steps.

Step 1 – Create a Resource Group

As a best practice, create a new Resource Group specifically for this ExpressRoute.  By having this level of separation between the different components, it helps ensure actions taken at a Resource Group level for a separate Resource Group doesn’t impact the newly created ExpressRoute.

Log into the Azure portal, navigate to Resource Groups and click Create.  Provide the name and location of this new Resource Group.

Graphical user interface, text, application, email

Description automatically generated

Adding tags is useful if you want to differentiate this ExpressRoute or associate it to a broader use or business unit (prod/dev, EUC/E-Commerce, etc).   Navigate to the newly created Resource Group once provisioning has completed.

Step 2 – Create the ExpressRoute

The newly created Equinix-ExpressRoute Resource Group should be empty.  Click Create Resources, or the Create button at the top left inside the Resource Group, ensuring not to click Create under Resource Groups or a new Resource Group may be created inadvertently.   Clicking Create will take you to the Marketplace, in the search bar, type “ExpressRoute” and look for the icon that looks like a triangle:

Graphical user interface, text, application

Description automatically generated

Click Create, select the new Resource Group we created, the desired region, provide a name, then click Next: Configuration.

Graphical user interface, text, application, email

Description automatically generated

Since this ExpressRoute connection is provided through Equinix, select Provider for the Port type, select Create new, and then select Equinix as the provider.  The peering location should be where the Equinix ports are, and select the bandwidth needed for the deployment.  You can increase bandwidth after provisioning, if needed.  Standard SKU should suffice for the majority of deployments, but if you want to connect more than 10 virtual networks, more than 4000 routes, or need global routing, select the Premium SKU.  The billing model has two options: pay for the amount of data transferred (Metered), or pay a flat rate for unlimited data transfer (Unlimited).  Since this is an all new, no “classic” operations are required.  Tags can be assigned as-needed to associate this ExpressRoute circuit with other resources in your environment.

Graphical user interface, application, email

Description automatically generated

The deployment of the ExpressRoute resource in Azure is  a two part process and must be completed in the Equinix Fabric.  The ExpressRoute provisioning process will take several minutes to complete,  you cannot proceed to the Equinix component until the Azure component is provisioned

Step 3 – Create Equinix Connection

As pointed out in the requirements section, you’ll need ports set up in Equinix Fabric for this next step.  It is assumed that “Standard” port types with Dot1q encapsulation are provisioned in Equinix Fabric.  Log into Equinix Fabric and either navigate to the port(s) and click Create Connection, or click the Connections drop-down menu and select Create Connection.  Locate Microsoft Azure and click “Select Services”.  The list is not alphabetical, so you may have to scroll down to locate it.  Since this is a regular ExpressRoute, select “Azure ExpressRoute” and click Create Connection.  The wizard outlines the steps needed to create the ExpressRoute, starting with the Service Key generated during the previous step of creating the ExpressRoute in Azure (obfuscated from the image below).  To get this Service Key, log into the Azure portal, navigate to the ExpressRoute Resource Group, click the newly created ExpressRoute, and copy the Service Key displayed.

Click “Create a Connection to Azure ExpressRoute” in the Equinix Fabric portal.  In the wizard, select the “Origin Port” that is connected to the VMware SDDC colocation environment.  For redundancy, two ports should be selected, but  only one required.

Graphical user interface

Description automatically generated

Click Port under Primary Origin and drill down to the port  connected to the colocation environment, repeat to create a redundant link, repeat to create a redundant link or select “Create a Single Connection” for Redundant Origin if a secondary connection is not needed.  Enter the Service Key in the field provided and wait for the portal to discover the Azure ExpressRoute.  The Equinix portal will reach over to Azure to pull the information using the Service Key, displaying a spinning circle while retrieving that information.

This should take less than a minute and will switch to a green check mark when done.

It should display the chosen location, connection speed, and provide an estimated monthly charge for the new ExpressRoute connection.  Click Next.

Provide the connection details from the requirements gathered earlier in this article:

Graphical user interface, text, application, email

Description automatically generated

 Provide a connection name, the VLAN ID from our requirements section for “Origin VLAN” should be provided and should be configured on the Equinix ToRs.  This article uses VLAN ID 40, and named the connection “AVS-ExpressRoute”.  Peering type depends on the services consumed on the Microsoft Azure side.   This article is using AVS on a Virtual Network, thus Private peering must be selected.  Microsoft peering would need to be selected if consuming O365, as an example.  Seller C-tag is optional, but this is where you should enter the VLAN ID of the Azure side, and is the “Destination VLAN” from the requirements above.  This article uses VLAN ID 81 for this deployment.  This would be the same VLAN ID that will be entered in the Azure portal while completing the peering. 

In the connection summary before submitting the connection configuration,  optionally add one, or more, email addresses to notify users when the provisioning is complete:

Submit the configuration if all looks well.   The provisioning process typically completes in less than 10 minutesWatch the connection status in the Equinix Fabric portal, Equinix Status will show Provisioned and Provider Status will show to Pending BGP when peering is ready to be configured in the Azure portal to complete the ExpressRoute configuration.

Step 4 – Configuring BGP Peering

This step consists of two parts to establish dynamic routing between Microsoft Azure and the Equinix VMware SDDC:

  • Configure the colo ToRs for BGP with the Azure ExpressRoute as the BGP peer/neighbor
    • The “origin VLAN” will be used here for the /30 network gathered in the requirements section.
    • The local ASN configured on the ToRswill be used in the Azure portal.
  • Configure Peering in the Azure portal to peer with our Equinix colo ToRs.

Configure BGP Neighbor(s) in Equinix colo ToRs

The brand and model of switches deployed will dictate the configuration needed inside the ToRs.  The BGP neighbor IP will be the second IP address of the /30 subnet from our requirements section.  Microsoft assigns the second IP in the range to routers inside Azure, and the ASN is 12076

  • BGP Neighbor IP: second IP of first subnet (primary or single port)
  • ASN: 12076
  • BGP Neighbor IP: second IP of second subnet (optional, only if secondary port configured for redundancy)
  • ASN: 12076

Configure Peering in Azure

Log into the Azure portal and navigate to the newly created ExpressRoute.  Under the section Peerings select Azure Private:

In the Private Peering section, provide the following information:

  • Peer ASN – this is the ASN of the colo ToRs in Equinix
  • Primary /30 subnet from the requirements gathering
    • Reminder: Microsoft uses the second IP in this subnet for it’s router, the first IP must be used for the interface on the ToR interface for this subnet.
  • Secondary /30 subnet from the requirements gathering
    • NOTE: Even if single port/link was chosen in the Equinix Fabric portal when setting up the connection, Microsoft Azure still requires the secondary subnet be provided.  Ensure it does not overlap any subnets in the environment.
  • VLAN ID – This is the Azure side VLAN and needs to match what was entered in Seller C-tag when creating the connection in the Equinix Fabric portal, the “Destination VLAN ID” from the requirements gathering.

Graphical user interface, text, application, email

Description automatically generated

Click Save to start the BGP peering between Equinix & Azure ExpressRoute.  This process will take a few minutes while Microsoft configures it’s virtual routers.  You can verify the BGP connection is established several ways:

  • In the Equinix Fabric portal, navigate to the newly created connection, both Equinix Status and Provider Status will say Provisioned.  The links on either side of the connection in the diagram will also be green.

  • Log into the ToR and verify the BGP neighbor status is ESTABLISHED

Text

Description automatically generated

Now that BGP is established, ping should work from the colo ToR in Equinix to the second IP address in the /30 range that is assigned to Microsoft’s router for ExpressRoute.

Step 5 – Connecting ExpressRoute to AVS

Up to this point in the article, the following items have been deployed:

  • A pre-existing AVS cluster
  • Equinix Fabric Connection connected to an Azure ExpressRoute Circuit

ExpressRoute Global Reach needs to be enabled on the AVS private cloud to bring the two ExpressRoutes together to allow communication and access to the AVS vCenter from the colocation VMware SDDC in Equinix.  A deeper explanation of this can be found under Microsoft Azure VMware Solution Networking in the Additional Resources/References section.

In the Azure portal, generate an authorization key on the newly created ExpressRoute by navigating to the Resource Group and click the ExpressRoute circuit:

On the left side of the screen under Settings, click Authorizations, enter a name for the new authorization key, and click save:

Once this is done, copy the authorization key and navigate to the AVS Private cloud resource.

On the left under Manage, click Connectivity, ExpressRoute Global Reach, and click Add:

The wording in the Azure portal can be confusing, the placement of “or” and the line break may mislead some to think the Authorization key is unnecessary, which is false.  The “or” in this instance is related to selecting the ExpressRoute circuit that’s in the same subscription OR providing the circuit ID of an ExpressRoute in a different subscription.  An authorization key is needed in either case.  Circuit ID is needed when connecting an ExpressRoute circuit from a subscription outside of AVS’s parent subscription in instances where business units may have their own separate Azure subscriptions.

Click Create.  The process takes a few minutes, run a continuous ping from the Equinix VMware SDDC to the AVS vCenter to see when the connection is made.  The ping will begin to succeed before Azure lists the connection is completed.

The Equinix colocation environment is now connected to the Microsoft Azure VMware Solution.

Troubleshooting

Follow these steps if everything is configured but neither the Equinix portal nor the ToRs show BGP is established

  • A very useful troubleshooting tool is viewing ARP records & the route table learned by the Azure private peering in the Azure portal.
    • Navigate to the ExpressRoute circuit and click the three-dot menu on the right of Azure private

  • This is useful in determining if the Azure router is communicating with the Equinix ToR:

  • The route table is a useful function to validate you’re receiving subnet information from the Equinix ToRs, or if you need to set up route filtering
  • Start at the ToR switch and verify the IP address assigned to the primary interface is the first IP in the /30 subnet.  This article used 192.168.40.0/30, thus the IP on the ToR’s interface should be configured as 192.168.40.1/30.
  • Verify ping can reach the IP address that is assigned to the Microsoft router, example: 192.168.4.2
    • If this fails, verify the Origin/Buy VLAN ID and Destination/Seller C-Tag VLAN IDs are properly configured:

Graphical user interface, application, website

Description automatically generated

  • Notice the destination VLAN tag.  This article used VLAN ID 81 for the Seller C-tag, which should match what was supplied when configuring peering in the Azure portal.  Equinix added the “60.” as it translates the VLANs.  An explanation of this can be found in the Equinix videos in the Additional Resources/References section.
  • If this is configured properly and ping still fails, engage Equinix to verify the Fabric port configuration.
  • If ping now succeeds to the second IP address, but BGP will not establish, verify the proper ASNs are supplied
    • This article used 192.168.4.1 for the ToR with an ASN of 64580.
    • Microsoft automatically configured 192.168.4.2 for their router, with an ASN of 12076
    • Log into the ToR switch and validate proper BGP neighbor configuration.
    • Log into the Azure portal, navigate to the ExpressRoute circuit, select Azure private under Peerings, and validate the Peer ASN is the ASN of the Equinix ToR switch.

Lessons Learned

One of the biggest hurdles when setting up the ExpressRoute with Equinix was figuring out the VLANs and how they play together.  The Equinix side is the “A-Side”, and the Microsoft Azure side is the “Z-Side”.  The A-Side is also referred to as the “Buy” side, which this blog refers to as the “Origin VLAN ID”.  When creating the connection in Equinix Fabric, it says “Buyer VLAN ID”.  Equally, the Z-Side may also be referred to as the “Seller” side, which is why it has “Seller C-tag” for the VLAN ID for the Azure side of the connection.  The Z-Side is also where the service profiles live, effectively profile-driven network connections, similar to how NSX-T operates.

Equinix has a pair of videos that walk through how the Equinix Cloud Exchange (ECX) switch handles this, linked below under Additional Resources/References.  The video references AWS, but in this case, it’s synonymous with Azure.

Author

Luke Huckaba is a Solutions Architect on the Partner Solutions Engineering team in VMware’s Office of the CTO.  Luke has over 20 years of experience in IT, more than half of that focusing on VMware solutions.  He’s spent the last several years working with large enterprise customers designing resilient VMware solutions that span multiple clouds and many geographic locations.

 


 

Filter Tags

Azure Services Equinix Services VMware Cloud Azure VMware Solution Document Technical Guide Technical Overview Intermediate Deploy Manage