VMware Cloud on AWS integration with Amazon FSx for NetApp ONTAP Deployment Guide

Overview

VMware Cloud on AWS integration with Amazon FSx for NetApp ONTAP is a jointly engineered, AWS-managed external NFS datastore built on NetApp's ONTAP file system that can be attached to VMware Cloud on AWS vSphere cluster. It provides customers with a flexible, high-performance virtualized storage infrastructure that scales independently of compute resources.

Purpose of This Guide

This deployment guide takes you through the steps to provision and attach an Amazon FSx for NetApp ONTAP (FSx for ONTAP) volume as an NFS datastore for VMware Cloud on AWS. In this document, you will also find best practices, supportability requirements, sizing considerations, and other information helping you plan, design, and implement the integration.

Before you provision and attach FSx for ONTAP as an NFS datastore, you must first set up an environment and deploy an SDDC from the VMC Console (vmc.vmware.com). For more information, see the Getting Started With VMware Cloud on AWS.

Audience

This tutorial is intended for Cloud Architects and Cloud Administrators familiar with VMware Cloud on AWS, VMware vSphere, AWS Console, and Amazon FSx for ONTAP.

Design, Architecture, and Supportability

Customer challenges

Inability to scale storage and compute independently increases cost: Migrating and running storage heavy workloads on the cloud is not easy today because attaching a storage option that scales independently of compute resources can be complex and expensive.

VMware Cloud on AWS customers have the following options for additional storage, however, these options come with their own challenges:

Purchase additional i3.metal hosts to scale storage capacity: Customers who do not need the additional compute and memory that comes with the increase in storage capacity will under-utilize these resources, leading to a higher than desired total cost of ownership (TCO).
Convert i3.metal to i3en.metal hosts to get more storage capacity: Customers who have purchased i3.metal instances under 1- or 3-year subscription agreement that is not yet near the end of its commitment term may not be able to do so without incurring unnecessary additional costs.
Purchase external storage from Managed Service Provider: Customers will have to engage 3rd party providers in case of support requirements and increase their vendor footprint.

Use native AWS storage services (EFS, S3, FSx) to extend storage capacity of individual virtual machines (VMs) that are hosted in VMware Cloud on AWS SDDCs: Customers will need to manage the storage configurations at individual VM level, reducing the efficiency with which storage can be managed and scaled.
Need a data store with agile data management capabilities: Customers also need comprehensive and consistent data management capabilities for external storage, such as snapshots, clones, replication, etc. to simplify their data storage and agile data management requirements.
Need a datastore with elastic and flexible capabilities. Customers need the ability to modify the datastore's size, throughput and IOPs dynamically.
Need to learn additional technology and acquiring new skills increase costs, time, and risk: Engaging new vendors, learning innovative solutions and technologies can be a drain on the productivity of IT and operations management personnel a company has. Customers need a storage solution that uses familiar technology with minimal learning curve and less complexity while migrating on-premises storage intensive workloads to the cloud.

VMware Cloud on AWS integration with Amazon FSx for NetApp ONTAP Features

For customers requiring high storage capacity for their workloads (example, Big Data, data warehousing, and VDI), this integration provides an AWS managed, high-performance NFS datastore built on NetApp's ONTAP file system that can be attached to the VMware Cloud on AWS vSphere cluster.
Customers can grow the storage capacity as needed without the need to purchase additional host instances.
This service provides the same features, performance, and administrative capabilities that hundreds of thousands of NetApp customers use on-premises, with the simplicity, agility, security, and scalability of the cloud.
This integration provides familiar NetApp ONTAP's data management capabilities in the cloud, such as space efficient snapshots, cloning, compression, deduplication, compaction, and replication, giving enhanced protection against ransomware.
Scheduled maintenance and backup orchestration are included in the service.
Storage capacity can be added by percentage or absolute value. The minimum storage capacity per file system (i.e., data not tiered to capacity pool storage) is 1024 GB. The maximum storage capacity per file system is 192 TiB uncompressed, however, customers can store more data by using deduplication, compression, and other efficiency mechanisms to reduce the required storage capacity.
Ability to change data throughput, IOPS, and the size of the file system in addition to ability to increase or decrease the size of volumes using the AWS Console or CLI.

Supportability

You cannot deploy FSx for ONTAP file system in the SDDC Connected VPC. Instead, you must deploy it in another new AWS VPC that you own to enable VMware Cloud on AWS integration with Amazon FSx for NetApp ONTAP. You connect the VPC to a VMware Managed Transit Gateway (vTGW) via SDDC groups.
You must deploy FSx for ONTAP within the same AWS Region as your SDDC.
It is recommended to deploy the primary FSX for ONTAP File system in the same AWS Availability Zone (AZ) as the SDDC. For an integration with a single-AZ FSx for ONTAP deploying into the same AZ as the SDDC helps preventing cross-AZ network charges.
VMware Cloud on AWS now supports both multi-AZ and single-AZ deployments of FSx for ONTAP.
VMware Cloud on AWS SDDC must be deployed with standard (single-AZ) clusters. A stretched clusters SDDC currently is not supported.
Access to external datastores is available on SDDC versions 1.20 and greater.
You can attach up to four (4) FSx for ONTAP volumes to a single vSphere cluster on VMware Cloud on AWS.
VMware Cloud Disaster Recovery (VCDR) is supported.
VMware Site Recovery (VSR) is currently not supported.
VMware Cloud on AWS utilizes NFS version 3 to access NFS datastores.

Note: All the items above are subject to change. Updated Jan 2023

High Level Architecture

VMware Cloud on AWS integration with Amazon FSx for NetApp ONTAP augments default vSAN datastore and enables ability to scale storage independent of compute. High level architecture is depicted in Figure 1.

Diagram

Description automatically generated

Amazon FSx for NetApp ONTAP (Multi-AZ) uses an SVM (Storage Virtual Machines) floating IP address that enables failover capability for NAS traffic in case of an AWS Availability Zone-level failure. The SVM floating IP address is outside of the AWS VPC CIDR address space and therefore cannot be routed to the SDDC via the VMware Cloud on AWS SDDC connected VPC ENI connection. Therefore, VMware Transit Connect is required to establish a connection between ESXi hosts and the floating IP address of the NAS interface of FSx for NetApp ONTAP. The traffic flow is depicted in Figure 2.

Diagram, schematic

Description automatically generated

Step-by-Step Deployment Procedure

This step-by-step deployment guide depicts the procedure of adding an Amazon FSx for ONTAP volume to a vSphere cluster on VMware Cloud on AWS. The steps are sequential and build upon one another, so make sure that you complete each step before going to the next step. You can also augment this guide with the step-by-step demo.

Prerequisites

To start, deploy a VMware Cloud on AWS SDDC or use an existing SDDC (SDDC version must be 1.20 or higher). Create a new AWS VPC in the same region and availability zone where SDDC resides. (These steps are not covered in this guide)

High-level deployment steps

Create an Amazon FSx for ONTAP file system in a newly designated VPC using the AWS Console.
Create a SDDC group using the VMware VMC Console.
Configure a TGW attachment to AWS VPC hosting FSx for ONTAP.
Configure routing between AWS VPC and SDDC Group. Configure AWS security groups.
Attach an NFS volume(s) hosted on FSx for ONTAP as a datastore(s) to the SDDC vSphere cluster.

Create Amazon FSx for ONTAP in a designated VPC

To create and mount the Amazon FSx for NetApp ONTAP file system, complete the following steps:

1. Open the Amazon FSx console at https://console.aws.amazon.com/fsx/ and choose to Create file system to start the File System Creation wizard.

2. On the Select File System Type page, select Amazon FSx for NetApp ONTAP and then click Next. The Create File System page appears.

3. For the creation method, choose Standard create.

4. Begin your configuration with the File system details section:

5. For File system name - optional, enter a name for your NetApp ONTAP file system.

6. For Deployment type you can choose between Multi-AZ and Single-AZ:

Multi-AZ file systems replicate your data and support automatic failover across multiple Availability Zones in the same AWS Region.
Single-AZ file systems replicate your data and offer automatic failover within a single Availability Zone.

7. For SSD storage capacity, Specify the SSD storage capacity of your file system: 1024 GiB – 192 TiB. The storage capacity can be increased after you create the file system.

8. For Provisioned SSD IOPS, you have two options to provision the number of IOPS for your file system:

Choose Automatic (the default) if you want Amazon FSx to automatically provision 3 IOPS per GiB of SSD storage.
Choose User-provisioned if you want to specify the number of IOPS. You can increase your provisioned SSD IOPs after the file system is created.

9. For Throughput capacity, you can choose between Recommended throughput capacity and Specify throughput capacity:

Recommended throughput capacity: This will be based on the SSD storage capacity you entered in step 7.
Specify throughput capacity: Choose a value that is your desired throughput capacity in MB per second (MBps).

Note: For more information on the filesystem SSD IOPs and Throughput capacity, see Amazon FSx for NetApp ONTAP performance

Note: The datastores sizes vary quite a bit from customer to customer. Although the recommended number of virtual machines per NFS datastore is subjective, many factors determine the optimum number of VMs that can be placed on each datastore. The amount of concurrent I/O being sent to the VMDKs is one of the most key factors for overall performance. Use performance statistics from on-premises to size the datastore volumes accordingly.

Note: Leverage Live Optics or other APM-based tools to size the datastores throughput and provisioned SSD IOPS. However, if starting with assumption, begin with smaller capacity, IOPs, and throughput and scale up as required as workloads are migrated or deployed.

10. In the Network & Security section for Virtual Private Cloud (VPC), choose the newly created VPC and preferred subnets along with the route table. In this case, Demo-FSxforONTAP-VPC is selected from the dropdown menu.

Note: Make sure this is a newly designated VPC, not the VMware Cloud on AWS SDDC's Connected VPC.

11. Specify a subnet for your file server. If you are creating a Multi-AZ filesystem, choose a Preferred subnet and a Standby subnet. These subnets must be in different AZs. If you are creating a Single-AZ filesystem, you only select a single subnet for the primary fileserver.

Note: (Single-AZ only) It is recommended to choose a subnet hosted within the same AZ as your SDDC to reduce cross AZ network charges.

12. Next, you specify the VPC route tables to be associated with the newly created filesystem:

VPC’s default route table: This is the default option
Select one or more VPC route tables: This option is only available for Multi-AZ

Note: (Multi-AZ only) FSx for ONTAP uses 198.19.0.0/16 as the default endpoint IP address range for the file system. Make sure that the Endpoint IP address range does not conflict with the VMware Cloud on AWS SDDC, associated VPC subnets, and on-premises infrastructure. If you are unsure, use a non-overlapping range with no conflicts.

13. In the Security & Encryption section for the encryption key, choose the AWS Key Management Service (AWS KMS) encryption key that protects the file system's data at rest. For the File System Administrative Password, enter a secure password for the fsxadmin user.

14. In the Default Storage Virtual Machine Configuration section, specify the name of the SVM.

15. In the Default Volume Configuration section, specify the volume name and size required for datastore and click Next. This should be an NFSv3 volume. For Storage Efficiency, choose Enabled to turn on the ONTAP storage efficiency features (compression, deduplication, and compaction). After creation, use the shell to modify the volume parameters using vol modify as follows:

Setting	Configuration
Volume guarantee (Space Guarantee Style)	None (thin provisioned) – set by default
fractional_reserve (fractional-reserve)	0% – set by default
snap_reserve (percent-snapshot-space)	0%
Autosize (autosize-mode)	grow_shrink
Storage efficiency	Enabled – set by default
Autodelete	volume / oldest_first
Volume Tiering Policy	Snapshot only – set by default
try_first	Autogrow
Snapshot policy	None

Use the following SSH command to create and modify volumes:

volume create -vserver FSxONTAPDatastoreSVM -volume DemoDS002 -aggregate aggr1 -size 1024GB -state online -tiering-policy snapshot-only -percent-snapshot-space 0 -autosize-mode grow -snapshot-policy none -junction-path /DemoDS002

Note: The volumes created via shell will take few minutes to show up in the AWS Console.

volume modify -vserver FSxONTAPDatastoreSVM -volume DemoDS002 -fractional-reserve 0
volume modify -vserver FSxONTAPDatastoreSVM -volume DemoDS002 -space-mgmt-try-first vol_grow
volume modify -vserver FSxONTAPDatastoreSVM -volume DemoDS002 -autosize-mode grow

You can learn more about this process in the following document.

Note: During initial migration scenario, the default snapshot policy can cause FSx for ONTAP volume to become full. To overcome it, modify the snapshot policy to suit the needs.

16. Tags are optional labels that you can assign to an AWS resource. Each tag consists of a key and a value. Using tags can help you manage, filter, and search your AWS resources, including your file system.

17. Click Next

18. Review the file system configuration shown on the Create File System page.

19. Click Create File System

Note: Repeat the previous steps to create more storage virtual machines or file systems and the datastore volumes according to the capacity and performance requirements.

Note: Currently, you can attach up to four FSx for ONTAP volumes to a single vSphere cluster on VMware Cloud on AWS.

20. Once all the required SVMs have been created; note down the relevant IP connectivity endpoints required for configuring the VMware Transit Connect and vSphere datastore mounting. You can find the NFS mount endpoints in the Amazon FSx Console by selecting your filesystem, choosing Storage virtual machines, then choosing the SVM where your volume resides.

Create SDDC Group

After the FSx for ONTAP file systems and SVMs have been created, use VMware VMC Console to create an SDDC Group and to configure VMware Transit Connect. To do so, complete the following steps and remember that you must navigate between the VMware Cloud Console and the AWS Console.

1. Log into the VMC Console at https://vmc.vmware.com.

2. On the Inventory page, click SDDC Groups.

3. On the SDDC Groups tab, click ACTIONS and select Create SDDC Group. For demo purposes, the SDDC group is called FSxONTAPDatastoreGrp.

4. On the Membership grid, select the SDDCs to include as group member. You can add a single or multiple SDDCs to the group. All SDDCs in the group will have access to Amazon FSx volumes.

5. Verify that "Configuring VMware Transit Connect for your group will incur charges per attachment and data transfers" is checked, then select Create Group. The process can take a few minutes to complete.

Note: SDDC Groups are an organization-level object. An SDDC Group cannot contain SDDCs from more than one organization. An SDDC Group can include members from up to three AWS regions. You can have a single SDDC as a member of an SDDC Group.

Configure VMware Transit Connect

Attach the newly created designated VPC to the SDDC group. Select the External VPC tab in the VMC Console and follow the instructions for attaching an External VPC to the group. This process can take 10-15 minutes to complete.

Click Add Account.

Provide the AWS account that was used to provision the FSx for ONTAP file system.
Click Add.

Back in the AWS console, log into the same AWS account and navigate to the Resource Access Manager service page. There is a button for you to accept the resource share.

Note: As part of the external VPC attachment process, you will be prompted via the AWS console to a new shared resource via the Resource Access Manager. The shared resource is the AWS Transit Gateway managed by VMware Transit Connect.

Click Accept resource share.

Back in the VMC Console, you now see that the External VPC is in an associated state. This can take several minutes to appear.

Create transit gateway attachment, configure routing and security groups

1. In the AWS Console, go to the VPC service page and navigate to the VPC that was used for provisioning the Amazon FSx for ONTAP file system. Here you create a transit gateway attachment by clicking Transit Gateway Attachment on the navigation pane on the right.

2. Under VPC Attachment, make sure that DNS Support is checked and select the VPC in which FSx for ONTAP was deployed.

3. Click Create transit gateway attachment.

4. Back in the VMC console, navigate to SDDC Group > External VPC tab. Select the AWS account ID used for Amazon FSx for ONTAP and click the VPC and click Accept.

Note: This option may take several minutes to appear.

5. Then in the External VPC tab in the Routes column, click the Add Routes option and add in the required routes:

A route for the NFS IP range for Amazon FSx for ONTAP
A route for the newly created AWS VPC CIDRs

6. In the AWS Console, create the route back to the SDDC by locating the VPC where Amazon FSx for ONTAP is provisioned in the VPC service page and select the main route table for the VPC.

7. Browse to the route table in the lower panel and click Edit routes.

8. In the Edit routes panel, click Add route and enter the CIDR for the SDDC management subnet by selecting Transit Gateway, and the associated TGW ID. Click Save changes.

The next step is to verify that the security group in the associated VPC is updated with the correct inbound rules for the SDDC management subnet CIDR.

9. Update the inbound rule with the CIDR block of the SDDC management. You should allow incoming ICMP and NFS traffic; or all traffic depending on your security requirements.

Note: Verify that the designated VPC (where FSx for ONTAP resides) route table is updated to avoid connectivity issues.

This is the last step in preparing the connectivity from the SDDC to Amazon FSx for ONTAP volumes. With the file system configured, routes added, and security groups updated, it is time to mount the datastores.

Attach NFS volume as a datastore to SDDC cluster

After the file system is provisioned and the connectivity is in place, access the VMC Console to mount NFS datastores.

1. In the VMC Console, open the Storage tab of the SDDC.

2. Click ATTACH DATASTORE and fill in the required values.

3. Click Validate on the right from the NFS Server address If the validation was not successful:

Check your security group inbound rules on the AWS VPC.
When using a multi-AZ FSx for ONTAP deployment, ensure to use the SAME route table for both FSx for NetApp ONTAP VPC subnets.
FSx for NetApp ONTAP should have been deployed in a new AWS VPC and subnnet.

Note: NFS server address is the NFS IP address which can be found under the FSx > Storage virtual machines tab > Endpoints within AWS console.

4. Click ATTACH DATASTORE to attach the datastore to the cluster.

5. Validate the NFS datastore by accessing vCenter Web Client as shown below:

Features, Considerations, and Integrations

Connectivity options - Avoid the myth

VMware Transit Connect enables customers to build high-speed, resilient connections between their VMware Cloud on AWS SDDCs and VPC resources hosting the FSx for ONTAP file system. This capability is enabled via a feature called SDDC Groups. Behind the simplification that SDDC Groups provide is the instantiation of a VMware Managed AWS Transit Gateway, a vTGW. It is automatically deployed when an SDDC group is created. The vTGW provides connectivity between the Amazon FSx for ONTAP NFS volumes and ESXi hosts running in the SDDC.

Data flows from the datastore mounted on the ESXi host to the VMware Transit Connect. From here the vTGW routes the storage traffic to the AWS Transit Connect attachment in the customer-managed VPC where it is finally routed to the SVM floating IP associated with the Active FSx for ONTAP ENI to the FSx for ONTAP filesystem. The data flow is depicted on the reference architecture diagram.

The transit gateway cost is based on the number of attachments made to the Transit Gateway and the amount of egress traffic that flows through AWS Transit Gateway. You can find more information in the following guide.

The table below will help with estimating the data transfer cost for “low IO” workloads. The recommendation is to engage AWS, VMware, or NetApp representatives to help with a detailed projection.

Projected NFS datastore Capacity in TB	R/W Change rate in %*	Egress Capacity (daily)	No. of TGW attachments	TGW single attachment monthly cost in $**	Total TGW data processing monthly cost in $ **	Total TGW data processing and attachments monthly cost in $ **
50TB	35%	17.5TB	2	$36.50	$10,752.00	$10,825
100TB	35%	35TB	2	$36.50	$21,504.00	$21,577
200TB	35%	70TB	2	$36.50	$43,008.00	$43,081

* Change rate Includes both daily churn rate, including updates and the amount of reads from the existing data (covers both reads and writes). Your actual charge rate will vary. Use this as an estimate only and use appropriate tools to identify the daily network transfers to accurately identify the transfer amount.

** We used AWS US East (N Virginia) Region for cost estimates. For other regions, costs might be different.

Note: Prices are actual as of September 2022. Prices vary per AWS Regions and are subject to change.

Another approach to identifying the accurate total data transfer capacity is to leverage vROPs, Live Optics, NetApp Cloud Insights, or similar performance tools to capture the storage network In/Out for virtual machines or at the ESXi host level. Once the data is collected, use the AWS pricing calculator to calculate the egress cost.

Pricing example:

Consider a real-time scenario wherein data egress on VMware Cloud on AWS SDDC is 133MB/s and data egress from FSx for NetApp ONTAP is 140MB/s, then the net data transfer is 273 MB/s. This would equate to 23.5TB of data per day spread across two TGW attachments.

Total data processed per all Transit Gateway attachments: 23,5TB x 30 days = 705TB per month. Let us convert to GB: 705TB x 1024 GB = 721,920 GB per month.

Pricing calculations*

Total Monthly Transit Gateway attachments cost: 730 hours in a month x 0.05 USD* = 36.50 USD x 2 TGW attachments = 73 USD.
Total Monthly Transit Gateway data processing cost: 721,920 GB per month x 0.02 USD = 14,438.4 USD.
Total Transit Gateway monthly cost: 73 USD attachments cost + 14,438.4 USD data processing cost = 14,511.4 USD.

Total cost:

For the scenario outlined above, estimated monthly cost is 14,511.4 USD*.

* We used AWS US East (N Virginia) Region for cost estimates. For other regions, costs might be different.

Performance considerations

It is important to understand that with the NFS version 3 there is only one active pipe for the connection between the ESXi host and a single NFS datastore. This means that although there might be alternate connections available for failover, the bandwidth for a single datastore and the underlying storage are limited to what a single connection can provide.

To leverage more available bandwidth with Amazon FSx for ONTAP volumes you can configure multiple datastores, with each datastore using separate connections between the ESXi hosts and the FSx for ONTAP filesystem.

Note: VMware Cloud on AWS supports up to four NFS datastores per each vSphere cluster in the SDDC.

Performance optimization

Although the recommended number of VM per NFS datastore is subjective, numerous factors determine the optimum number of VMs that can be placed on each datastore. Although most administrators only consider capacity, the amount of concurrent I/O being sent to the VMDKs is one of the most key factors for overall performance. The ESXi host has various mechanisms to ensure fairness between VMs competing for datastore resources. However, the easiest way to control performance is by regulating the number of virtual machines that are placed on each datastore. If the concurrent virtual machine I/O patterns are sending too much traffic to the datastore, the disk queues fill, and higher latency are generated.

Volume and datastore sizing

When a volume is created on Amazon FSx for ONTAP for datastore purposes, the best practice is to create a volume no larger than required. A general recommendation is to begin with a small datastore capacity and increase it as needed. Right sizing datastores prevent accidentally placing too many virtual machines on the datastore and decreases the probability of resource contention. Since datastores and VMDK sizes can be easily increased, if a VM needs extra capacity, it is not necessary to size datastores larger than required. For optimal performance, the best practice is to increase the number of datastores rather than increase their size as an interim measure.

Note: An average production size for an FSx for ONTAP NFS datastore is between 10TB to 20TB.

Note: Place 15-20 VMs on a single datastore. Depending on the VM storage and performance requirements, this can be increased to 35-40 VMs. This guidance will change in the future.

Contact AWS, VMware, and NetApp to plan and size storage and host requirements accurately. We recommend identifying storage performance requirements before finalizing the datastore layout for production deployments.

Increasing the size of the datastore

Resizing the datastore volume is completely transparent to the SDDC. Increase the size of NFS datastores by resizing the volume from the AWS console or by using the FSx for ONTAP CLI. After you are done, access vCenter, go to the datastore tab, right-click the appropriate datastore and select Refresh Capacity Information. This process is also completely transparent to VMs and applications consuming the datastore.

Migration

One of the most common use cases is migration based on various factor. Customers can use VMware HCX or other migrations tools to migrate VMs.

For additional information about migration options and on how to migrate workloads from on-premises to VMware Cloud on AWS, see VMware Cloud TechZone, VMware HCX User Guide, and Migrate workloads to FSx for ONTAP datastore using VMware HCX Guide.

Advanced Amazon FSX for NetApp ONTAP options

Backing up VMs and quickly recovering them are among the great strengths of AWS FSx for ONTAP datastores. Use Snapshot copies to make quick copies of your VMs or even the whole NFS datastore without affecting performance, and then send them to a secondary AWS region of your choice using NetApp SnapMirror replication for disaster recovery purposes. This approach minimizes storage space and network bandwidth by storing changed information only.

Use Amazon FSx for ONTAP snapshot copies for general protection and use application tools like NetApp SnapCenter (for guest-connected storage) or third-party tools to protect virtual machines and transactional data such as Microsoft SQL Server or Oracle residing on the guest VMs.

Note: NetApp SnapCenter Plug-in for VMware vSphere is currently not supported with AWS FSx for ONTAP

ROI Calculations

You can calculate how much you could save by using VMware Cloud on AWS integration with Amazon FSx for NetApp ONTAP. VMware, NetApp, and AWS have built a new ROI tool based on the current VMC Cloud Sizer to help estimate the savings potential. The tool can be accessed here.

The sizer works with manual Inputs as well as with RVtools and considers reserved instance pricing as well.

Here is a quick sizing result which highlights the TCO optimization whilst using FSx for ONTAP as the supplemental datastore.

Conclusion