Google Cloud Storage with Google Cloud VMware Engine - Part 3

In this third part of our series on storage options for Google Cloud VMware Engine, we will discuss Google Cloud Storage, an object-based storage architecture designed to work with large amounts of unstructured data.

Previous posts in the series:

Google Cloud Storage - Overview

If you are unfamiliar with Object Storage, a good primer to read is “What is Object Storage?” available on Google’s learning site. As mentioned earlier, Google Cloud Storage is object storage, which differs from file storage or block storage mainly because it is accessed using RESTful APIs over HTTP/HTTPS. In addition, object storage uses a construct called “buckets,” in which items are stored individually in a highly scalable and elastic manner and can be accessed by metadata or its unique identifier. Although this storage pool is highly scalable, one of the potential downsides to object storage is that to modify an object, it needs to be rewritten entirely.

Table 1: Main Google Cloud Storage Bucket Options

Option	Choices
Bucket Name	Must be globally unique and follow naming guidelines
Location Type	Multi-region, Dual Region, Single Region
Storage Class	Autoclass, Standard, Nearline, Coldline, Archive
Access control	Prevent Public Access (Enabled/Disabled), Uniform, Fine-grained
Protection	None, Object Versioning OR Retention Policy
Data Encryption	Google Managed encryption key, Customer Managed encryption key (CMEK)

Configuration examples

The main uses for object storage are in developing cloud-native applications, big data analytics, or rich media storage. However, there are other practical uses, so I will share a few.

Archiving server files

In this first configuration example, I will create a new bucket to archive my intranet web server’s Apache DocumentRoot directory. This will be done from one of my web servers running Ubuntu 22.04.2 LTS in my Google Cloud VMware Engine web farm.

Note: If you haven’t already, you will need to install the gcloud CLI, which will allow you to use the gcloud, gsutil, and bq commands.

If this is the first time running the Google Cloud CLI commands from a host machine, you will have to authenticate and set the preferred project name.

dschmitz@web01:~$ gcloud auth login
Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth?response_type=*****big-long-url*****

Enter authorization code: ***********************Paste-the-auth-code-here**************************

You are now logged in as [************@**********.com].
Your current project is [None].  You can change this setting by running:
  $ gcloud config set project PROJECT_ID

dschmitz@web01:~$ gcloud projects list
PROJECT_ID                 NAME                       PROJECT_NUMBER
cibg-tmm-ds                CIBG-TMM-DS                1066123456789

dschmitz@web01:~$ gcloud config set project cibg-tmm-ds
Updated property [core/project].

Once you are logged in and the preferred project is set, you can start creating buckets. Remember that bucket names must be unique and follow the naming guidelines. If you are creating one or more buckets programmatically with a script, you can even use a tool like “uuidgen” to generate unique bucket names for you.

dschmitz@web01:~$ uuidgen
9092f1bb-58c0-4172-8e20-4589866618a2

dschmitz@web01:~$ gsutil mb -b on -l us-central1 -c standard gs://9092f1bb-58c0-4172-8e20-4589866618a2/
Creating gs://9092f1bb-58c0-4172-8e20-4589866618a2/...

Scripting tip: Within shell scripts, you can embed the uuidgen command into the bucket creation command like this:

gsutil mb -b on -l us-central1 -c standard gs://`uuidgen`/

With the bucket created, I can easily archive the host’s web server html directory in case I need it later.

Figure 1: Copying files from an Ubuntu server to a Google Cloud storage bucket

Note: Object storage has a flat namespace, so it doesn’t actually have any concept of directories or folders. In other words, copying an object in the format “foldername/filename.txt” to the bucket just appends the folder to the filename. Because of this, the Google Cloud console and the gsutil will visually represent a file and folder tree to help you keep everything organized.

Archiving old virtual machines or templates

Another thing that object storage can be used for is archiving old virtual machines or templates. Instead of storing that old virtual machine data on primary storage, such as within the vSAN cluster or Compute Engine Instances, sometimes it makes more sense to migrate the VM to a lower-cost Google Cloud Storage bucket. In some cases, and depending on the storage class and location assigned, the costs can be as low as a fraction of a penny per gigabyte.

From my Google Compute Engine management instance, I can export old virtual machines to OVA and subsequently copy them to my bucket.

Figure 2: Copying an exported virtual machine to a Google Cloud storage bucket

Note: If this type of operation is something you might be doing frequently, Google Cloud Storage FUSE allows an admin to mount a bucket to a folder on a Debian or Ubuntu host.

Google Cloud Storage as a backup target

There are countless backup software vendors that can use Google Cloud Storage as a backup target. In this final example, I have installed a free and open-source backup software named Duplicati on one of my Windows servers in my Google Cloud VMware Engine private cloud. When I create a new backup job, the storage type dropdown allows me to specify the bucket I created earlier as the storage target.

Figure 3: Specifying a Google Cloud Storage bucket as a backup target

After the job has been run, you can see the encrypted backup segments Duplicati has uploaded to the bucket.

Figure 4: Viewing the contents of a Google Cloud Storage bucket

Data Protection

To protect the data within each bucket, an admin can assign their desired options for controlling the data lifecycle. In a nutshell, the options either allow you to specify rules for how many versions of each object you would like to keep, or a static retention period.

Figure 5: Protection options for a Google Cloud Storage bucket

Conclusion

Using Google Cloud Storage is a simple and cost-effective way to store data, either for use with cloud-native applications, or even administrative use cases. There are still a few more storage options to explore in future posts, so be sure to check back again!

For the latest technical guidance, blogs, or demo videos for Google Cloud VMware Engine, check out VMware Cloud Tech Zone.

Update: Check out the next post in the series -- NetApp Cloud Volumes Service with Google Cloud VMware Engine - Part 4