AWS Storage Blog

Instant data sharing with Multi-Attach enabled Amazon EBS volumes

Mission-critical applications relying on highly performant and low latency access to data (such as business intelligence platforms) require resilient and highly available (HA) architecture. Implementing active-active and active-standby architectures with shared storage in cloud environments is now possible with introduction of EBS Multi-Attach enabled EBS volumes. Multi-Attach, coupled with Cluster Volume Manager (CVM), offers functionality to further segment shared volumes and create instant volume snapshots of datasets for resiliency in addition to rapid prototyping needs.

In part 1 of this series, I cover getting started with Multi-Attach enabled EBS volumes and CVM. I walk through creating a Veritas disk group and a virtual volume, while using snapshots to instantly share read/write replica datasets across connected instances. This approach enables applications to support stringent recovery point and time objectives, measured in seconds rather than minutes.

Amazon EBS is an easy to use, high-performance block storage service designed for use with Amazon EC2 for both throughput and transaction intensive workloads at any scale. With Multi-Attach enabled on Amazon EBS Provisioned IOPS io1 volume, you can allow a single volume to be concurrently attached to up to 16 AWS Nitro System-based Amazon EC2 instances within the same Availability Zone (AZ). Each attached instance has full read and write permission to the shared volume.

Multi-Attach makes it easier to achieve higher application availability in clustered Linux applications through use of cluster volume manager. In this blog, I focus on CVM, an extension of Veritas InfoScale Volume Manager (VxVM), which enables sharing access to the same logical volume data across multiple compute nodes. Each node recognizes the same VxVM logical volume layout, and more importantly, the same state of all volume resources.

CVM supports performance-enhancing capabilities, such as striping, mirroring, and mirror break-off (snapshot) for off-host backup. You can use standard VxVM commands from one node in the cluster to manage all storage. All other nodes immediately recognize any changes in disk group and volume configuration with no user interaction.

Veritas Technologies LLC is an AWS Partner Network (APN) Advanced Technology Partner with AWS competencies in archiving, disaster recovery (DR), high availability, high performance computing (HPC), and storage. Veritas products are available in AWS Marketplace.

Instant sharing of datasets

Figure 1 - Instant snapshot with Veritas CVM

Figure 1: Instant snapshot with Veritas CVM

CVM enables you to create instant, full copies of existing logical volumes. Once created, such copies are referred to as “CVM volume snapshots,” and can be mounted and used across instances as autonomous volumes. CVM volume snapshots can be used as a backup mechanism. In addition, with Multi-Attach enabled volumes, you can use CVM volume snapshot to create instantly available exact copies of volume-based datasets for reporting, rapid experimentation, and testing purposes.

Migrating clustered HA workloads and minimizing recovery time

Figure 2 - HA scenario with Veritas CVM

Figure 2: HA scenario with Veritas CVM

Shared storage-based failover techniques are common in on-premises enterprise architectures using either storage area network (SAN) SAN or directly attached storage (DAS) models. With Multi-Attach enabled EBS volumes migrating such architecture can be simplified by removing infrastructure-related blockers such as lack of performant shared block storage, and time consuming attach/detach operations. As all virtual volumes are visible to every instance with access to the CVM disk group, you can also consider adapting your high availability strategy to account for failover scenarios. You can do so with shared, Multi-Attach enabled EBS volumes in addition to the shared, clustered file system option using virtual volumes.

Note: If using clustered file system approach, you must consider cluster-aware file system choices in addition to adequate mechanisms to ensure data resiliency and reliability. Attempting to deploy a non-cluster-aware file system will result in a loss of data. 

Solution overview

This solution requires two AWS Nitro System-based instances with Veritas InfoScale deployed to a private subnet in the same Availability Zone as an Amazon Virtual Private Cloud (VPC).

Figure 3 - AWS Nitro System-based instances with Multi-Attach enabled EBS volume and CVM

Figure 3: AWS Nitro System-based instances with Multi-Attach enabled EBS volume and CVM

 

This solution consists of a two instances, deployed to subnets in a Single-AZ VPC. These C5 instances are configured with one Multi-Attach enabled Amazon EBS Provisioned IOPS io1 volume on top of which a shared VxVM disk group named multi_dg was configured. This solution enables you to seamlessly share information about clustered volume manager virtualized storage layout such as disk groups, volumes, and snapshots across all instances within a cluster.

Prerequisites

  • Setting up a configuration with Multi-Attach enabled volumes requires at least two AWS Nitro System-based instances in the same Availability Zone. For this example, we have already created two C5.large instances.
  • Instances must be provisioned with Linux Operating System supported by Veritas InfoScale.
  • Veritas InfoScale (7.4.1) should be installed and configured for clustered volume environment as outlined in configuration guide to manage distributed access and visibility of shared storage

Note: For this solution deployment I used Red Hat Enterprise Linux image (RHEL-7.7_HVM-20190923-x86_64-0-Hourly2-GP2 (ami-078a6a18fb73909b2) and Veritas InfoScale version 7.4.1. 

Solution deployment

Step 1: Create an io1 volume with Multi-Attach enabled

To use Multi-Attach enabled Amazon EBS, you must select appropriate options during creation of the EBS volume.

Complete the following steps for Proof-of-concept (PoC) configuration:

  1. On the Amazon EC2 console, choose ELASTIC BLOCK STORE.
  2. Choose Volumes.
  3. Choose Create Volume.
  4. Select Volume Type as Provisioned IOPS SSD (io1), set Size, requested IOPS to match your requirements.
  5. Select Availability Zone (select the same Availability Zone as one where AWS Nitro System-based instances were launched).
  6. Check off Enable for Multi-Attach, then choose Create Volume.

Figure 4 - Creating an io1 EBS volume with Multi-Attach enabled

Figure 4: Creating an io1 volume with Multi-Attach enabled

 

  1. Attach the volume to both C5 instances.

Step 2: Verify that the volume is available to both C5 instances

You want to make sure that the volume is accessible from your instances.

  1. Log in to the first C5 instance via SSH and execute the following command (substitute disk name with disk name corresponding to your configuration, for example, nvme1).
[ec2-user@test-1 ~]$ sudo lsblk --nodeps -o name,serial|grep nvme1
nvme1n1 vol090c9e1b3bcfb534a
  1. Verify if both instances report the same serial number by logging in to the second instance.
[ec2-user@test-2 ~]$ sudo lsblk --nodeps -o name,serial|grep nvme1
nvme1n1 vol090c9e1b3bcfb534a

Step 3: Initialize the volume under VxVM

After successful verification, the next step focuses on initializing a Multi-Attach enabled EBS volume and allowing VxVM to manage it.

  1. Identify the disk name (for example test-1_nvme1_0).
[ec2-user@test-1 ~]$ sudo vxdisk list 
DEVICE          TYPE            DISK         GROUP        STATUS
nvme0n1      auto:none       -            -            online invalid
test-1_nvme1_0 auto:none       -            -            online invalid
  1. Initialize disk with VxVM.
[ec2-user@test-1 ~]$ sudo vxdisksetup -i test-1_nvme1_0
  1. Verify disk status. Note that once initialized, the disk is automatically available via VxVM. NVMe controller ordering upon reboot has no impact on disk group import.
[ec2-user@test-1 ~]$ sudo vxdisk list 
DEVICE          TYPE            DISK         GROUP        STATUS
nvme0n1      auto:none       -            -            online invalid
test-1_nvme1_0 auto:cdsdisk    -            -            online

Step 4: Create a shared disk group and verify its state on both instances

In the next step, you are creating software defined VxVM constructs, such as a shared disk group, which will be visible to all instances running CVM with Multi-Attach enabled EBS volumes.

  1. Create a disk group and verify its state on the first node (for example, test-1).
[ec2-user@test-1 ~]$ sudo vxdg -s init multi_dg test-1_nvme1_0
 [ec2-user@test-1 ~]$ sudo vxdg list
NAME         STATE           ID
multi_dg     enabled,shared,cds   1589427070.20.test-1
  1. Verify group state on the second node (for example, test-2).
[ec2-user@test-2 ~]$ sudo vxdg list
NAME         STATE           ID
multi_dg     enabled,shared,cds   1589427070.20.test-1

Step 5: Create virtual volumeA (choose volume size 10 GB)

Once a shared disk group is created, you can start provisioning virtual volumes for your data (volumeA).

[ec2-user@test-1 ~]$ sudo vxassist -g multi_dg make volumeA 10g
[ec2-user@test-1 ~]$ sudo vxprint -A
Disk group: multi_dg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg multi_dg     multi_dg     -        -        -        -        -       -

dm test-1_nvme1_0 test-1_nvme1_0 -    209639920 -       -        -       -

v  volumeA      fsgen        ENABLED  20971520 -        ACTIVE   -       -
pl volumeA-01   volumeA      ENABLED  20971520 -        ACTIVE   -       -
sd test-1_nvme1_0-01 volumeA-01 ENABLED 20971520 0      -        -       -

Step 6: Prepare volumeA for use

In order to prepare volumeA for use, you must start by building a Linux file system on our virtual volume. You must also download a picture jpg file and calculate file checksum for future reference.

  1. Create the file system (for example, ext4).

Note: We can use a non-cluster-aware file system, such as ext4, because virtual volumeA will be mounted by a single instance. 

[ec2-user@test-1 ~]$ sudo mkfs.ext4 /dev/vx/dsk/multi_dg/volumeA
  1. Mount it on first EC2 instance (for example, test-1)
[ec2-user@test-1 ~]$ sudo mount /dev/vx/dsk/multi_dg/volumeA /mnt/
  1. Verify successful mount.
[ec2-user@test-1 ~]$ df -h /mnt
Filesystem                    Size  Used Avail Use% Mounted on
/dev/vx/dsk/multi_dg/volumeA  9.8G   37M  9.2G   1% /mnt
  1. Create sample file on volume.
[ec2-user@test-1 ~]$ sudo yum -y install wget
[ec2-user@test-1 ~]$ sudo wget https://tinyurl.com/y439nvse -O /mnt/cutedog.jpg
  1. Calculate and record checksum for sample file.
[ec2-user@test-1 ~]$ sudo md5sum /mnt/cutedog.jpg 
97dbf1fa302fe8100ae92c598a6c8834  /mnt/cutedog.jpg

Step 7: Prepare volumeA instant snapshot

In this step, create an instant volumeA instant snapshot using VxVM and the underlying Multi-Attach enabled EBS volume.

  1. Prepare volumeA for instant snapshot operations, and note volumeA size (20971520).
[ec2-user@test-1 ~]$ sudo vxsnap -g multi_dg -b prepare volumeA
  1. Create the instant snapshot volume of the same size as volumeA and name it.
[ec2-user@test-1 ~]$ sudo vxassist -g multi_dg make volumeA-snapshot 20971520
  1. Prepare snapshot volume.
[ec2-user@test-1 ~]$ sudo vxsnap -g multi_dg -b prepare volumeA-snapshot
  1. Use the vxsnap command to create a snapshot volume of volumeA – note that all operations are instantaneous, and there is no additional delay involved in preparing or creating snapshots.
[ec2-user@test-1 ~]$ sudo vxsnap -g multi_dg -o nofreeze make source=volumeA/snapvol=volumeA-snapshot

Step 8: Verify snapshots

After a successful snapshot operation, volumeA-snapshots can be mounted on second EC2 instance in order to validate test file integrity.

  1. Mount the volumeA-snapshot on the second EC2 instance.
[ec2-user@test-2 ~]$ sudo mount /dev/vx/dsk/multi_dg/volumeA-snapshot /mnt/
  1. Calculate checksum and compare the results with Step 6.5.
[ec2-user@test-2 ~]$ sudo md5sum /mnt/cutedog.jpg 
97dbf1fa302fe8100ae92c598a6c8834  /mnt/cutedog.jpg

Cleaning up

Congratulations! You have reached the last step of Multi-Attached EBS solution deployment with cluster volume manager. To ensure you avoid incurring unwanted future costs, please terminate both EC2 C5 instances in addition to Multi-Attach enabled io1 EBS volume using the Amazon EC2 console.

Conclusion

In this blog, I explained how to use Veritas CVM to share data between active-active or active-passive nodes in a highly available cluster deployed to a single Availability Zone.

Using Multi-Attach with Veritas CVM enables you to instantly create snapshots of your volume-based datasets and share them across multiple instances within the same Availability Zone. Use cases benefiting from this approach range from rapid data base cloning for frequent, mission-critical data-altering operations and experimentation (such as billing, rating and reconciliation), and volume sharing for stateful applications.

You can also use Multi-Attach enabled EBS volumes with cluster volume manager to remove infrastructure obstacles when migrating your workloads from on-premises data centers to Amazon VPC.

To learn more about Amazon EBS, visit the Amazon EBS product page. Happy building! In part 2 of this blog series, I focus on migrating HA cluster with shared storage architecture to AWS Cloud. Thanks for reading this blog post! If you have any comments or questions, please don’t hesitate to leave them in the comments section.

Veritas Technologies LLC – APN Partner spotlight

Veritas Technologies LLC is an AWS Competency Partner. They help enterprises address information management challenges including backup and recovery, business continuity, software-defined storage, and information governance.