Live Migration of Virtual Machines (VMs) with OpenShift Virtualization on ROSA and Amazon FSx for NetApp ONTAP

We are often faced with innovative technologies that challenge the existing paradigms. OpenShift Virtualization, built on the upstream KubeVirt project, is one such technology.

OpenShift Virtualization allows organizations to run traditional Virtual Machines (VMs) alongside containerized applications in a unified platform. This is particularly valuable for enterprises with legacy applications that need modernization without a complete overhaul. Using Kubernetes as a control plane provides significant benefits, such as scalability, self-healing, and management consistency. For organizations already invested in OpenShift, this integration simplifies operations, as teams can leverage existing tools and workflows. This is also attractive to customers taking advantage of the cloud and migrating workloads to AWS, then further modernizing their business once in the cloud. OpenShift Virtualization provides lift and shift approach to getting VMs into AWS and evolving them using OpenShift and AWS native services.

OpenShift Virtualization is included with OpenShift at no additional licensing or subscription cost. On AWS, it can run on managed OpenShift,  Red Hat OpenShift Service on AWS (ROSA), leveraging its orchestration capabilities to manage VMs alongside containerized applications. This integration allows teams to utilize the same management tools and practices for both VMs and containers, streamlining operations and enhancing efficiency.

The approach of running VMs on a container orchestrator addresses the growing need for flexibility in application deployment. Organizations can choose to run both VMs and containers based on specific workload requirements, without the need to manage separate infrastructure environments.

provides fully managed shared storage on AWS with the popular data access and management capabilities of ONTAP. Amazon FSx requires the addition of storage compute nodes or attached storage within the customer account. The storage scaling and resilience are achieved within a service team account with only an endpoint manifesting in the customer account. This reduces the underlying infrastructure costs, as well as costs related to inter Availability Zone (AZ) data transfer costs to achieve resilience.

Customers with enterprise discount program (EDP) agreements can retire their committed spend with AWS and get support directly from them as well. FSxN can be consumed by applications and VMs running on OpenShift using NetApp Trident as the storage orchestrator.

Solution overview

Below we will look into setting up FSxN as the default storage class for the ROSA cluster, and then create a VM that will leverage FSxN storage for its volumes. We will also look into connecting to the VM using the guest credentials and follow that by performing a live migration of the VM from the current node to a new node.

Prerequisites:

• AWS account
• A Red Hat account
• IAM user with appropriate permissionsto create and access ROSA cluster
• AWS CLI
• ROSA CLI
• OpenShift command-line interface(oc)
• Helm 3 Documentation 
• (with at least 2 bare-metal worker nodes)
• OpenShift Virtualization installed on ROSA Cluster

This diagram shows the ROSA Hosted Control Plane (HCP) cluster deployed in multiple AZs. In a ROSA cluster, the control plane (Master) nodes are in a service team . Some of the worker nodes are metal instance type to support OpenShift Virtualization. The FSxN file system is deployed within the same VPC. The NetApp Trident provisioner is installed in the ROSA cluster, allowing all the subnets of this VPC to connect to the file system. OpenShift Virtualization is installed using the operator from the OpenShift OperatorHub.

ROSA with FSx arch

FSx trident-csi driver setup:

 Ensure default StorageClass is set to trident-csi.

The following yaml was used to create the StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: trident-csi
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-nas"
  fsType: "ext4"
allowVolumeExpansion: True
reclaimPolicy: Retain

New OpenShift StorageClass

Prior to creating the StorageClass, the following yaml files were used to create a secret and the backend object using ontap-nas driver.

apiVersion: v1
kind: Secret
metadata:
  name: backend-fsx-ontap-nas-secret
  namespace: trident
type: Opaque
stringData:
  username: vsadmin
  password: <SVM password configured while creating FSxN>

apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: backend-fsx-ontap-nas
  namespace: trident
spec:
  version: 1
  backendName: fsx-ontap
  storageDriverName: ontap-nas
  managementLIF: <Management DNS name>
  dataLIF: <NFS DNS name>
  svm: <SVM NAME>
  credentials:
    name: backend-fsx-ontap-nas-secret

Ensure default VolumeSnapShotClasses is set as shown

The following yaml was used to create the VolumeSnapshotClass

kind: VolumeSnapshotClass
metadata:
  name: fsx-snapclass
driver: csi.trident.netapp.io
deletionPolicy: Delete

VolumeSnapshotClasses

If the defaults are not set, you can set them up either from the console or from command line

oc patch storageclass trident-csi -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

oc patch VolumeSnapshotClasses fsx-snapclass -p '{"metadata": {"annotations": {"snapshot.storage.kubernetes.io/is-default-class": "true"}}}'

Creating a Virtual Machine from a template in OpenShift Virtualization:

Use the web console to create a VM from a template.

Step 1: From the Red Hat OpenShift Virtualization console, create a VM. There are templates available on the cluster that can be used to create the VM.

create VM from template

Step 2: Select VM Operating System:

We are choosing the Fedora VM template from this list.

VM template selection

Step 3: Configure the VM specifics:

Give the VM a name, and then click on Customize Virtual Machine. Select the Disks tab and click on Add disks. Change the name of the disk, preferably to something meaningful, ensure that trident-csi is selected for storage class. Click on Save. Click on Create VirtualMachine

VM Configuration

Step 4: define the VM attached storage:

VM storage

Step 5: Add Share storage using the FSx StorageClass created earlier:

add FSx storage to VM

After a few minutes, the VM is in the running state

VM status

Exploring attached Storage:

We can now review the storage, first inspecting the disks then the file systems. The file systems of the VM will show the Partitions, File system type and the Mount points.

Disks

File Systems

PVCs are created for the VM, one from the boot disk and one for the hot plug disk.

volume claims

Looking at the Volume Claim details we can see the volume is provided by the Trident-CSI driver and that the volume is a shared or read write many volume.

Volume Claim details

Let us take a moment and explore this from within the OS of the VM.

Step 7: Connect to the VM by clicking on ‘Open web console’ button, and login using Guest Credentials.

Open VM console

Step 8: Next we shall confirm the amount of disk space used and create a test file on the file system.

df

dd if=/dev/urandom of=random.dat bs=1M count=10240

df

df output

Virtual Machine live migration

Live migration refers to the process of moving a running VM from one underlying host to another host without disrupting normal operations or causing any downtime, or other adverse effects for the end user. Live migration is considered a major step in Virtualization. It allows an entire VM to be moved with a running operating system (OS), Memory, storage, and network connectivity from their current node to the destination.

Below we will see how to perform a Live Migration of the VM from the current node to a new node and thanks to the shared storage via EFS, data will be accessible no matter which underlying OpenShift node or which AZ the VM is running on.

Step 9: Click on Migrate under the 3 dot menu

VM migrate

From the overview tab we can see that the migration has been successful.

VM migration status complete

Explore storage post migration

Again we shall connect to the VM from the VirtualMachines tab and run commands to inspect the storage from within the VM OS.

df

ls

inspect storage

Even though the VM is now running on a new node looking at the Storage, File Systems, and Volume Claims will show all these items unchanged.

Conclusion:

Combining OpenShift Virtualization with FSx for NetApp ONTAP creates a powerful solution for organizations seeking to optimize their hybrid environments. OpenShift Virtualization allows users to manage both VMs and containerized applications within a unified Kubernetes platform, enhancing operational efficiency and flexibility. Meanwhile, FSx for NetApp ONTAP provides scalable, high-performance storage that seamlessly integrates with OpenShift, ensuring that data is readily accessible and protected.

This combination empowers businesses to modernize their applications while efficiently managing legacy workloads. With the ability to run diverse workloads in a single environment, teams can streamline their operations, reduce complexity, and improve resource utilization. Together, OpenShift Virtualization and FSx for NetApp ONTAP deliver a robust infrastructure that meets the demands of today’s dynamic enterprise landscape, enabling innovation and growth.

For more information about Trident, refer to the NetApp Trident documentation. See the Red Hat OpenShift Virtualization section in the NetApp Solutions documentation for additional information and videos.