5G Core implementation on Amazon Elastic Kubernetes Service Anywhere on bare metal

Customers want a simplified approach to enable Telco workloads on Kubernetes on their on-premises customer-managed infrastructure. They want a consistent Kubernetes management and operations experience across their on-premises, edge, and cloud environments. They must adhere to local regulatory and data residency requirements with a disconnected cloud mode support.

In this post, we discuss how customers can utilize Amazon Elastic Kubernetes Service Anywhere (Amazon EKS-A) to automate the deployment of 5G Core on their customer-managed on-premises infrastructure on bare metal. By combining Cloud Native capabilities of Open5GS Core Network Functions (NFs) with Amazon EKS-A on bare metal, we can demonstrate the deployment of Open5GS Core in a few easy steps and a matter of minutes. Furthermore, we demonstrate how customers can integrate additional AWS Services, such as Amazon EKS Connector, Amazon CloudWatch Container Insights, and Amazon CloudWatch to enable centralized visibility and monitor the health of the entire network. Amazon EKS-A also allows customers to enable both Cloud Connected mode and Disconnected Cloud mode. This provides the flexibility to adhere to local regulatory and data residency requirements.

5G Core on AWS

Customers want flexible and multi deployment models for 5G Core based on their use cases and deployment strategy. AWS offers wide range of services to support and meet customer needs and various deployment models. This includes a Cloud-based deployment with all 5G Core NFs and management systems running in an AWS Region. A Telco hybrid cloud deployment with certain NFs, like user plane deployed at the edge or on-premises on AWS Outpost, and rest of the NFs and management running in a Region. There is an on-premises deployment with 5G Core deployed on AWS Outpost, and a disconnected cloud deployment with 5G Core running on Snow family. The introduction of Amazon EKS-A allows customers to support the deployment of 5G Core NFs on customer managed on-premises infrastructure.

Moreover, Amazon EKS-A lets us bring the same Kubernetes advantages to customers’ on-premises infrastructure, as well as support both connected or disconnected cloud deployment models. Amazon EKS-A leverage the same Kubernetes distribution as Amazon Elastic Kubernetes Service (Amazon EKS) that is used on Regions, Local Zones, and Outposts. In addition to providing a stable, secure Kubernetes distribution, Amazon EKS-A bundles additional components to ease the operational burden of customers who require a robust solution to support their on-premises container workloads. The following figure shows summary of these deployment models.

Figure 1 5G Core deployment models on AWS

Open5GS on Amazon EKS-A

In a previous post, we showed the benefits of using AWS as well as the implementation steps for deploying Open5GS Mobile Core on AWS EKS in a Region. In this post, we extend these benefits by implementing Open5GS Mobile Core on customer-managed on-premises Commercial Off The Shelf (COTS) hardware using Amazon EKS-A.

The following figure shows a deployment example of 5G Mobile Core on Amazon EKS-A on a COTS server. Amazon EKS-A lets customers deploy and run Telco Workloads in both connected and disconnected cloud modes as required. The disconnected cloud is required in certain use cases to adhere to regulatory and data residency requirements. In addition, connected cloud mode lets customers use Amazon EKS Connector to connect an on-premises Amazon EKS cluster to the Amazon EKS Cloud Console to monitor and observe the health of EKS clusters. Furthermore, it allows customers a centralized single pane of glass to monitor all of their Amazon EKS clusters, whether they’re running on AWS infrastructure or on-premises customer-managed hardware.

Figure 2 Open5GS Mobile Core deployment on Amazon EKS-A on COTS server

Some assets must be in place before you can create an EKS Anywhere cluster and deploy the 5G Core:

1. COTS Bare Metal Hosts

• • Two hosts, which will be used for Kubernetes cluster with one control-plane node and one Kubernetes node.
  • Each Bare Metal host need two connections: one for IPMI/ILO and one interface which will be used for Kubernetes installation. The IP for Kubernetes installation interface will be provided by DHCP and the MAC address of this interface will be provided to Tinkerbell along with the IPMI credentials using the Tinkerbell hardware specification.

2. Admin Machine

• • This is the machine where the user runs the “eksctl anywhere” commands with the cluster configuration file.

3. Network Setup

• • The admin machine/host should be placed in the same L2 network as all of the other bare metal hosts, which is required for the ability to run a DHCP server.

Once you have access to the admin node, install the required Amazon EKS-A package to configure the Amazon EKS-A cluster and validate the cluster. Refer to and follow the steps as provided in this post. For the hardware, we use HPE ProLiant DL360 and Ubuntu V20.04.5 Operating System to configure the Amazon EKS-A cluster. You can choose any of the published validated hardware.

After you validate that the cluster is setup correctly, you can deploy the Open5GS 5G Core Container Network Functions (CNFs) using the following steps:

1. Enable local storage by running the following commands from the admin server and validate that it was enabled properly:

% kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.22/deploy/local-path-storage.yaml

% kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Validate the SC creation by running below command
% kubectl get sc
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default) rancher.io/local-path Delete         
 WaitForFirstConsumer   false                  23s

2. Add helm repo for Open5GS 5G Core CNFs by running the following command:

% helm repo add openverso https://gradiant.github.io/openverso-charts/

3. Install Open5GS 5G Core CNFs by running the following commands:

% kubectl create ns open5gs

% helm install -n open5gs openverso/open5gs --generate-name
NAME: open5gs-1666622545
LAST DEPLOYED: Mon Oct 24 14:42:26 2022
NAMESPACE: open5gs
STATUS: deployed
REVISION: 1
TEST SUITE: None

4. The process will take approximately 5 mins, you can check the status of the pods of the Open5GS 5G Core CNFs by running the following commands. Make sure that all of the PODs are running and in the 1/1 READY STATE as shown in the following:

% kubectl get pods -n open5gs 
NAME                                           READY   STATUS    RESTARTS       AGE
open5gs-1666622545-amf-6445c8d5db-z6hcz        1/1     Running   0              2m10s
open5gs-1666622545-ausf-764c5699cc-qcz84       1/1     Running   0              2m10s
open5gs-1666622545-bsf-6b5ff44c85-j5jfp        1/1     Running   0              2m10s
open5gs-1666622545-hss-bd8866fc4-sx8kz         1/1     Running   2 (102s ago)   2m9s
open5gs-1666622545-mme-58f7dfc6f9-2vb8x        1/1     Running   0              2m10s
open5gs-1666622545-mongodb-5df45d8985-xd99x    1/1     Running   0              2m9s
open5gs-1666622545-nrf-657f8fb9fd-mw8hz        1/1     Running   0              2m8s
open5gs-1666622545-nssf-5bc46b757c-qvvln       1/1     Running   0              2m9s
open5gs-1666622545-pcf-77f55cb8f8-8v2jt        1/1     Running   2 (101s ago)   2m9s
open5gs-1666622545-pcrf-74cf7b59-hmzjk         1/1     Running   1 (99s ago)    2m8s
open5gs-1666622545-populate-57785786dd-6xp4b   1/1     Running   0              2m10s
open5gs-1666622545-sgwc-846ddc7544-nl8xb       1/1     Running   0              2m9s
open5gs-1666622545-sgwu-5598785747-9n95m       1/1     Running   0              2m9s
open5gs-1666622545-smf-77d9c7846f-2sbfn        1/1     Running   0              2m9s
open5gs-1666622545-udm-7cfb7fdc99-4vr7n        1/1     Running   0              2m8s
open5gs-1666622545-udr-7f58b6774b-wfcc8        1/1     Running   1 (106s ago)   2m8s
open5gs-1666622545-upf-76f55f5988-4hh8f        1/1     Running   0              2m8s
open5gs-1666622545-webui-59c65fbbf5-gn8dc      1/1     Running   0              2m9s

1. Enable the node port for the Web User Interface (UI) service.

1. Get the service name by running the following command:

% kubectl get svc -n open5gs | grep web
open5gs-1666622545-webui         ClusterIP   10.101.10.80     <none>        3000/TCP     2m34s

2. Edit the service to enable NodePort:

% kubectl edit svc -n open5gs open5gs-1666622545-webui
ports:
  - name: http
    port: 3000
    NodePort: 32500	/* Add this line after port:3000*/
    protocol: TCP
    targetPort: http
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/instance: open5gs-1666622545
    app.kubernetes.io/name: webui
  sessionAffinity: None
  type: ClusterIP /* Change the type from ClusterIP to NodePort. After editing the line should show type: NodePort */	

3. Validate that the service is updated with NodePort with the following command:

  % kubectl describe svc -n open5gs open5gs-1666622545-webui
Name:                     open5gs-1666622545-webui
Namespace:                open5gs
Labels:                   app.kubernetes.io/instance=open5gs-1666622545
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=webui
                          helm.sh/chart=webui-2.0.3
Annotations:              meta.helm.sh/release-name: open5gs-1666622545
                          meta.helm.sh/release-namespace: open5gs
Selector:                 app.kubernetes.io/instance=open5gs-1666622545,app.kubernetes.io/name=webui
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.101.10.80
IPs:                      10.101.10.80
Port:                     http  3000/TCP
TargetPort:               http/TCP
NodePort:                 http  32500/TCP
Endpoints:                192.168.0.33:3000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

6. To access the URL, you must first find the node IP address using the following command. This IP address is a configurable IP that is created during the EKS-A deployment using step 1 above:

% kubectl get nodes -o wide
NAME             STATUS   ROLES                  AGE   VERSION                INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
bm-master   Ready    control-plane,master   10d   v1.23.12-eks-a64d4ad   10.80.30.26   <none>        Ubuntu 20.04.5 LTS   5.4.0-128-generic   containerd://1.5.9

7. The highlighted IP above is the host IP. Open any browser and enter the IP address:port 10.80.30.26:32500 to access the Open5GS GUI. For User name, you can use admin and the password is 1423.

Conclusion

Amazon EKS-A provides flexible deployment options for our customers. It leverages the same AWS services and tools that customers are familiar with in the Cloud for their on-premises managed hardware. In this post, we showed how to extend these benefits to the network domain by deploying CNFs on Amazon EKS-A on COTS hardware. By deploying 5G Core CNFs on Amazon EKS-A, customers can now leverage existing hardware investments, enable latency-sensitive network deployment, as well as adhere to various regularities and data residency requirements.

We are excited to see how you can use this new capability and love for you to try and give us your feedback directly on the EKS Anywhere GitHub repo or through your account representatives. If you’d like to try other provisioners, check out the full EKS Anywhere documentation here.