亚马逊AWS官方博客

在 AWS EKS 容器部署 Nitro Enclaves 可信计算应用程序

一、概述

随着 Web3.0 技术的兴起,数据隐私又成为热点话题。实现数据隐私保护需要多种技术集成,包括算法层面和硬件层面。算法层面的实现方案层出不穷,并且更新迭代速度较快,本文不多做介绍。而底层硬件层面,目前主流的实现是通过 TEE 可信执行环境来确保隐私数据的安全处理不被任何人窥探。 在传统技术领域需要可信计算的场景很多,集中在金融,医疗,政府等行业,在 Web3 领域同样有广泛应用场景,如 资产托管,MPC 钱包,跨链桥,去中心化存储等等。

AWS Nitro Enclaves 是亚马逊云科技提供的 TEE 解决方案,可帮助客户创建隔离的计算环境,帮助客户减小他们用于处理最敏感数据的应用程序的攻击面,进一步保护和安全地处理高度敏感数据。凭借亚马逊云科技的硬件创新能力,Nitro Enclaves 支持几乎所有基于 Nitro 虚拟化技术的计算实例,可同时兼容 Intel,AMD x86 处理器以及 Graviton arm 架构处理器。

除了可信计算的部分,Kubernetes (K8S) 也是目前主流的容器控管平台。随着 IT 规模的增长,企业内部常常需要一个方便的工具来控管所有的容器。亚马逊云科技提供了托管式 Kubernetes 服务 —  Amazon EKS ,利用该服务轻松地在 AWS 上运行 Kubernetes,而无需安装、操作和维护您自己的 Kubernetes 控制面板或 Worker 节点。

Nitro Enclaves 除了为 EC2 实例提供隔离环境,也支持使用 Amazon EKS 和 Kubernetes 编排 Nitro Enclaves。您现在可以使用 Nitro Enclaves Kubernetes 设备插件开源工具,为 Kubernetes 容器组(pod)提供管理 Enclave 生命周期的功能。

本文将结合以上两个技术方案,介绍如何在亚马逊云上构建和部署基于容器的可信计算应用程序。源代码可参考 Github

二、准备工作

1. AWS CLI version 2

$ aws --version
aws-cli/2.9.20 Python/3.9.11 Linux/5.15.0-1028-aws exe/x86_64.ubuntu.22 prompt/off

参考文档:安装或升级 AWS CLI

2. eksctl

$ eksctl version
0.127.0

安装或升级 eksctl

3. docker

$ docker --version
Docker version 20.10.23, build 7155243

安装 Docker

4. jq(json 处理工具)

$ jq --version
jq-1.6

jq 工具下载

5. kubectl version 1.20 or later

$ kubectl version --short --client
Client Version: "v1.23.15-eks-49d8fe8"

安装或升级 kubectl

三、创建 EKS 集群和节点

在创建过程种需要注意两个关键组件:

1. Launch templates(启动模板)

  • 此启动模板将用于在集群中创建启用了 Nitro Enclaves 功能的 worker 节点
  • 必须确保启用 Nitro Enclaves 功能
  • 必须包含指定的 user data,用于执行一些环境初始化操作

2. Nitro Enclaves Kubernetes device plugin

  • 使 Pod 和容器能够使用 Enclaves CLI 创建和终止 enclave
  • 支持 Amazon EKS 以及 自建的 K8S 节点

以下提供手动创建启用 Nitro Enclaves 功能的 EKS 集群的操作步骤,同时,您也可以选择使用 enclavectl 命令行工具进行集群的创建,跳至 3.2 部分。

3.1 手动创建 EKS 集群和节点

Step 1: 创建 launch template

我们需要在 EKS worker 节点上启用 Nitro Enclaves 功能,因此在模板配置时选择的 EC2 父实例需满足以下条件:

  • 采用 Nitro  hypervisor 虚拟化技术的实例
    • 基于 Intel or AMD 处理器的实例配置不少于 4 vCPUs,但不包含 T3T3a 和 u-*
    • 基于 AWS Graviton 处理器的实例配置不少于 2 vCPUs,但不包含 A1T4gG5gIm4gnIs4gen
  • 运行 Linux 或者 Windows(2012 R2 or later)操作系统,推荐使用 Amazon Linux 2
  • 安装 Nitro Enclaves CLI 工具

相比 SGX,AWS Nitro Enclaves TEE 技术兼容更多的处理器架构(x86,ARM)和型号,这里我们以 AMD 实例 c6a 为例:

注意:不要在 launch template 里指定 Subnet,将会导致 Nodegroup 创建失败。

确保启用 Nitro Enclave 功能

添加以下脚本信息到 User data,用于系统参数配置和 Nitro enclaves cli 工具安装:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -e
readonly NE_ALLOCATOR_SPEC_PATH="/etc/nitro_enclaves/allocator.yaml"
# Node resources that will be allocated for Nitro Enclaves
readonly CPU_COUNT=2
readonly MEMORY_MIB=768

# This step below is needed to install nitro-enclaves-allocator service.
amazon-linux-extras install aws-nitro-enclaves-cli -y

# Update enclave's allocator specification: allocator.yaml
sed -i "s/cpu_count:.*/cpu_count: $CPU_COUNT/g" $NE_ALLOCATOR_SPEC_PATH
sed -i "s/memory_mib:.*/memory_mib: $MEMORY_MIB/g" $NE_ALLOCATOR_SPEC_PATH
# Restart the nitro-enclaves-allocator service to take changes effect.
systemctl restart nitro-enclaves-allocator.service
systemctl enable nitro-enclaves-allocator
# Enable HugePages
echo "vm.nr_hugepages=2048" >> /etc/sysctl.conf
sysctl -w vm.nr_hugepages=2048
echo "NE user data script has finished successfully."
--==MYBOUNDARY==

创建完成后,记录下 Launch template ID,以便下个步骤使用。

Step 2:创建 EKS Cluster 和 nodegroup

推荐用通过 eksctl 命令行工具进行 eks 集群和节点的创建,确保在准备工作中已经安装好 eksctl 工具。

准备集群配置文件,ne_cluster_config.yaml 示例:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ne-eks
  region: ap-northeast-1
  version: "1.23"

managedNodeGroups:
  - name: ng-amd-node
    launchTemplate:
      id: lt-01ffded3acb818087
      version: "1"
    desiredCapacity: 1

通过 eksctl 创建 eks 集群:

$ eksctl create cluster -f ne_cluster_config.yaml

2023-02-03 13:42:11 [ℹ]  eksctl version 0.127.0
2 sequential tasks: { create cluster control plane "ne-eks", wait for control plane to become ready 
}
2023-02-03 13:42:11 [ℹ]  building cluster stack "eksctl-ne-eks-cluster"
2023-02-03 13:42:12 [ℹ]  deploying stack "eksctl-ne-eks-cluster"
2023-02-05 23:04:05 [ℹ]  waiting for CloudFormation stack "eksctl-ne-eks-cluster"
...
2023-02-05 23:06:08 [ℹ]  waiting for the control plane to become ready
2023-02-05 23:06:10 [✔]  saved kubeconfig as "/home/username/.kube/config"
2023-02-05 23:06:10 [ℹ]  no tasks
2023-02-05 23:06:10 [✔]  all EKS cluster resources for "ne-eks" have been created
2023-02-05 23:06:12 [ℹ]  kubectl command should work with "/home/username/.kube/config", try 'kubectl get nodes'
2023-02-05 23:06:12 [✔]  EKS cluster "ne-eks" in "ap-northeast-1" region is ready

如在添加节点时遇到问题,请参考 Troubleshooting 文档解决:

https://docs.aws.amazon.com/zh_cn/eks/latest/userguide/troubleshooting.html#worker-node-fail

Step 3:安装 Nitro Enclaves Kubernetes device plugin

Enclaves Kubernetes device plugin 插件使每个 worker 节点上的 pod 能够访问 Nitro Enclaves 设备驱动程序,该插件作为守护进程部署到 Kubernetes 集群。

$ kubectl apply -f https://raw.githubusercontent.com/aws/aws-nitro-enclaves-k8s-device-plugin/main/aws-nitro-enclaves-k8s-ds.yaml
namespace/nitro-enclaves created
daemonset.apps/aws-nitro-enclaves-k8s-daemonset created

查看集群内的节点,并为每个节点加上 enclaves enable 标签:

$ kubectl get nodes
NAME                                              STATUS   ROLES    AGE     VERSION
ip-192-168-94-3.ap-northeast-1.compute.internal   Ready    <none>   9m38s   v1.23.15-eks-49d8fe8

$ kubectl label node ip-192-168-94-3.ap-northeast-1.compute.internal aws-nitro-enclaves-k8s-dp=enabled
node/ip-192-168-94-3.ap-northeast-1.compute.internal labeled

注意:当 eks 集群内新加入节点,需要对每个新节点添加一遍 enclaves enable 标签。

3.2 通过 enclavectl 创建 EKS 集群和节点

除了手动创建 eks 集群,我们还可以利用 enclavectl 命令行工具简化创建步骤。

Step 1:获取 enclavectl 工具

$ git clone git@github.com:aws/aws-nitro-enclaves-with-k8s.git && cd aws-nitro-enclaves-with-k8s

# 如遇到 git@github.com: Permission denied (publickey).报错,可以通过以下方式clone:
$ git clone https://github.com/aws/aws-nitro-enclaves-with-k8s.git && cd aws-nitro-enclaves-with-k8s
# 设置enclavectl环境变量
cd aws-nitro-enclaves-with-k8s/
$ source env.sh

Step 2:修改集群配置

$ vi settings.json
    #配置示例:
    {
      "region" : "ap-northeast-1",
      "instance_type" : "c6a.xlarge",
      "eks_cluster_name" : "ne-eks",
      "eks_worker_node_name" : "ng-amd",
      "eks_worker_node_capacity": "2",
      "k8s_version" : "1.23",
      "node_enclave_cpu_limit": 2,
      "node_enclave_memory_limit_mib": 768
    }

为 enclavectl 指定群配置文件:

$ enclavectl configure --file settings.json

Output example:

[enclavectl] Setup UUID doesn't exist. Creating one...
[enclavectl] Using setup UUID: ca005cb2-94cd-423c-9856-6955d9855cb0
[enclavectl] Using configuration
{
  "region": "ap-northeast-1",
  "instance_type": "c6a.xlarge",
  "eks_cluster_name": "ne-eks",
  "eks_worker_node_name": "ne-amd",
  "eks_worker_node_capacity": "2",
  "k8s_version": "1.23",
  "node_enclave_cpu_limit": 2,
  "node_enclave_memory_limit_mib": 768
}
[enclavectl] Configuration finished successfully.

注意:如果 settings.json 内容做了更新,需要执行 enclavectl cleanup 后再重新通过 enclavectl configure 指定配置文件。

Step 3:自动化部署集群

通过 enclavectl 工具可以快速创建一个启用了 Nitro Enclaves 功能的 EKS 集群,当执行 enclavectl setup 时会自动执行以下几个步骤:

  1. 生成基本的 ec2 launch template
  2. 根据 json 定义的配置创建 eks 集群
  3. 部署 Nitro Enclaves Kubernetes device plugin 插件

整个创建过程不需要人为介入,需等待 20 分钟左右。

$ enclavectl setup

Output example:

[enclavectl] Running setup...
[enclavectl] Creating launch template with name: lt_aba86e99-db6c-40cd-a1b1-482f92422afa
{
    "LaunchTemplate": {
        "LaunchTemplateId": "lt-03803ab04471f68fa",
        "LaunchTemplateName": "lt_aba86e99-db6c-40cd-a1b1-482f92422afa",
        "CreateTime": "2023-02-12T08:03:08+00:00",
        "CreatedBy": "arn:aws:iam::123456543210:user/wsadmin",
        "DefaultVersionNumber": 1,
        "LatestVersionNumber": 1
    }
}
[enclavectl] Launch template has been created successfully in ap-northeast-1 region.
[enclavectl] ####################################################
[enclavectl] Using cluster_config.yaml to create an EKS cluster.
[enclavectl] Cluster creation will take some time.
[enclavectl] ####################################################

        create managed nodegroup "ng-amd",
    } 
}
2023-02-12 16:03:11 [ℹ]  building cluster stack "eksctl-ne-eks-cluster"
2023-02-12 16:03:12 [ℹ]  deploying stack "eksctl-ne-eks-cluster"
2023-02-12 16:03:42 [ℹ]  waiting for CloudFormation stack "eksctl-ne-eks-cluster"
....
2023-02-12 16:16:18 [ℹ]  building managed nodegroup stack "eksctl-ne-eks-nodegroup-ng-amd"
2023-02-12 16:16:19 [ℹ]  deploying stack "eksctl-ne-eks-nodegroup-ng-amd"
2023-02-12 16:16:19 [ℹ]  waiting for CloudFormation stack "eksctl-ne-eks-nodegroup-ng-amd"
....
2023-02-12 16:20:18 [ℹ]  nodegroup "ng-amd" has 2 node(s)
2023-02-12 16:20:18 [ℹ]  node "ip-192-168-47-53.ap-northeast-1.compute.internal" is ready
2023-02-12 16:20:18 [ℹ]  node "ip-192-168-66-46.ap-northeast-1.compute.internal" is ready
2023-02-12 16:20:20 [ℹ]  kubectl command should work with "/home/user/.kube/config", try 'kubectl get nodes'
2023-02-12 16:20:20 [✔]  EKS cluster "ne-eks" in "ap-northeast-1" region is ready
[enclavectl] Attempt to unlabel node ip-192-168-34-159.ap-northeast-1.compute.internal...
label "aws-nitro-enclaves-k8s-dp" not found.
node/ip-192-168-34-159.ap-northeast-1.compute.internal not labeled
[enclavectl] Enabling Nitro Enclaves K8s device plugin...
namespace/nitro-enclaves created
daemonset.apps/aws-nitro-enclaves-k8s-daemonset created
[enclavectl] Labelling node ip-192-168-34-159.ap-northeast-1.compute.internal...
node/ip-192-168-34-159.ap-northeast-1.compute.internal labeled
[enclavectl] Done.

3.3 注意事项

为 Worker 节点添加 enclaves enable 标签

EKS 集群创建成功后,再次确保为每个节点加上 enclaves enable 标签,否则可能会在部署时遇到 “Insufficient aws.ec2.nitro/nitro_enclaves” 的报错信息:

$ kubectl get nodes
NAME                                              STATUS   ROLES    AGE     VERSION
ip-192-168-94-3.ap-northeast-1.compute.internal   Ready    <none>   9m38s   v1.23.15-eks-49d8fe8

$ kubectl label node ip-192-168-94-3.ap-northeast-1.compute.internal aws-nitro-enclaves-k8s-dp=enabled
node/ip-192-168-94-3.ap-northeast-1.compute.internal labeled

为 Worker 节点开启 hugepage

因为 EKS 的版本在追随 K8S 社区持续更新,如果新的版本在部署时出现错误,常见的错误如下:insufficient hugepages-2Mi,则需要在新版本的工作节点手动开启 hugepage,流程如下:

切换到 EC2 控制台,找到对应的 Worker 节点,点击[Connect]

使用 Instance Connect 登录到节点执行远程 Linux 命令

注意:确保 worker 节点的安全组允许访问 ssh 22 端口。

在命令行中执行以下开启 hugepage 脚本:

# 查看hugepage设置
sudo cat /proc/sys/vm/nr_hugepages

# 开启HugePages
sudo sysctl -w vm.nr_hugepages=2048
sudo echo "vm.nr_hugepages=2048" >> /etc/sysctl.conf

# 重启节点
reboot

# 再次检查hugepage状态及大小
sudo cat /proc/sys/vm/nr_hugepages
sudo grep Huge /proc/meminfo

开启 hugepage 以后,新版本的工作节点也可以正常部署 pod 了。

四、将 enclave 应用部署到 EKS

4.1 构建 enclave 应用容器镜像

此阶段会启一个 docker 容器用于构建目标应用程序(如果它存在于 container 目录中)并将其打包成 docker 镜像。 所有应用程序交付内容,包括 Nitro Enclave EIF,都放在 container/bin/ 文件夹中。

注意:执行 build 前,请检查一下 “container/builder/Dockerfile.builder” 文件指定的基础镜像为: “FROM amazonlinux:2″,否则可能在 build 过程遇到“amazon-linux-extras: command not found”报错。

$ enclavectl build --image hello

Output example:

Sending build context to Docker daemon  41.98kB
Step 1/5 : FROM amazonlinux
latest: Pulling from library/amazonlinux
1e78b99dd1fd: Pull complete 
Digest: sha256:66d22a2b8749283aa0764e6983fbb8bf8c143511cf16c9773598acca2a1cd6d2
Status: Downloaded newer image for amazonlinux:latest
 ---> 92310fbb9f4c
Step 2/5 : RUN amazon-linux-extras install aws-nitro-enclaves-cli && yum install wget git aws-nitro-enclaves-cli-devel -y
 ---> Running in c2ae4adcd422
Complete!
Removing intermediate container 968d660f3c92
 ---> 5e41d70e3155
Step 3/5 : WORKDIR /home
 ---> Running in 199262c60151
Removing intermediate container 199262c60151
 ---> d7b38e6da276
Step 4/5 : COPY builder/run.sh run.sh
 ---> 2ceb99bba307
Step 5/5 : CMD ["/home/run.sh"]
 ---> Running in 14767426930e
Removing intermediate container 14767426930e
 ---> 674fd696f04c
Successfully built 674fd696f04c
Successfully tagged ne-example-builder:latest
[BUILDER] ---- Using config: ----
export MANIFEST_NAME="hello-nitro-enclaves"
export MANIFEST_REPOSITORY="<https://github.com/aws/aws-nitro-enclaves-cli.git>"
export MANIFEST_TAG="v1.2.1"
export MANIFEST_EIF_NAME="hello.eif"
export MANIFEST_EIF_DOCKER_IMAGE_NAME="ne-build-hello-eif"
export MANIFEST_EIF_DOCKER_IMAGE_TAG="1.0"
export MANIFEST_EIF_DOCKER_TARGET=""
export MANIFEST_EIF_DOCKER_FILE_NAME="Dockerfile"
export MANIFEST_EIF_DOCKER_FILE_PATH="examples/x86_64/hello"
export MANIFEST_EIF_DOCKER_BUILD_PATH="examples/x86_64/hello"
[BUILDER] -----------------------
Cloning into 'hello-nitro-enclaves'...
remote: Enumerating objects: 187, done.

Turn off this advice by setting config variable advice.detachedHead to false

Sending build context to Docker daemon   5.12kB
Step 1/4 : FROM busybox
latest: Pulling from library/busybox
205dae5015e7: Pull complete 
Digest: sha256:7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c
Status: Downloaded newer image for busybox:latest
 ---> 66ba00ad3de8
Step 2/4 : ENV HELLO="Hello from the enclave side!"
 ---> Running in c63eac39efcf
Removing intermediate container c63eac39efcf
 ---> 7de46b32b704
Step 3/4 : COPY hello.sh /bin/hello.sh
 ---> 9e8be8b6a724
Step 4/4 : CMD ["/bin/hello.sh"]
 ---> Running in e942e79afe3b
Removing intermediate container e942e79afe3b
 ---> dfc1292341c5
Successfully built dfc1292341c5
Successfully tagged ne-build-hello-eif:1.0
Start building the Enclave Image...
Using the locally available Docker image...
Enclave Image successfully created.
{
  "Measurements": {
    "HashAlgorithm": "Sha384 { ... }",
    "PCR0": "d4498f0b0ab406e66cd662e00d15285d795969654dfaae2ab2a65bfd11bf54d0e29543af43085d7ea2b89984626c128f",
    "PCR1": "bcdf05fefccaa8e55bf2c8d6dee9e79bbff31e34bf28a99aa19e6b29c37ee80b214a414b7607236edf26fcb78654e63f",
    "PCR2": "ece8682b2c509be721841e4a7ee52129ff6346cfc526604923734fbf962661135606c74518799ae483eb82077e076bdc"
  }
}
Error: No such image: hello-aba86e99-db6c-40cd-a1b1-482f92422afa:latest
Sending build context to Docker daemon  10.82MB

Step 1/10 : FROM amazonlinux:2 as full_image
Step 2/10 : RUN amazon-linux-extras install aws-nitro-enclaves-cli &&     yum install aws-nitro-enclaves-cli-devel jq -y
 ---> Running in 5c145e15e2b4
Installed:
  aws-nitro-enclaves-cli-devel.x86_64 0:1.2.1-0.amzn2
Complete!
Removing intermediate container 5c145e15e2b4
 ---> 606bfb8daf41
Step 3/10 : WORKDIR /ne-deps
 ---> Running in 50a8914ec6ea
Removing intermediate container 50a8914ec6ea
 ---> 78ddedfe4008
Step 4/10 : RUN BINS="    /usr/bin/nitro-cli     /usr/bin/nitro-enclaves-allocator     /usr/bin/jq     " &&     for bin in $BINS; do         { echo "$bin"; ldd "$bin" | grep -Eo "/.*lib.*/[^ ]+"; } |             while read path; do                 mkdir -p ".$(dirname $path)";                 cp -fL "$path" ".$path";             done     done
 ---> Running in 53df352192a1
Removing intermediate container 53df352192a1
 ---> 981ecc3f2866
Step 5/10 : RUN     mkdir -p /ne-deps/etc/nitro_enclaves &&     mkdir -p /ne-deps/run/nitro_enclaves &&     mkdir -p /ne-deps/var/log/nitro_enclaves &&     cp -rf /usr/share/nitro_enclaves/ /ne-deps/usr/share/ &&     cp -f /etc/nitro_enclaves/allocator.yaml /ne-deps/etc/nitro_enclaves/allocator.yaml
 ---> Running in 0035e108175a
Removing intermediate container 0035e108175a
 ---> 96e20260ed39
Step 6/10 : FROM amazonlinux:2 as image
 ---> 92310fbb9f4c
Step 7/10 : COPY --from=full_image /ne-deps /
 ---> 71c2fa0ff1ac
Step 8/10 : COPY bin/hello.eif /home
 ---> 69cc6c47d179
Step 9/10 : COPY hello/run.sh  /home
 ---> 23f0640164d5
Step 10/10 : CMD ["/home/run.sh"]
 ---> Running in cf7b6710b265
Removing intermediate container cf7b6710b265
 ---> f1bccd652f13
[Warning] One or more build-args [config_region] were not consumed
Successfully built f1bccd652f13

Build 完后,可以看到新生成的镜像:

$ docker images |grep hello

    hello-aba86e99-db6c-40cd-a1b1-482f92422afa   latest    f1bccd652f13   8 minutes ago   244MB
    ne-build-hello-eif                           1.0       dfc1292341c5   9 minutes ago   4.87MB

4.2 将容器镜像推送到仓库

为了方便后续部署应用,建议将刚 build 好的 enclave 镜像放到 ECR 镜像仓库:

$ enclavectl push --image hello

Output example:

[enclavectl] Creating an ECR repository: hello-aba86e99-db6c-40cd-a1b1-482f92422afa.
{
    "repository": {
        "repositoryArn": "arn:aws:ecr:ap-northeast-1:123456543210:repository/hello-aba86e99-db6c-40cd-a1b1-482f92422afa",
        "registryId": "123456543210",
        "repositoryName": "hello-aba86e99-db6c-40cd-a1b1-482f92422afa",
        "repositoryUri": "123456543210.dkr.ecr.ap-northeast-1.amazonaws.com/hello-aba86e99-db6c-40cd-a1b1-482f92422afa",
        "createdAt": "2023-02-12T19:56:36+08:00",
        "imageTagMutability": "MUTABLE",
        "imageScanningConfiguration": {
            "scanOnPush": false
        },
        "encryptionConfiguration": {
            "encryptionType": "AES256"
        }
    }
}
WARNING! Your password will be stored unencrypted in /home/user/.docker/config.json.
Configure a credential helper to remove this warning. See
<https://docs.docker.com/engine/reference/commandline/login/#credentials-store>

Login Succeeded
Using default tag: latest
The push refers to repository [123456543210.dkr.ecr.ap-northeast-1.amazonaws.com/hello-aba86e99-db6c-40cd-a1b1-482f92422afa]
b7b44fe71b7b: Pushed 
d10aee14ebbe: Pushed 
ee98ccdc2ab6: Pushed 
8c5ff259ef50: Pushed 
latest: digest: sha256:53420e277f87baf98e92f4cf5d054c13030f3cf9bcffa1fb14010b9f1a753145 size: 1159

4.3 部署到 EKS 集群运行

为了清晰的了解 Enclaves 应用镜像部署到 EKS 集群过程,我们先将此步分解执行:

Step 1:生成 hello 应用的部署参数(deployment specification):

$ enclavectl run --image hello --prepare-only
    [enclavectl] Generated deployment file: hello_deployment.yaml.

Step 2:打开自动生成的部署配置 .yaml 文件,进行检查和修改:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello-aba86e99-db6c-40cd-a1b1-482f92422afa
        image: 123456543210.dkr.ecr.ap-northeast-1.amazonaws.com/hello-aba86e99-db6c-40cd-a1b1-482f92422afa:latest
        command: ["/home/run.sh"]
        imagePullPolicy: Always
        resources:
          limits:
            aws.ec2.nitro/nitro_enclaves: "1"
            hugepages-2Mi: 768Mi
            cpu: 250m
          requests:
            aws.ec2.nitro/nitro_enclaves: "1"
            hugepages-2Mi: 768Mi
        volumeMounts:
        - mountPath: /hugepages-2Mi
          name: hugepage-2mi
          readOnly: false
        # Enable if 1Gi pages are required
        #- mountPath: /hugepages-1Gi
        #  name: hugepage-1gi
        #  readOnly: false
      volumes:
      - name: hugepage-2mi
        emptyDir:
          medium: HugePages-2Mi
      - name: hugepage-1gi
        emptyDir:
          medium: HugePages-1Gi
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists

Step 3:执行部署

$ kubectl apply -f hello_deployment.yaml
    deployment.apps/hello-deployment created

以上 3 步可以通过 enclavectl  run 命令一步执行:

$ enclavectl run --image hello

[enclavectl] Generated deployment file: hello_deployment.yaml.
deployment.apps/hello-deployment created

检查 hello 应用的部署状态:

$ kubectl get pods --selector app=hello --watch

NAME                               READY   STATUS    RESTARTS   AGE
hello-deployment-6fd6b8c5c-blt4h   0/1     Pending   0          53s
hello-deployment-6fd6b8c5c-blt4h   0/1     Pending   0          31m
hello-deployment-6fd6b8c5c-blt4h   0/1     ContainerCreating   0          46m
hello-deployment-6fd6b8c5c-blt4h   0/1     CrashLoopBackOff    1 (2s ago)   46m
hello-deployment-6fd6b8c5c-blt4h   0/1     CrashLoopBackOff    2 (14s ago)   47m
hello-deployment-6fd6b8c5c-blt4h   1/1     Running             8 (5m4s ago)  48m

待应用运行起来后,查看容器终端日志显示,正常情况将看到“Hello from the enclave side”的信息:

$ kubectl logs hello-deployment-6fd6b8c5c-blt4h

[   1] Hello from the enclave side!
[   2] Hello from the enclave side!
[   3] Hello from the enclave side!
[   4] Hello from the enclave side!
[   5] Hello from the enclave side!
[   6] Hello from the enclave side!
...

这里我们总结一下利用 enclavectl 将 enclave应用部署到 eks 集群的步骤,并快速部署另一个 kmstools 示例应用:

# 1.构建enclave应用镜像
$ enclavectl build --image kms
# 2.推送到镜像仓库 
$ enclavectl push --image kms
# 3.部署到eks集群运行
$ enclavectl run --image kms

通过 kubectl 工具查看部署进度:

$ kubectl get pods --selector app=kms --watch

NAME                              READY   STATUS    RESTARTS   AGE
kms-deployment-845fc478fb-jsb4d   1/1     Running   0          84s

部署成功后可以通过 enclave 运行日志查看应用的执行状态:

$ kubectl logs kms-deployment-845fc478fb-jsb4d

[kms-example] Creating a KMS key...
[kms-example] Encrypting message...
[kms-example] ******************************
[kms-example] KMS Key ARN: arn:aws:kms:ap-northeast-1:123456543210:key/0215b696-4511-44bb-a8ed-a7308ab4549e
[kms-example] Account ID: 123456543210
[kms-example] Unencrypted message: Hello, KMS\\!
[kms-example] Ciphertext: AQICAHiuB3PUOFwhkteOeU+0RVZvz10I/46tEEwc6qWg2cT2ygH7XyeD18YTErRKykQBzNUwAAAAZDBiBgkqhkiG9w0BBwagVTBTAgEAME4GCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMRhRR+VYs5bsh6fHTAgEQgCHnGz2Vx7tjGwcxGrBiEjFJc9wpeEKugjJUynI8b48mzQg=
[kms-example] ******************************
Start allocating memory...
Started enclave with enclave-cid: 16, memory: 128 MiB, cpu-ids: [1, 3]
[kms-example] Requesting from the enclave to decrypt message...
[kms-example] ------------------------
[kms-example] > Got response from the enclave!
[kms-example] Object = { "Status": "Ok" } Object = { "Status": "Ok", "Message": "HelloKMS" }
[kms-example] ------------------------
Successfully terminated enclave i-0213c681fe3361bc8-enc18645cb8ba737e0.

五、环境清理

完成以上部署验证后,为避免持续产生费用,请按照以下步骤停止应用或者删除所有已部署的组件。

5.1 停止应用

可以通过 enclavectl stop 指令删除部署,以停止应用:

$ enclavectl stop --image kms

deployment.apps "kms-deployment" deleted

这个命令会在后台执行  kubectl -f delete kms_deployment.yaml 命令 , 所有通过 enclavectl run 命令初始化的资源都会被一并清除。

5.2 删除集群

利用 enclavectl 提供的 cleanup 指令删除 eks 集群:

$ enclavectl cleanup

[enclavectl] Attempt to delete cluster node group: ng-amd
Error: unable to describe cluster control plane: operation error EKS: DescribeCluster, https response error StatusCode: 404, RequestID: a99865df-2494-45c7-b468-fec23ed75dc1, ResourceNotFoundException: No cluster found for name: ne-eks.
[enclavectl] Cluster node group cannot be deleted.
[enclavectl] Attempt to delete launch template: lt_56c00212-47c9-4650-880f-dae4e3834386
{
    "LaunchTemplate": {
        "LaunchTemplateId": "lt-05976cce6f382b662",
        "LaunchTemplateName": "lt_284ecd59-9a85-4ba9-90f1-242fd3fd380d",
        "CreateTime": "2023-02-12T07:59:56+00:00",
        "CreatedBy": "arn:aws:iam::123456543210:user/wsadmin",
        "DefaultVersionNumber": 1,
        "LatestVersionNumber": 1
    }
}
[enclavectl] Deleting cluster_config.yaml...
[enclavectl] Deleting .config.ne.k8s.ctl...
[enclavectl] Deleting .setup.ne.k8s.ctl...

六、总结

本文介绍了如何手动创建以及借助 enclavectl 工具快速创建启用 Nitro Enclaves 功能的 EKS 集群和节点,并通过示例应用的部署体验完整过程。其中用户可以参考 EKS 集群创建和应用部署方案部分,Enclaves 应用开发及容器镜像的构建还需按照自己实际应用程序进行修改。

本篇作者

倪赛龙

AWS 解决方案架构师,负责基于 AWS 云计算方案的架构咨询和设计实现,同时致力于无服务计算的应用和推广。

肖学嵩

亚马逊云科技初创生态解决方案架构师,15 年 IT 从业经验,现主要负责云计算方案架构的咨询和设计,以技术赋能助力初创企业成长。拥有多年企业级数据中心系统架构设计和实施经验,曾任职于 Sinodata, IBM, Lenovo 等厂商,并先后在私有云和大数据领域有从 0 到 1 的创业经历。

郝志煜

亚马逊云科技解决方案架构师, 他与客户合作构建各类创新解决方案,帮助客户解决业务问题并快速使用 AWS 服务。在工作之余,他还热爱各种极限运动,滑翔伞、极限飞盘和自由潜。