在 AWS EKS 容器部署 Nitro Enclaves 可信计算应用程序

一、概述

随着 Web3.0 技术的兴起，数据隐私又成为热点话题。实现数据隐私保护需要多种技术集成，包括算法层面和硬件层面。算法层面的实现方案层出不穷，并且更新迭代速度较快，本文不多做介绍。而底层硬件层面，目前主流的实现是通过 TEE 可信执行环境来确保隐私数据的安全处理不被任何人窥探。在传统技术领域需要可信计算的场景很多，集中在金融，医疗，政府等行业，在 Web3 领域同样有广泛应用场景，如资产托管，MPC 钱包，跨链桥，去中心化存储等等。

AWS Nitro Enclaves 是亚马逊云科技提供的 TEE 解决方案，可帮助客户创建隔离的计算环境，帮助客户减小他们用于处理最敏感数据的应用程序的攻击面，进一步保护和安全地处理高度敏感数据。凭借亚马逊云科技的硬件创新能力，Nitro Enclaves 支持几乎所有基于 Nitro 虚拟化技术的计算实例，可同时兼容 Intel，AMD x86 处理器以及 Graviton arm 架构处理器。

除了可信计算的部分，Kubernetes (K8S) 也是目前主流的容器控管平台。随着 IT 规模的增长，企业内部常常需要一个方便的工具来控管所有的容器。亚马逊云科技提供了托管式 Kubernetes 服务 — Amazon EKS ，利用该服务轻松地在 AWS 上运行 Kubernetes，而无需安装、操作和维护您自己的 Kubernetes 控制面板或 Worker 节点。

Nitro Enclaves 除了为 EC2 实例提供隔离环境，也支持使用 Amazon EKS 和 Kubernetes 编排 Nitro Enclaves。您现在可以使用 Nitro Enclaves Kubernetes 设备插件开源工具，为 Kubernetes 容器组（pod）提供管理 Enclave 生命周期的功能。

本文将结合以上两个技术方案，介绍如何在亚马逊云上构建和部署基于容器的可信计算应用程序。源代码可参考 Github。

二、准备工作

1. AWS CLI version 2

$ aws --version
aws-cli/2.9.20 Python/3.9.11 Linux/5.15.0-1028-aws exe/x86_64.ubuntu.22 prompt/off

参考文档：安装或升级 AWS CLI

2. eksctl

$ eksctl version
0.127.0

安装或升级 eksctl

3. docker

$ docker --version
Docker version 20.10.23, build 7155243

安装 Docker

4. jq（json 处理工具）

$ jq --version
jq-1.6

jq 工具下载

5. kubectl version 1.20 or later

$ kubectl version --short --client
Client Version: "v1.23.15-eks-49d8fe8"

安装或升级 kubectl

三、创建 EKS 集群和节点

在创建过程种需要注意两个关键组件：

1. Launch templates（启动模板）

此启动模板将用于在集群中创建启用了 Nitro Enclaves 功能的 worker 节点
必须确保启用 Nitro Enclaves 功能
必须包含指定的 user data，用于执行一些环境初始化操作

2. Nitro Enclaves Kubernetes device plugin

使 Pod 和容器能够使用 Enclaves CLI 创建和终止 enclave
支持 Amazon EKS 以及自建的 K8S 节点

以下提供手动创建启用 Nitro Enclaves 功能的 EKS 集群的操作步骤，同时，您也可以选择使用 enclavectl 命令行工具进行集群的创建，跳至 3.2 部分。

3.1 手动创建 EKS 集群和节点

Step 1: 创建 launch template

我们需要在 EKS worker 节点上启用 Nitro Enclaves 功能，因此在模板配置时选择的 EC2 父实例需满足以下条件：

采用 Nitro hypervisor 虚拟化技术的实例
- 基于 Intel or AMD 处理器的实例配置不少于 4 vCPUs，但不包含 T3， T3a 和 u-*
- 基于 AWS Graviton 处理器的实例配置不少于 2 vCPUs，但不包含 A1， T4g，G5g，Im4gn 和 Is4gen
运行 Linux 或者 Windows（2012 R2 or later）操作系统，推荐使用 Amazon Linux 2
安装 Nitro Enclaves CLI 工具

相比 SGX，AWS Nitro Enclaves TEE 技术兼容更多的处理器架构（x86，ARM）和型号，这里我们以 AMD 实例 c6a 为例：

注意：不要在 launch template 里指定 Subnet，将会导致 Nodegroup 创建失败。

确保启用 Nitro Enclave 功能

添加以下脚本信息到 User data，用于系统参数配置和 Nitro enclaves cli 工具安装：

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -e
readonly NE_ALLOCATOR_SPEC_PATH="/etc/nitro_enclaves/allocator.yaml"
# Node resources that will be allocated for Nitro Enclaves
readonly CPU_COUNT=2
readonly MEMORY_MIB=768

# This step below is needed to install nitro-enclaves-allocator service.
amazon-linux-extras install aws-nitro-enclaves-cli -y

# Update enclave's allocator specification: allocator.yaml
sed -i "s/cpu_count:.*/cpu_count: $CPU_COUNT/g" $NE_ALLOCATOR_SPEC_PATH
sed -i "s/memory_mib:.*/memory_mib: $MEMORY_MIB/g" $NE_ALLOCATOR_SPEC_PATH
# Restart the nitro-enclaves-allocator service to take changes effect.
systemctl restart nitro-enclaves-allocator.service
systemctl enable nitro-enclaves-allocator
# Enable HugePages
echo "vm.nr_hugepages=2048" >> /etc/sysctl.conf
sysctl -w vm.nr_hugepages=2048
echo "NE user data script has finished successfully."
--==MYBOUNDARY==

创建完成后，记录下 Launch template ID，以便下个步骤使用。

Step 2：创建 EKS Cluster 和 nodegroup

推荐用通过 eksctl 命令行工具进行 eks 集群和节点的创建，确保在准备工作中已经安装好 eksctl 工具。

准备集群配置文件，ne_cluster_config.yaml 示例：

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ne-eks
  region: ap-northeast-1
  version: "1.23"

managedNodeGroups:
  - name: ng-amd-node
    launchTemplate:
      id: lt-01ffded3acb818087
      version: "1"
    desiredCapacity: 1

通过 eksctl 创建 eks 集群：

$ eksctl create cluster -f ne_cluster_config.yaml

2023-02-03 13:42:11 [ℹ]  eksctl version 0.127.0
2 sequential tasks: { create cluster control plane "ne-eks", wait for control plane to become ready 
}
2023-02-03 13:42:11 [ℹ]  building cluster stack "eksctl-ne-eks-cluster"
2023-02-03 13:42:12 [ℹ]  deploying stack "eksctl-ne-eks-cluster"
2023-02-05 23:04:05 [ℹ]  waiting for CloudFormation stack "eksctl-ne-eks-cluster"
...
2023-02-05 23:06:08 [ℹ]  waiting for the control plane to become ready
2023-02-05 23:06:10 [✔]  saved kubeconfig as "/home/username/.kube/config"
2023-02-05 23:06:10 [ℹ]  no tasks
2023-02-05 23:06:10 [✔]  all EKS cluster resources for "ne-eks" have been created
2023-02-05 23:06:12 [ℹ]  kubectl command should work with "/home/username/.kube/config", try 'kubectl get nodes'
2023-02-05 23:06:12 [✔]  EKS cluster "ne-eks" in "ap-northeast-1" region is ready

如在添加节点时遇到问题，请参考 Troubleshooting 文档解决：

https://docs.aws.amazon.com/zh_cn/eks/latest/userguide/troubleshooting.html#worker-node-fail

Step 3：安装 Nitro Enclaves Kubernetes device plugin

Enclaves Kubernetes device plugin 插件使每个 worker 节点上的 pod 能够访问 Nitro Enclaves 设备驱动程序，该插件作为守护进程部署到 Kubernetes 集群。

$ kubectl apply -f https://raw.githubusercontent.com/aws/aws-nitro-enclaves-k8s-device-plugin/main/aws-nitro-enclaves-k8s-ds.yaml
namespace/nitro-enclaves created
daemonset.apps/aws-nitro-enclaves-k8s-daemonset created

查看集群内的节点，并为每个节点加上 enclaves enable 标签：

$ kubectl get nodes
NAME                                              STATUS   ROLES    AGE     VERSION
ip-192-168-94-3.ap-northeast-1.compute.internal   Ready    <none>   9m38s   v1.23.15-eks-49d8fe8

$ kubectl label node ip-192-168-94-3.ap-northeast-1.compute.internal aws-nitro-enclaves-k8s-dp=enabled
node/ip-192-168-94-3.ap-northeast-1.compute.internal labeled

注意：当 eks 集群内新加入节点，需要对每个新节点添加一遍 enclaves enable 标签。

3.2 通过 enclavectl 创建 EKS 集群和节点

除了手动创建 eks 集群，我们还可以利用 enclavectl 命令行工具简化创建步骤。

Step 1：获取 enclavectl 工具

$ git clone git@github.com:aws/aws-nitro-enclaves-with-k8s.git && cd aws-nitro-enclaves-with-k8s

# 如遇到 git@github.com: Permission denied (publickey).报错，可以通过以下方式clone：
$ git clone https://github.com/aws/aws-nitro-enclaves-with-k8s.git && cd aws-nitro-enclaves-with-k8s
# 设置enclavectl环境变量
cd aws-nitro-enclaves-with-k8s/
$ source env.sh

Step 2：修改集群配置

$ vi settings.json
    #配置示例：
    {
      "region" : "ap-northeast-1",
      "instance_type" : "c6a.xlarge",
      "eks_cluster_name" : "ne-eks",
      "eks_worker_node_name" : "ng-amd",
      "eks_worker_node_capacity": "2",
      "k8s_version" : "1.23",
      "node_enclave_cpu_limit": 2,
      "node_enclave_memory_limit_mib": 768
    }

为 enclavectl 指定群配置文件：

$ enclavectl configure --file settings.json

Output example:

[enclavectl] Setup UUID doesn't exist. Creating one...
[enclavectl] Using setup UUID: ca005cb2-94cd-423c-9856-6955d9855cb0
[enclavectl] Using configuration
{
  "region": "ap-northeast-1",
  "instance_type": "c6a.xlarge",
  "eks_cluster_name": "ne-eks",
  "eks_worker_node_name": "ne-amd",
  "eks_worker_node_capacity": "2",
  "k8s_version": "1.23",
  "node_enclave_cpu_limit": 2,
  "node_enclave_memory_limit_mib": 768
}
[enclavectl] Configuration finished successfully.

注意：如果 settings.json 内容做了更新，需要执行 enclavectl cleanup 后再重新通过 enclavectl configure 指定配置文件。

Step 3：自动化部署集群

通过 enclavectl 工具可以快速创建一个启用了 Nitro Enclaves 功能的 EKS 集群，当执行 enclavectl setup 时会自动执行以下几个步骤：

生成基本的 ec2 launch template
根据 json 定义的配置创建 eks 集群
部署 Nitro Enclaves Kubernetes device plugin 插件

整个创建过程不需要人为介入，需等待 20 分钟左右。

$ enclavectl setup

Output example:

[enclavectl] Running setup...
[enclavectl] Creating launch template with name: lt_aba86e99-db6c-40cd-a1b1-482f92422afa
{
    "LaunchTemplate": {
        "LaunchTemplateId": "lt-03803ab04471f68fa",
        "LaunchTemplateName": "lt_aba86e99-db6c-40cd-a1b1-482f92422afa",
        "CreateTime": "2023-02-12T08:03:08+00:00",
        "CreatedBy": "arn:aws:iam::123456543210:user/wsadmin",
        "DefaultVersionNumber": 1,
        "LatestVersionNumber": 1
    }
}
[enclavectl] Launch template has been created successfully in ap-northeast-1 region.
[enclavectl] ####################################################
[enclavectl] Using cluster_config.yaml to create an EKS cluster.
[enclavectl] Cluster creation will take some time.
[enclavectl] ####################################################

        create managed nodegroup "ng-amd",
    } 
}
2023-02-12 16:03:11 [ℹ]  building cluster stack "eksctl-ne-eks-cluster"
2023-02-12 16:03:12 [ℹ]  deploying stack "eksctl-ne-eks-cluster"
2023-02-12 16:03:42 [ℹ]  waiting for CloudFormation stack "eksctl-ne-eks-cluster"
....
2023-02-12 16:16:18 [ℹ]  building managed nodegroup stack "eksctl-ne-eks-nodegroup-ng-amd"
2023-02-12 16:16:19 [ℹ]  deploying stack "eksctl-ne-eks-nodegroup-ng-amd"
2023-02-12 16:16:19 [ℹ]  waiting for CloudFormation stack "eksctl-ne-eks-nodegroup-ng-amd"
....
2023-02-12 16:20:18 [ℹ]  nodegroup "ng-amd" has 2 node(s)
2023-02-12 16:20:18 [ℹ]  node "ip-192-168-47-53.ap-northeast-1.compute.internal" is ready
2023-02-12 16:20:18 [ℹ]  node "ip-192-168-66-46.ap-northeast-1.compute.internal" is ready
2023-02-12 16:20:20 [ℹ]  kubectl command should work with "/home/user/.kube/config", try 'kubectl get nodes'
2023-02-12 16:20:20 [✔]  EKS cluster "ne-eks" in "ap-northeast-1" region is ready
[enclavectl] Attempt to unlabel node ip-192-168-34-159.ap-northeast-1.compute.internal...
label "aws-nitro-enclaves-k8s-dp" not found.
node/ip-192-168-34-159.ap-northeast-1.compute.internal not labeled
[enclavectl] Enabling Nitro Enclaves K8s device plugin...
namespace/nitro-enclaves created
daemonset.apps/aws-nitro-enclaves-k8s-daemonset created
[enclavectl] Labelling node ip-192-168-34-159.ap-northeast-1.compute.internal...
node/ip-192-168-34-159.ap-northeast-1.compute.internal labeled
[enclavectl] Done.

3.3 注意事项

为 Worker 节点添加 enclaves enable 标签

EKS 集群创建成功后，再次确保为每个节点加上 enclaves enable 标签，否则可能会在部署时遇到 “Insufficient aws.ec2.nitro/nitro_enclaves” 的报错信息：

$ kubectl get nodes
NAME                                              STATUS   ROLES    AGE     VERSION
ip-192-168-94-3.ap-northeast-1.compute.internal   Ready    <none>   9m38s   v1.23.15-eks-49d8fe8

$ kubectl label node ip-192-168-94-3.ap-northeast-1.compute.internal aws-nitro-enclaves-k8s-dp=enabled
node/ip-192-168-94-3.ap-northeast-1.compute.internal labeled

为 Worker 节点开启 hugepage

因为 EKS 的版本在追随 K8S 社区持续更新，如果新的版本在部署时出现错误，常见的错误如下：insufficient hugepages-2Mi，则需要在新版本的工作节点手动开启 hugepage，流程如下：

切换到 EC2 控制台，找到对应的 Worker 节点，点击[Connect]

使用 Instance Connect 登录到节点执行远程 Linux 命令

注意：确保 worker 节点的安全组允许访问 ssh 22 端口。

在命令行中执行以下开启 hugepage 脚本：

# 查看hugepage设置
sudo cat /proc/sys/vm/nr_hugepages

# 开启HugePages
sudo sysctl -w vm.nr_hugepages=2048
sudo echo "vm.nr_hugepages=2048" >> /etc/sysctl.conf

# 重启节点
reboot

# 再次检查hugepage状态及大小
sudo cat /proc/sys/vm/nr_hugepages
sudo grep Huge /proc/meminfo

开启 hugepage 以后，新版本的工作节点也可以正常部署 pod 了。

四、将 enclave 应用部署到 EKS

4.1 构建 enclave 应用容器镜像

此阶段会启一个 docker 容器用于构建目标应用程序（如果它存在于 container 目录中）并将其打包成 docker 镜像。所有应用程序交付内容，包括 Nitro Enclave EIF，都放在 container/bin/ 文件夹中。

注意：执行 build 前，请检查一下 “container/builder/Dockerfile.builder” 文件指定的基础镜像为: “FROM amazonlinux:2″，否则可能在 build 过程遇到“amazon-linux-extras: command not found”报错。

$ enclavectl build --image hello

Output example:

Sending build context to Docker daemon  41.98kB
Step 1/5 : FROM amazonlinux
latest: Pulling from library/amazonlinux
1e78b99dd1fd: Pull complete 
Digest: sha256:66d22a2b8749283aa0764e6983fbb8bf8c143511cf16c9773598acca2a1cd6d2
Status: Downloaded newer image for amazonlinux:latest
 ---> 92310fbb9f4c
Step 2/5 : RUN amazon-linux-extras install aws-nitro-enclaves-cli && yum install wget git aws-nitro-enclaves-cli-devel -y
 ---> Running in c2ae4adcd422
Complete!
Removing intermediate container 968d660f3c92
 ---> 5e41d70e3155
Step 3/5 : WORKDIR /home
 ---> Running in 199262c60151
Removing intermediate container 199262c60151
 ---> d7b38e6da276
Step 4/5 : COPY builder/run.sh run.sh
 ---> 2ceb99bba307
Step 5/5 : CMD ["/home/run.sh"]
 ---> Running in 14767426930e
Removing intermediate container 14767426930e
 ---> 674fd696f04c
Successfully built 674fd696f04c
Successfully tagged ne-example-builder:latest
[BUILDER] ---- Using config: ----
export MANIFEST_NAME="hello-nitro-enclaves"
export MANIFEST_REPOSITORY="<https://github.com/aws/aws-nitro-enclaves-cli.git>"
export MANIFEST_TAG="v1.2.1"
export MANIFEST_EIF_NAME="hello.eif"
export MANIFEST_EIF_DOCKER_IMAGE_NAME="ne-build-hello-eif"
export MANIFEST_EIF_DOCKER_IMAGE_TAG="1.0"
export MANIFEST_EIF_DOCKER_TARGET=""
export MANIFEST_EIF_DOCKER_FILE_NAME="Dockerfile"
export MANIFEST_EIF_DOCKER_FILE_PATH="examples/x86_64/hello"
export MANIFEST_EIF_DOCKER_BUILD_PATH="examples/x86_64/hello"
[BUILDER] -----------------------
Cloning into 'hello-nitro-enclaves'...
remote: Enumerating objects: 187, done.

Turn off this advice by setting config variable advice.detachedHead to false

Sending build context to Docker daemon   5.12kB
Step 1/4 : FROM busybox
latest: Pulling from library/busybox
205dae5015e7: Pull complete 
Digest: sha256:7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c
Status: Downloaded newer image for busybox:latest
 ---> 66ba00ad3de8
Step 2/4 : ENV HELLO="Hello from the enclave side!"
 ---> Running in c63eac39efcf
Removing intermediate container c63eac39efcf
 ---> 7de46b32b704
Step 3/4 : COPY hello.sh /bin/hello.sh
 ---> 9e8be8b6a724
Step 4/4 : CMD ["/bin/hello.sh"]
 ---> Running in e942e79afe3b
Removing intermediate container e942e79afe3b
 ---> dfc1292341c5
Successfully built dfc1292341c5
Successfully tagged ne-build-hello-eif:1.0
Start building the Enclave Image...
Using the locally available Docker image...
Enclave Image successfully created.
{
  "Measurements": {
    "HashAlgorithm": "Sha384 { ... }",
    "PCR0": "d4498f0b0ab406e66cd662e00d15285d795969654dfaae2ab2a65bfd11bf54d0e29543af43085d7ea2b89984626c128f",
    "PCR1": "bcdf05fefccaa8e55bf2c8d6dee9e79bbff31e34bf28a99aa19e6b29c37ee80b214a414b7607236edf26fcb78654e63f",
    "PCR2": "ece8682b2c509be721841e4a7ee52129ff6346cfc526604923734fbf962661135606c74518799ae483eb82077e076bdc"
  }
}
Error: No such image: hello-aba86e99-db6c-40cd-a1b1-482f92422afa:latest
Sending build context to Docker daemon  10.82MB

Step 1/10 : FROM amazonlinux:2 as full_image
Step 2/10 : RUN amazon-linux-extras install aws-nitro-enclaves-cli &&     yum install aws-nitro-enclaves-cli-devel jq -y
 ---> Running in 5c145e15e2b4
Installed:
  aws-nitro-enclaves-cli-devel.x86_64 0:1.2.1-0.amzn2
Complete!
Removing intermediate container 5c145e15e2b4
 ---> 606bfb8daf41
Step 3/10 : WORKDIR /ne-deps
 ---> Running in 50a8914ec6ea
Removing intermediate container 50a8914ec6ea
 ---> 78ddedfe4008
Step 4/10 : RUN BINS="    /usr/bin/nitro-cli     /usr/bin/nitro-enclaves-allocator     /usr/bin/jq     " &&     for bin in $BINS; do         { echo "$bin"; ldd "$bin" | grep -Eo "/.*lib.*/[^ ]+"; } |             while read path; do                 mkdir -p ".$(dirname $path)";                 cp -fL "$path" ".$path";             done     done
 ---> Running in 53df352192a1
Removing intermediate container 53df352192a1
 ---> 981ecc3f2866
Step 5/10 : RUN     mkdir -p /ne-deps/etc/nitro_enclaves &&     mkdir -p /ne-deps/run/nitro_enclaves &&     mkdir -p /ne-deps/var/log/nitro_enclaves &&     cp -rf /usr/share/nitro_enclaves/ /ne-deps/usr/share/ &&     cp -f /etc/nitro_enclaves/allocator.yaml /ne-deps/etc/nitro_enclaves/allocator.yaml
 ---> Running in 0035e108175a
Removing intermediate container 0035e108175a
 ---> 96e20260ed39
Step 6/10 : FROM amazonlinux:2 as image
 ---> 92310fbb9f4c
Step 7/10 : COPY --from=full_image /ne-deps /
 ---> 71c2fa0ff1ac
Step 8/10 : COPY bin/hello.eif /home
 ---> 69cc6c47d179
Step 9/10 : COPY hello/run.sh  /home
 ---> 23f0640164d5
Step 10/10 : CMD ["/home/run.sh"]
 ---> Running in cf7b6710b265
Removing intermediate container cf7b6710b265
 ---> f1bccd652f13
[Warning] One or more build-args [config_region] were not consumed
Successfully built f1bccd652f13

Build 完后，可以看到新生成的镜像：

$ docker images |grep hello

    hello-aba86e99-db6c-40cd-a1b1-482f92422afa   latest    f1bccd652f13   8 minutes ago   244MB
    ne-build-hello-eif                           1.0       dfc1292341c5   9 minutes ago   4.87MB

4.2 将容器镜像推送到仓库

为了方便后续部署应用，建议将刚 build 好的 enclave 镜像放到 ECR 镜像仓库：

$ enclavectl push --image hello

Output example:

[enclavectl] Creating an ECR repository: hello-aba86e99-db6c-40cd-a1b1-482f92422afa.
{
    "repository": {
        "repositoryArn": "arn:aws:ecr:ap-northeast-1:123456543210:repository/hello-aba86e99-db6c-40cd-a1b1-482f92422afa",
        "registryId": "123456543210",
        "repositoryName": "hello-aba86e99-db6c-40cd-a1b1-482f92422afa",
        "repositoryUri": "123456543210.dkr.ecr.ap-northeast-1.amazonaws.com/hello-aba86e99-db6c-40cd-a1b1-482f92422afa",
        "createdAt": "2023-02-12T19:56:36+08:00",
        "imageTagMutability": "MUTABLE",
        "imageScanningConfiguration": {
            "scanOnPush": false
        },
        "encryptionConfiguration": {
            "encryptionType": "AES256"
        }
    }
}
WARNING! Your password will be stored unencrypted in /home/user/.docker/config.json.
Configure a credential helper to remove this warning. See
<https://docs.docker.com/engine/reference/commandline/login/#credentials-store>

Login Succeeded
Using default tag: latest
The push refers to repository [123456543210.dkr.ecr.ap-northeast-1.amazonaws.com/hello-aba86e99-db6c-40cd-a1b1-482f92422afa]
b7b44fe71b7b: Pushed 
d10aee14ebbe: Pushed 
ee98ccdc2ab6: Pushed 
8c5ff259ef50: Pushed 
latest: digest: sha256:53420e277f87baf98e92f4cf5d054c13030f3cf9bcffa1fb14010b9f1a753145 size: 1159

4.3 部署到 EKS 集群运行

为了清晰的了解 Enclaves 应用镜像部署到 EKS 集群过程，我们先将此步分解执行：

Step 1：生成 hello 应用的部署参数（deployment specification）：

$ enclavectl run --image hello --prepare-only
    [enclavectl] Generated deployment file: hello_deployment.yaml.

Step 2：打开自动生成的部署配置 .yaml 文件，进行检查和修改：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello-aba86e99-db6c-40cd-a1b1-482f92422afa
        image: 123456543210.dkr.ecr.ap-northeast-1.amazonaws.com/hello-aba86e99-db6c-40cd-a1b1-482f92422afa:latest
        command: ["/home/run.sh"]
        imagePullPolicy: Always
        resources:
          limits:
            aws.ec2.nitro/nitro_enclaves: "1"
            hugepages-2Mi: 768Mi
            cpu: 250m
          requests:
            aws.ec2.nitro/nitro_enclaves: "1"
            hugepages-2Mi: 768Mi
        volumeMounts:
        - mountPath: /hugepages-2Mi
          name: hugepage-2mi
          readOnly: false
        # Enable if 1Gi pages are required
        #- mountPath: /hugepages-1Gi
        #  name: hugepage-1gi
        #  readOnly: false
      volumes:
      - name: hugepage-2mi
        emptyDir:
          medium: HugePages-2Mi
      - name: hugepage-1gi
        emptyDir:
          medium: HugePages-1Gi
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists

Step 3：执行部署

$ kubectl apply -f hello_deployment.yaml
    deployment.apps/hello-deployment created

以上 3 步可以通过 enclavectl run 命令一步执行：

$ enclavectl run --image hello

[enclavectl] Generated deployment file: hello_deployment.yaml.
deployment.apps/hello-deployment created

检查 hello 应用的部署状态：

$ kubectl get pods --selector app=hello --watch

NAME                               READY   STATUS    RESTARTS   AGE
hello-deployment-6fd6b8c5c-blt4h   0/1     Pending   0          53s
hello-deployment-6fd6b8c5c-blt4h   0/1     Pending   0          31m
hello-deployment-6fd6b8c5c-blt4h   0/1     ContainerCreating   0          46m
hello-deployment-6fd6b8c5c-blt4h   0/1     CrashLoopBackOff    1 (2s ago)   46m
hello-deployment-6fd6b8c5c-blt4h   0/1     CrashLoopBackOff    2 (14s ago)   47m
hello-deployment-6fd6b8c5c-blt4h   1/1     Running             8 (5m4s ago)  48m

待应用运行起来后，查看容器终端日志显示，正常情况将看到“Hello from the enclave side”的信息：

$ kubectl logs hello-deployment-6fd6b8c5c-blt4h

[   1] Hello from the enclave side!
[   2] Hello from the enclave side!
[   3] Hello from the enclave side!
[   4] Hello from the enclave side!
[   5] Hello from the enclave side!
[   6] Hello from the enclave side!
...

这里我们总结一下利用 enclavectl 将 enclave应用部署到 eks 集群的步骤，并快速部署另一个 kmstools 示例应用：

# 1.构建enclave应用镜像
$ enclavectl build --image kms
# 2.推送到镜像仓库 
$ enclavectl push --image kms
# 3.部署到eks集群运行
$ enclavectl run --image kms

通过 kubectl 工具查看部署进度：

$ kubectl get pods --selector app=kms --watch

NAME                              READY   STATUS    RESTARTS   AGE
kms-deployment-845fc478fb-jsb4d   1/1     Running   0          84s

部署成功后可以通过 enclave 运行日志查看应用的执行状态：

$ kubectl logs kms-deployment-845fc478fb-jsb4d

[kms-example] Creating a KMS key...
[kms-example] Encrypting message...
[kms-example] ******************************
[kms-example] KMS Key ARN: arn:aws:kms:ap-northeast-1:123456543210:key/0215b696-4511-44bb-a8ed-a7308ab4549e
[kms-example] Account ID: 123456543210
[kms-example] Unencrypted message: Hello, KMS\\!
[kms-example] Ciphertext: AQICAHiuB3PUOFwhkteOeU+0RVZvz10I/46tEEwc6qWg2cT2ygH7XyeD18YTErRKykQBzNUwAAAAZDBiBgkqhkiG9w0BBwagVTBTAgEAME4GCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMRhRR+VYs5bsh6fHTAgEQgCHnGz2Vx7tjGwcxGrBiEjFJc9wpeEKugjJUynI8b48mzQg=
[kms-example] ******************************
Start allocating memory...
Started enclave with enclave-cid: 16, memory: 128 MiB, cpu-ids: [1, 3]
[kms-example] Requesting from the enclave to decrypt message...
[kms-example] ------------------------
[kms-example] > Got response from the enclave!
[kms-example] Object = { "Status": "Ok" } Object = { "Status": "Ok", "Message": "HelloKMS" }
[kms-example] ------------------------
Successfully terminated enclave i-0213c681fe3361bc8-enc18645cb8ba737e0.

五、环境清理

完成以上部署验证后，为避免持续产生费用，请按照以下步骤停止应用或者删除所有已部署的组件。

5.1 停止应用

可以通过 enclavectl stop 指令删除部署，以停止应用：

$ enclavectl stop --image kms

deployment.apps "kms-deployment" deleted

这个命令会在后台执行 kubectl -f delete kms_deployment.yaml 命令 , 所有通过 enclavectl run 命令初始化的资源都会被一并清除。

5.2 删除集群

利用 enclavectl 提供的 cleanup 指令删除 eks 集群：

$ enclavectl cleanup

[enclavectl] Attempt to delete cluster node group: ng-amd
Error: unable to describe cluster control plane: operation error EKS: DescribeCluster, https response error StatusCode: 404, RequestID: a99865df-2494-45c7-b468-fec23ed75dc1, ResourceNotFoundException: No cluster found for name: ne-eks.
[enclavectl] Cluster node group cannot be deleted.
[enclavectl] Attempt to delete launch template: lt_56c00212-47c9-4650-880f-dae4e3834386
{
    "LaunchTemplate": {
        "LaunchTemplateId": "lt-05976cce6f382b662",
        "LaunchTemplateName": "lt_284ecd59-9a85-4ba9-90f1-242fd3fd380d",
        "CreateTime": "2023-02-12T07:59:56+00:00",
        "CreatedBy": "arn:aws:iam::123456543210:user/wsadmin",
        "DefaultVersionNumber": 1,
        "LatestVersionNumber": 1
    }
}
[enclavectl] Deleting cluster_config.yaml...
[enclavectl] Deleting .config.ne.k8s.ctl...
[enclavectl] Deleting .setup.ne.k8s.ctl...

六、总结

本文介绍了如何手动创建以及借助 enclavectl 工具快速创建启用 Nitro Enclaves 功能的 EKS 集群和节点，并通过示例应用的部署体验完整过程。其中用户可以参考 EKS 集群创建和应用部署方案部分，Enclaves 应用开发及容器镜像的构建还需按照自己实际应用程序进行修改。

亚马逊AWS官方博客

在 AWS EKS 容器部署 Nitro Enclaves 可信计算应用程序

一、概述

二、准备工作

三、创建 EKS 集群和节点

3.1 手动创建 EKS 集群和节点

Step 1: 创建 launch template

Step 2：创建 EKS Cluster 和 nodegroup

Step 3：安装 Nitro Enclaves Kubernetes device plugin

3.2 通过 enclavectl 创建 EKS 集群和节点

Step 1：获取 enclavectl 工具

Step 2：修改集群配置

Step 3：自动化部署集群

3.3 注意事项

为 Worker 节点添加 enclaves enable 标签

为 Worker 节点开启 hugepage

四、将 enclave 应用部署到 EKS

4.1 构建 enclave 应用容器镜像

4.2 将容器镜像推送到仓库

4.3 部署到 EKS 集群运行

Step 1：生成 hello 应用的部署参数（deployment specification）：

Step 2：打开自动生成的部署配置 .yaml 文件，进行检查和修改：

Step 3：执行部署

五、环境清理

5.1 停止应用

5.2 删除集群

六、总结

本篇作者