如何使用用户数据自动配置 Amazon EKS 工作线程节点的 HTTP 代理?

上次更新时间:2020 年 4 月 9 日

我想使用用户数据自动配置 Amazon Elastic Kubernetes Service (Amazon EKS) 工作线程节点的 HTTP 代理。

解决方法

1.    查找您的集群的 IP CIDR 块:

$ kubectl get service kubernetes -o jsonpath='{.spec.clusterIP}'; echo

前述命令将返回 10.100.0.1172.20.0.1,表示您的集群 IP CIDR 块为 10.100.0.0/16172.20.0.0/16

2.    根据第 1 步中的命令的输出创建一个名为 proxy-env-vars-config.yaml的 ConfigMap 文件。

如果该输出含有来自范围 172.20.x.x 内的 IP,则将按下述方式设计 ConfigMap 文件的结构:

apiVersion: v1
kind: ConfigMap
metadata:
   name: proxy-environment-variables
   namespace: kube-system
data:
   HTTP_PROXY: http://customer.proxy.host:proxy_port
   HTTPS_PROXY: http://customer.proxy.host:proxy_port
   NO_PROXY: 172.20.0.0/16,localhost,127.0.0.1,VPC_CIDR_RANGE,169.254.169.254,.internal,s3.amazonaws.com,.s3.us-east-1.amazonaws.com,api.ecr.us-east-1.amazonaws.com,dkr.ecr.us-east-1.amazonaws.com,ec2.us-east-1.amazonaws.com

注意:请将 VPC_CIDR_RANGE 替换为您的集群 VPC 的 IPv4 CIDR 数据块。

如果该输出含有来自范围 10.100.x.x 内的 IP,则将按下述方式设计 ConfigMap 文件的结构:

apiVersion: v1
kind: ConfigMap
metadata:
   name: proxy-environment-variables
   namespace: kube-system
data:
   HTTP_PROXY: http://customer.proxy.host:proxy_port
   HTTPS_PROXY: http://customer.proxy.host:proxy_port
   NO_PROXY: 10.100.0.0/16,localhost,127.0.0.1,VPC_CIDR_RANGE,169.254.169.254,.internal,s3.amazonaws.com,.s3.us-east-1.amazonaws.com,api.ecr.us-east-1.amazonaws.com,dkr.ecr.us-east-1.amazonaws.com,ec2.us-east-1.amazonaws.com

注意:请将 VPC_CIDR_RANGE 替换为您的集群 VPC 的 IPv4 CIDR 数据块。

如果通过专用 API 服务器终端节点访问、专用子网和无互联网访问构建 Amazon EKS 集群,则您必须为以下对象创建和添加终端节点:

Amazon Elastic Container Registry (Amazon ECR)
Amazon Simple Storage Service (Amazon S3)
Amazon Elastic Compute Cloud (Amazon EC2)
Amazon Virtual Private Cloud (Amazon VPC)

例如,您可以使用以下终端节点:

api.ecr.us-east-1.amazonaws.com
dkr.ecr.us-east-1.amazonaws.com
s3.amazonaws.com
s3.us-east-1.amazonaws.com
ec2.us-east-1.amazonaws.com

重要提示:您必须为 NO_PROXY 变量添加公共终端节点子域。例如,为 us-east-1 AWS 区域中的 Amazon S3 添加 .s3.us-east-1.amazonaws.com 域。

3.    验证 configmap/proxy-environment-variables(由 kube-proxyaws-node pods 使用)中的 NO_PROXY 变量是否包含 Kubernetes 集群 IP 地址空间。例如,在 IP 范围来自 10.100.x.x 的 ConfigMap 文件的前述代码示例中,使用 10.100.0.0/16

4.    应用 ConfigMap:

$ kubectl apply -f /path/to/yaml/proxy-env-vars-config.yaml

5.    要配置 Docker 守护程序和 kubelet,请将用户数据注入您的工作线程节点。请参阅以下示例:

Content-Type: multipart/mixed; boundary="==BOUNDARY=="
MIME-Version:  1.0
--==BOUNDARY==
Content-Type: text/cloud-boothook; charset="us-ascii"

#Set the proxy hostname and port
PROXY="proxy.local:3128"
MAC=$(curl -s http://169.254.169.254/latest/meta-data/mac/)
VPC_CIDR=$(curl -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC/vpc-ipv4-cidr-blocks | xargs | tr ' ' ',')

#Create the docker systemd directory
mkdir -p /etc/systemd/system/docker.service.d

#Configure yum to use the proxy
cloud-init-per instance yum_proxy_config cat << EOF >> /etc/yum.conf
proxy=http://$PROXY
EOF

#Set the proxy for future processes, and use as an include file
cloud-init-per instance proxy_config cat << EOF >> /etc/environment
http_proxy=http://$PROXY
https_proxy=http://$PROXY
HTTP_PROXY=http://$PROXY
HTTPS_PROXY=http://$PROXY
no_proxy=$VPC_CIDR,localhost,127.0.0.1,169.254.169.254,.internal,s3.amazonaws.com,.s3.us-east-1.amazonaws.com,api.ecr.us-east-1.amazonaws.com,dkr.ecr.us-east-1.amazonaws.com,ec2.us-east-1.amazonaws.com
NO_PROXY=$VPC_CIDR,localhost,127.0.0.1,169.254.169.254,.internal,s3.amazonaws.com,.s3.us-east-1.amazonaws.com,api.ecr.us-east-1.amazonaws.com,dkr.ecr.us-east-1.amazonaws.com,ec2.us-east-1.amazonaws.com
EOF

#Configure docker with the proxy
cloud-init-per instance docker_proxy_config tee <<EOF /etc/systemd/system/docker.service.d/proxy.conf >/dev/null
[Service]
EnvironmentFile=/etc/environment
EOF

#Configure the kubelet with the proxy
cloud-init-per instance kubelet_proxy_config tee <<EOF /etc/systemd/system/kubelet.service.d/proxy.conf >/dev/null
[Service]
EnvironmentFile=/etc/environment
EOF

#Reload the daemon and restart docker to reflect proxy configuration at launch of instance
cloud-init-per instance reload_daemon systemctl daemon-reload 
cloud-init-per instance enable_docker systemctl enable --now --no-block docker

--==BOUNDARY==
Content-Type:text/x-shellscript; charset="us-ascii"

#!/bin/bash
set -o xtrace

#Set the proxy variables before running the bootstrap.sh script
set -a
source /etc/environment

/etc/eks/bootstrap.sh ${ClusterName} ${BootstrapArguments}

# Use the cfn-signal only if the node is created through an AWS CloudFormation stack and needs to signal back to an AWS CloudFormation resource (CFN_RESOURCE_LOGICAL_NAME) that waits for a signal from this EC2 instance to progress through either:
# - CreationPolicy https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-creationpolicy.html
# - UpdatePolicy https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-updatepolicy.html
# cfn-signal will signal back to AWS CloudFormation using https transport, so set the proxy for an HTTPS connection to AWS CloudFormation
/opt/aws/bin/cfn-signal
    --exit-code $? \
    --stack  ${AWS::StackName} \
    --resource CFN_RESOURCE_LOGICAL_NAME  \
    --region ${AWS::Region} \
    --https-proxy $HTTPS_PROXY

--==BOUNDARY==--

重要提示:启动 Docker 守护程序和 kubelet 之前,您必须更新或创建 yumdockerkubelet 配置文件。

有关使用从 AWS 管理控制台启动的 AWS CloudFormation 模板将用户数据注入工作线程节点的示例,请参阅启动 Amazon EKS Linux 工作线程节点

6.    更新 aws-nodekube-proxy pod:

$ kubectl patch -n kube-system -p '{ "spec": {"template": { "spec": { "containers": [ { "name": "aws-node", "envFrom": [ { "configMapRef": {"name": "proxy-environment-variables"} } ] } ] } } } }' daemonset aws-node
$ kubectl patch -n kube-system -p '{ "spec": {"template":{ "spec": { "containers": [ { "name": "kube-proxy", "envFrom": [ { "configMapRef": {"name": "proxy-environment-variables"} } ] } ] } } } }' daemonset kube-proxy

如果您更改了 ConfigMap,则应用更新,并重新设置 pod 中的 ConfigMap。请参阅以下示例:

$ kubectl set env daemonset/kube-proxy --namespace=kube-system --from=configmap/proxy-environment-variables --containers='*'
$ kubectl set env daemonset/aws-node --namespace=kube-system --from=configmap/proxy-environment-variables --containers='*'

重要提示:升级这些对象时,您必须将所有的 YAML 修改更新到 Kubernetes 对象 kube-proxyaws-node 中。要将 ConfigMap 更新为默认值,请使用 eksctl utils update-kube-proxyeksctl utils update-aws-node 命令。

提示:如果代理丢失了与 API 服务器的连接,则该代理会变为单点故障,且您的集群行为可能不可预测。为了防止您的代理变为单点故障,在服务发现命名空间或负载均衡器后台运行您的代理。

7.    确认 kube-proxyaws-node pod 中使用代理变量:

$ kubectl describe pod kube-proxy-xxxx -n kube-system

输出应类似于以下内容:

Environment:
      HTTPS_PROXY:  <set to the key 'HTTPS_PROXY' of config map 'proxy-environment-variables'>  Optional: false
      HTTP_PROXY:   <set to the key 'HTTP_PROXY' of config map 'proxy-environment-variables'>   Optional: false
      NO_PROXY:     <set to the key 'NO_PROXY' of config map 'proxy-environment-variables'>     Optional: false

如果您没有使用 AWS PrivateLink,请验证是否能够通过 Amazon EC2、Amazon ECR 和 Amazon S3 的代理服务器访问 API 终端节点。


这篇文章对您有帮助吗?

我们可以改进什么?


需要更多帮助?