当我尝试将慢日志发布到 CloudWatch Logs 时,如何解决 CloudFormation 中收到的错误?
上次更新日期:2021 年 3 月 16 日
我想解决在尝试将慢日志发布到 Amazon CloudWatch Logs 时在 AWS CloudFormation 中收到的错误。错误是:“为 CloudWatch Logs 日志组 /aws/aes/domains/search/search-logs 指定的资源访问策略未授予 Amazon Elasticsearch Service 创建日志流的足够权限。”
简短描述
要解决此错误,请在日志组级别使用单独的策略,以允许 Amazon Elasticsearch Service (Amazon ES) 将日志推送到 CloudWatch Logs。然后,使用 AWS::Elasticsearch::Domain 资源中的 AccessPolicies 为 Amazon ES 域设置权限。
以下步骤将向您介绍如何通过 Python 3.6 中创建的 AWS Lambda 支持的自定义资源,使用 CloudFormation 将慢日志发布到 CloudWatch 中。自定义资源将触发 Lambda 函数,该函数将触发 PutResourcePolicy API 以发布慢日志。
注意:当 CloudFormation 启用向 CloudWatch 发布日志时,AWS::Logs::LogGroup 资源没有用于分配资源访问策略的属性。为 CloudWatch Logs 日志组指定的资源访问策略应为 Amazon ES 授予足够的权限来发布日志流。您无法直接使用 CloudFormation 资源创建访问策略权限。这是因为 AWS::Logs::LogGroup 资源的 PutResourcePolicy API 调用不受 CloudFormation 支持。
解决方法
注意:如果您在运行 AWS Command Line Interface (AWS CLI) 命令时收到错误讯息,请确保您运行的是最新版本的 AWS CLI。
以下 CloudFormation 模板使用自定义资源以获取日志组的名称。然后,模板应用允许 Amazon ES 服务对日志组进行 API 调用的策略。
1. 创建名为 ESlogsPermission.yaml 的 CloudFormation 模板:
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
Permission is hereby granted, free of charge, to any person obtaining a copy of this
software and associated documentation files (the "Software"), to deal in the Software
without restriction, including without limitation the rights to use, copy, modify,
merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
AWSTemplateFormatVersion: 2010-09-09
Description: AWS cloudFormation template to publish slow logs to Amazon CloudWatch Logs.
Parameters:
LogGroupName:
Type: String
Description: Please don't change the log group name while updating
ESDomainName:
Description: A name for the Amazon Elastic Search domain
Type: String
LambdaFunctionName:
Description: Lambda Function Name
Type: String
Resources:
AwsLogGroup:
Type: 'AWS::Logs::LogGroup'
Properties:
LogGroupName: !Ref LogGroupName
LambdaLogGroup:
Type: 'AWS::Logs::LogGroup'
Properties:
LogGroupName: !Sub '/aws/lambda/${LambdaFunctionName}'
LambdaExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: root1
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: !Sub >-
arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/${LambdaFunctionName}:log-stream:*
- PolicyName: root2
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'logs:CreateLogGroup'
Resource:
- !Sub >-
arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/${LambdaFunctionName}
- !Sub >-
arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/${LogGroupName}
- Effect: Allow
Action:
- 'logs:PutResourcePolicy'
- 'logs:DeleteResourcePolicy'
Resource: !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:*'
logGroupPolicyFunction:
DependsOn: LambdaLogGroup
Type: 'AWS::Lambda::Function'
Properties:
FunctionName: !Ref LambdaFunctionName
Code:
ZipFile: >
import urllib3
import json
import boto3
http = urllib3.PoolManager()
SUCCESS = "SUCCESS"
FAILED = "FAILED"
def send(event, context, responseStatus, responseData,
physicalResourceId=None, noEcho=False):
responseUrl = event['ResponseURL']
print(responseUrl)
responseBody = {}
responseBody['Status'] = responseStatus
responseBody['Reason'] = 'See the details in CloudWatch Log Stream: ' + context.log_stream_name
responseBody['PhysicalResourceId'] = physicalResourceId or context.log_stream_name
responseBody['StackId'] = event['StackId']
responseBody['RequestId'] = event['RequestId']
responseBody['LogicalResourceId'] = event['LogicalResourceId']
responseBody['NoEcho'] = noEcho
responseBody['Data'] = responseData
json_responseBody = json.dumps(responseBody)
print("Response body:\n" + json_responseBody)
headers = {
'content-type' : '',
'content-length' : str(len(json_responseBody))
}
try:
response = http.request('PUT',responseUrl,body=json_responseBody.encode('utf-8'),headers=headers)
print("Status code: " + response.reason)
except Exception as e:
print("send(..) failed executing requests.put(..): " + str(e))
def handler(event, context):
logsgroup_policy_name=event['ResourceProperties']['CWLOGS_NAME']
cw_log_group_arn=event['ResourceProperties']['CWLOG_ARN']
cwlogs = boto3.client('logs')
loggroup_policy={
"Version": "2012-10-17",
"Statement": [{
"Sid": "",
"Effect": "Allow",
"Principal": { "Service": "es.amazonaws.com"},
"Action":[
"logs:PutLogEvents",
" logs:PutLogEventsBatch",
"logs:CreateLogStream"
],
'Resource': f'{cw_log_group_arn}'
}]
}
loggroup_policy = json.dumps(loggroup_policy)
if(event['RequestType'] == 'Delete'):
print("Request Type:",event['RequestType'])
cwlogs.delete_resource_policy(
policyName=logsgroup_policy_name
)
responseData={}
send(event, context, SUCCESS, responseData)
elif(event['RequestType'] == 'Create'):
try:
cwlogs.put_resource_policy(
policyName = logsgroup_policy_name,
policyDocument = loggroup_policy
)
responseData={}
print("Sending response to custom resource")
send(event, context, SUCCESS, responseData)
except Exception as e:
print('Failed to process:', e)
send(event, context, FAILED, responseData)
elif(event['RequestType'] == 'Update'):
try:
responseData={}
print("Update is not supported on this resource")
send(event, context, SUCCESS, responseData)
except Exception as e:
print('Failed to process:', e)
send(event, context, FAILED, responseData)
Handler: index.handler
Role: !GetAtt
- LambdaExecutionRole
- Arn
Runtime: python3.6
logGroupPolicycustomresource:
Type: 'Custom::LogGroupPolicy'
Properties:
ServiceToken: !GetAtt
- logGroupPolicyFunction
- Arn
CWLOGS_NAME: !Ref LogGroupName
CWLOG_ARN: !GetAtt
- AwsLogGroup
- Arn
ElasticsearchDomain:
Type: 'AWS::Elasticsearch::Domain'
DependsOn: logGroupPolicycustomresource
Properties:
DomainName: !Ref ESDomainName
ElasticsearchVersion: '6.2'
EBSOptions:
EBSEnabled: true
VolumeSize: 10
VolumeType: gp2
LogPublishingOptions:
SEARCH_SLOW_LOGS:
CloudWatchLogsLogGroupArn: !GetAtt
- AwsLogGroup
- Arn
Enabled: true
2. 要使用 ESlogsPermission.yaml 文件启动 CloudFormation 堆栈,请使用 CloudFormation 控制台或以下 AWS CLI 命令:
aws cloudformation create-stack --stack-name yourStackName --template-body file://yourTemplateName --parameters ParameterKey=LogGroupName,ParameterValue=Your-LogGroup-Name, ParameterKey=ESDomainName,ParameterValue=Your-ES-Name --capabilities CAPABILITY_NAMED_IAM --region yourRegion
注意:请将 yourStackName、yourTemplateName、Your-LogGroup-Name、Your-ES-Name 和 yourRegion 替换为您的值。
CloudFormation 模板为您执行以下操作:
1. 创建日志组。
2. 创建 Lambda 函数。Lambda 函数通过使用自定义资源从 CloudFormation 模板的参数部分中获取日志组名称。Lambda 函数为日志组名称调用 PutResourcePolicy API。日志组必须具有允许 Amazon ES 域放置日志的策略。
3. 创建由 Lambda 支持的自定义资源以调用在步骤 2 中创建的 Lambda 函数。自定义资源有助于将 PutResourcePolicy 应用于日志组 Amazon 资源名称 (ARN),以便 Amazon ES 可以流式传输日志。在模板中,CloudFormation 使用自定义资源通过 LogPublishingOption 创建 Amazon ES 域。