亚马逊AWS官方博客

Amazon SES邮件备用方案初探

摘要

企业客户在使用亚马逊云科技（Amazon Web Services）的云服务开发应用时，向企业内部或者合作伙伴的收件人发送通知邮件是经常遇到的功能需求。亚马逊云服务在境外区域使用Amazon Simple Email Service (SES)提供大规模入站和出站云电子邮件服务，境内的北京区域（cn-north-1）与宁夏区域（cn-northwest-1）的SES服务在截至本文撰写日还未可用。另一个可选项是亚马逊云科技的Amazon Simple Notification Service (SNS)服务，SNS提供订阅邮件推送的功能，但是在实际使用中SNS与订阅者需要通过简单的握手，向随机收件人发送邮件通知仍有局限性。为了方便云上企业应用向企业内部或者合作伙伴的随机收件人发送邮件通知，本文探讨一种SES邮件服务的备用方案（Workaround Solution），使用亚马逊云科技无服务器化（Serverless）的Amazon Simple Queue Service (SQS)与AWS Lambda服务，结合使用企业自有的SMTP服务器中继完成发送邮件通知的功能。

方案

背景

为了更安全高效的使用云上服务，亚马逊云科技的企业客户普遍使用多AWS账号以组织的方式规划云上环境，并且将应用部署在与Internet隔绝的（Air-gapped）私有网络环境中以满足更高的安全要求。由于北京区域（cn-north-1）与宁夏区域（cn-northwest-1）的SES服务暂时未有发布，当这些应用需要向企业内或合作伙伴的随机收件人发送邮件通知时不能使用SES服务，但是企业通常有购买第三方的企业级邮箱服务或者企业IT基础架构在云上部署的SMTP服务器支持企业日常的邮件服务。

本文所述的方案（后称MailDelivery方案）假定两个部署在中国区域的私有网络环境中的应用：

  • Alpha应用处理API调用，发送调用结果邮件给指定收件人；
  • Bravo应用处理调度编排，发送申请或批复邮件通知到指定收件人；

MailDelivery方案适用于满足以下部署条件的应用：

  • 企业应用需要发送通知邮件给随机收件人，邮件不含附件
  • 不方便使用SES服务的区域
  • 应用部署在Internet隔绝的私有网络
  • 公有网络中部署的SMTP服务器，或者公有网络可访问到第三方邮件提供商提供的企业邮件服务

架构


MailDelivery使用无服务器架构（SMTP 服务器除外），假定部署环境为北京（cn-north-1）区域，实现的功能包括：邮件组装、消息转送、邮件发送、应用日志等；使用到的亚马逊云科技的云服务包括：VPC相关服务、Endpoint、Lambda、SQS；其它：Python 3.7、Python smtplib、Boto3。

一）邮件组装（Mail producer）

邮件的组装在私有网络中的应用中完成，MailDelivery假定应用可以调用Lambda函数或者应用逻辑原本直接在Lambda函数中处理（架构图，标1）。邮件数据按照约定的JSON结构组织，作为Lambda函数的Payload传入，通过调用Boto3库将邮件作为消息发送到SQS队列。

二）消息转送（Mail broker）

邮件消息的装送通过SQS的标准队列完成，MailDelivery假定企业在云上部署有公有网络环境的账号。在该账号下部署邮件SQS（架构图，标2）以及死信SQS队列。

三）邮件发送（Mail delivery）

在邮件SQS相同账号的私有子网中部署Lambda函数（架构图，标3），该函数使用SQS消息作为触发事件，获取邮件消息并根据约定的JSON格式解析邮件，使用Python smtplib库将邮件以MIME对象将通过SMTP服务器发送。邮件发送需要用到的SMTP服务器账号按照约定格式作为参数预先在ParameterStore中配置。

实现

MailDelivery部署样例使用CloudFormation模板编写，Lambda函数代码使用Inline方式嵌入CloudFormation模版中。实现的技术要点如下：

一）SMTP账号的格式约定

账号以SecureString类型保存在ParameterStore的/MailDelivery/Ticket/路径之下。例如，参数/MailDelivery/Ticket/DemoTicket，DemoTicket由用户指定。

{
    "Host": "demo-smtp.host.com",
    "Port": 25,
    "User": "demo-user1",
    "Pass": "demo-pass1"
}

二）JSON邮件的格式约定

邮件数据使用如下JSON格式作为Lambda payload传入或作为消息发送到SQS

{
    "Ticket": "DemoTicket",
    "Subject": "Demo",
    "From": "from@demo.com",
    "To": ["to.1@demo.com", "to.2@demo.com", "to.3@demo.com"],
    "Cc": ["cc.1@demo.com", "cc.2@demo.com"],
    "Text": "Hello Demo!",
    "Html": "<html>Hello Demo!</html>"
}

三）邮件组装的部署模板

模版cfn-app-mail-producer.yaml样例代码

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: |
  This template deploys mail producer application.

Parameters:

  VpcId:
    Type: String
    Description: Vpc id
  PrivateSubnetIds:
    Type: String
    Description: Comma delimited list of private subnet ids
  MailDeliveryAccountId:
    Type: String
    Description: Mail delivery account id
  MailDeliverySqsName:
    Type: String
    Description: Mail delivery sqs name

Resources:
          
  MailProducerFunction:
    Type: AWS::Serverless::Function
    DependsOn: MailProducerFunctionLogGroup
    Properties:
      FunctionName: MailProducer
      Description: Mail producer function
      # <Inline code for mail producer>
      InlineCode: |
        import os
        import boto3
        import logging
        import json

        logger = logging.getLogger()
        logger.setLevel(logging.INFO)

        def lambda_handler(event, context):
            logger.info('-------- Producing Mails --------')
            msg = {}
            
            if 'Ticket' not in event:
                logger.error("Error: Mail's Ticket is not provided.")
                raise Exception("IncompleteMailException")
            msg['Ticket'] = event['Ticket']
            
            if 'Subject' not in event:
                logger.error("Error: Mail's Subject is not provided.")
                raise Exception("IncompleteMailException")
            msg['Subject'] = event['Subject']
            
            if 'From' not in event:
                logger.error("Error: Mail's From is not provided.")
                raise Exception("IncompleteMailException")
            msg['From'] = event['From']
            
            if 'To' not in event:
                logger.error("Error: Mail's To is not provided.")
                raise Exception("IncompleteMailException")
            msg['To'] = event['To']
            
            if 'Cc' not in event:
                logger.warning("Warning: Mail's Cc is not provided.")
            else:
                msg['Cc'] = event['Cc']
            
            if 'Html' not in event and 'Html' not in event:
                logger.error("Error: Neither Mail's Text or Html is provided.")
                raise Exception("IncompleteMailException")
                
            if 'Html' in event:
                msg['Html'] = event['Html']
            
            if 'Text' in event:
                msg['Text'] = event['Text']
                
            sqs = boto3.client('sqs')
            response = sqs.send_message(
                QueueUrl = os.getenv('mail_delivery_sqs'),
                MessageBody = json.dumps(msg)
            )
            
            logger.info('-------- Mails Produced --------')
            return {'Status':'OK'}
      # </Inline code for mail producer>
      Handler: index.lambda_handler
      Runtime: python3.7
      MemorySize: 128
      Timeout: 32
      VpcConfig:
        SecurityGroupIds:
          - !GetAtt MailProducerFunctionSecurityGroup.GroupId
        SubnetIds: !Split [',', !Ref PrivateSubnetIds]
      Environment:
        Variables:
          mail_delivery_sqs: !Sub https://sqs.${AWS::Region}.amazonaws.com.cn/${MailDeliveryAccountId}/${MailDeliverySqsName}
      Policies:
        - AWSLambdaBasicExecutionRole
        - AWSXrayWriteOnlyAccess
        - AWSLambdaVPCAccessExecutionRole
        - Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - lambda:InvokeFunction
              Resource:
                - !Sub arn:aws-cn:lambda:${AWS::Region}:${AWS::AccountId}:function:*
            - Effect: Allow
              Action:
                - SQS:GetQueueAttributes
                - SQS:SendMessage
                - SQS:ReceiveMessage
                - SQS:GetQueueUrl
              Resource: !Sub arn:aws-cn:sqs:${AWS::Region}:${MailDeliveryAccountId}:*
            - Effect: Allow
              Action:
                - "kms:Decrypt"
                - "kms:Encrypt"
                - "kms:GenerateDataKey"
              Resource: "*"

  MailProducerFunctionSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Mail producer function security group
      VpcId: !Ref VpcId

  MailProducerFunctionLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub /aws/lambda/MailProducer

  SqsVpcEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      PrivateDnsEnabled: False
      SecurityGroupIds:
        - !GetAtt SqsVpcEndpointSecurityGroup.GroupId
      ServiceName: !Sub cn.com.amazonaws.${AWS::Region}.sqs
      SubnetIds: !Split [',', !Ref PrivateSubnetIds]
      VpcEndpointType: Interface
      VpcId: !Ref VpcId

  SqsVpcEndpointSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      VpcId: !Ref VpcId
      GroupDescription: Sqs vpc endpoint security group
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          SourceSecurityGroupId: !GetAtt MailProducerFunctionSecurityGroup.GroupId

四）邮件发送的部署模板

模版cfn-app-mail-delivery.yaml样例代码

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: |
  This template deploys mail delivery application.

Parameters:

  VpcId:
    Type: String
    Description: Vpc id
  PrivateSubnetIds:
    Type: String
    Description: Comma delimited list of private subnet ids
  PublicSubnetIds:
    Type: String
    Description: Comma delimited list of public subnet ids
  MailProducerAccountIds:
    Type: String
    Description: Comma delimited list of mail producer account ids

Resources:
          
  MailDeliveryFunction:
    Type: AWS::Serverless::Function
    DependsOn: MailDeliveryFunctionLogGroup
    Properties:
      FunctionName: MailDelivery
      Description: Mail delivery
      # <Inline code for mail delivery>
      InlineCode: |
        import os
        import boto3
        import logging
        import json
        import smtplib

        from email.mime.multipart import MIMEMultipart
        from email.mime.text import MIMEText

        logger = logging.getLogger()
        logger.setLevel(logging.INFO)

        def lambda_handler(event, context):
            logger.info('-------- Delivering Mails --------')
            
            for sqs_record in event['Records']:
                msg_record = json.loads(sqs_record["body"])
                mime= MIMEMultipart('alternative')
                
                if 'Subject' not in msg_record:
                    logger.error("Error: Mail's Subject is not provided.")
                    raise Exception("IncompleteMailException")
                mime['Subject'] = msg_record['Subject']
                        
                if 'From' not in msg_record:
                    logger.error("Error: Mail's From is not provided.")
                    raise Exception("IncompleteMailException")
                mime['From'] = msg_record['From']
                        
                if 'To' not in msg_record:
                    logger.error("Error: Mail's To is not provided.")
                    raise Exception("IncompleteMailException")
                mime['To'] = ";".join(msg_record['To'])
                        
                if 'Cc' not in msg_record:
                    logger.warning("Warning: Mail's Cc is not provided.")
                else:
                    mime['Cc'] = ";".join(msg_record['Cc'])
                
                if 'Text' not in msg_record and 'Html' not in msg_record:
                    logger.error("Error: Neither Mail's Text or Html is provided.")
                    raise Exception("IncompleteMailException")
                    
                if 'Html' in msg_record:
                    mime.attach(MIMEText(msg_record["Html"], 'html'))
                    
                if 'Text' in msg_record:
                    mime.attach(MIMEText(msg_record["Text"], 'plain'))
                    
                if 'Ticket' not in msg_record:
                    logger.error("Error: Mail's Ticket is not provided.")
                    raise Exception("IncompleteMailException")
                    
                ssm = boto3.client('ssm')
                ticket = json.loads(
                    ssm.get_parameter(
                        Name = "/MailDelivery/Ticket/{0}".format(msg_record['Ticket']),
                        WithDecryption = True
                    )['Parameter']['Value']
                )
                
                with smtplib.SMTP(ticket['Host'], ticket['Port'], timeout = 64) as server:
                    server.starttls()
                    if 'User' in ticket and 'Pass' in ticket:
                        server.login(ticket['User'], ticket['Pass'])
                    server.send_message(mime)
                    server.quit()
                    
            logger.info('-------- Mails Delivered --------')
            return {'Status':'OK'}
      # </Inline code for mail delivery>
      Handler: index.lambda_handler
      Runtime: python3.7
      MemorySize: 128
      Timeout: 32
      VpcConfig:
        SecurityGroupIds:
          - !GetAtt MailDeliveryFunctionSecurityGroup.GroupId
        SubnetIds: !Split [',', !Ref PrivateSubnetIds]
      Events:
        SQSEvent:
          Type: SQS
          Properties:
            Queue: !GetAtt MailDeliveryQueue.Arn
            BatchSize: 1
            Enabled: true
      Policies:
        - AWSLambdaBasicExecutionRole
        - AWSXrayWriteOnlyAccess
        - AWSLambdaVPCAccessExecutionRole
        - Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - lambda:InvokeFunction
              Resource:
                - !Sub arn:aws-cn:lambda:${AWS::Region}:${AWS::AccountId}:function:*
            - Effect: Allow
              Action:
                - SQS:GetQueueAttributes
                - SQS:SendMessage
                - SQS:ReceiveMessage
                - SQS:GetQueueUrl
              Resource:
                - !Sub arn:aws-cn:sqs:${AWS::Region}:${AWS::AccountId}:*
            - Effect: Allow
              Action:
                - "kms:Decrypt"
                - "kms:Encrypt"
                - "kms:GenerateDataKey"
              Resource: "*"
            - Effect: Allow
              Action:
                - ssm:DescribeParameters
                - ssm:GetParameter
                - ssm:GetParameterHistory
                - ssm:GetParameters
                - ssm:GetParametersByPath
              Resource:
                - !Sub arn:aws-cn:ssm:${AWS::Region}:${AWS::AccountId}:parameter/*

  MailDeliveryFunctionLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub /aws/lambda/MailDelivery

  MailDeliveryFunctionSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Mail delivery function security group
      VpcId: !Ref VpcId

  MailDeliveryQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: MailDelivery
      KmsMasterKeyId: alias/aws/sqs
      VisibilityTimeout: 64
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt MailDeliveryDeadLetterQueue.Arn
        maxReceiveCount: 8

  MailDeliveryQueuePolicy:
    DependsOn: MailDeliveryQueue
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref MailDeliveryQueue
      PolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - SQS:GetQueueAttributes
              - SQS:SendMessage
              - SQS:ReceiveMessage
              - SQS:GetQueueUrl
            Resource: !GetAtt MailDeliveryQueue.Arn
            Principal:
              AWS: !Split [',', !Ref MailProducerAccountIds]

  MailDeliveryDeadLetterQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: MailDeliveryDeadLetter

  SqsVpcEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      PrivateDnsEnabled: False
      SecurityGroupIds:
        - !GetAtt SqsVpcEndpointSecurityGroup.GroupId
      ServiceName: !Sub cn.com.amazonaws.${AWS::Region}.sqs
      SubnetIds: !Split [',', !Ref PrivateSubnetIds]
      VpcEndpointType: Interface
      VpcId: !Ref VpcId

  SqsVpcEndpointSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      VpcId: !Ref VpcId
      GroupDescription: Sqs vpc endpoint security group
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          SourceSecurityGroupId: !GetAtt MailDeliveryFunctionSecurityGroup.GroupId

  SsmVpcEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      PrivateDnsEnabled: False
      SecurityGroupIds:
        - !GetAtt SsmVpcEndpointSecurityGroup.GroupId
      ServiceName: !Sub com.amazonaws.${AWS::Region}.ssm
      SubnetIds: !Split [',', !Ref PrivateSubnetIds]
      VpcEndpointType: Interface
      VpcId: !Ref VpcId

  SsmVpcEndpointSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      VpcId: !Ref VpcId
      GroupDescription: Ssm vpc endpoint security group
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          SourceSecurityGroupId: !GetAtt MailDeliveryFunctionSecurityGroup.GroupId

Outputs:

  MailDeliveryQueue:
    Value: !Ref MailDeliveryQueue
  MailDeliveryQueueArn:
    Value: !GetAtt MailDeliveryQueue.Arn

讨论

一）申请解除邮件服务限制

默认情况下，AWS会阻止所有EC2实例和 Lambda 函数的端口25（SMTP）上的出站流量。如要在端口25上发送出站流量，可以请求取消此限制。参考博文《如何从Amazon EC2实例或AWS Lambda函数删除端口25上的限制》。

二）使用KMS加密数据

请根据安全合规的要求修改部署模板，使用客户而非alias/aws/名称下的KMS密钥，需要对数据加密的对象包括：SQS、ParameterStore。

三）邮件限制

因SQS消息最大限制为256KB，邮件大小要尽量控制其小于256KB，且无附件。可以考虑使用Kinesis增大邮件大小的限制。

四）部署与调用

  • Step 1：可选。使用AWS QuickStart模版部署Mail producer以及Mail delivery的VPC环境。若现有账号以及网络与架构图相符且可用，跳过此步骤。
  • Step 2：必须。在Mail delivery账号中使用模版cfn-app-mail-delivery.yaml部署Mail broker以及Mail delivery。
  • Step 3：必须。在Mail producer账号中使用模版cfn-app-mail-producer.yaml部署Mail producer。
  • Step 4：必须。手动将SMTP账号以SecureString配置到ParameterStore的/MailDelivery/Ticket/路径之下。举例，/MailDelivery/Ticket/DemoTicket。
  • Step 5：可选。在API接口调用Mail producer的Lambda函数，在Payload传入邮件资料的JSON对象，指定Ticket字段值，例如等于DemoTicket。

本篇作者

赵鑫

亚马逊云科技专业服务团队数据架构师，专注于生命科学、自动驾驶领域的数据架构与数据分析

牛壬硕

AWS专业服务团队大数据咨询顾问。 负责企业级数据平台架构设计/迁移/安全/运维和优化，数据仓库以及数据治理等相关咨询服务。 在大数据领域约有10年经验，对针对各类依托数据平台进行数字化转型的解决方案以及数据应用有着丰富的经验。