Cloud Formation 将多个 S3 触发器添加到 LambdaConfigurations 中的同一个 S3 存储桶

Posted 2023-03-04

【中文标题】Cloud Formation 将多个 S3 触发器添加到 LambdaConfigurations 中的同一个 S3 存储桶

【英文标题】：Cloud Formation to add multiple S3 triggers to same S3 bucket in LambdaConfigurations

【发布时间】：2020-06-28 18:41:05

【问题描述】：

如果input.txt 文件在 S3 存储桶中创建，我的要求是触发 Lambda_Function_1，如果 output.txt 文件在同一个 S3 存储桶中创建，则触发 Lambda_Function_2。

下面的 cfn 不起作用，但如果我只将一个事件而不是两个事件放在同一个 LambdaConfigurations 中，它就可以正常工作。

有人可以帮我吗？

Parameters:
  S3BucketBaseName:
    Type: String
    Description: The base name of the Amazon S3 bucket.
    Default: dw-trip


Resources:
  LambdaStart: 
    DependsOn:
      - LambdaStartStopEC2
    Type: "AWS::Lambda::Function"
    Properties:
      FunctionName: "dw-trip-start-ec2"
      Handler: "index.handler"
      Role: !GetAtt LambdaStartStopEC2.Arn
      Runtime: python3.7
      MemorySize: 3008
      Timeout: 900
      Code:
        ZipFile: |
          import boto3
          region = 'us-east-1'
          instances = ['i-05d5fbec4c82956b6']
          ec2 = boto3.client('ec2', region_name=region)
          def lambda_handler(event, context):
              ec2.start_instances(InstanceIds=instances)
              print('started your instances: ' + str(instances))

  ProcessingLambdaPermissionStart:
    Type: AWS::Lambda::Permission
    DependsOn:
      - LambdaStart    
    Properties:
      Action: 'lambda:InvokeFunction'
      FunctionName: !Ref LambdaStart
      Principal: s3.amazonaws.com
      SourceArn:
                Fn::Join: 
                  - ''
                  - - 'arn:aws:s3:::'
                    - !Join ["-",[!Ref "S3BucketBaseName",!Ref "AWS::AccountId"]]
      SourceAccount: !Ref AWS::AccountId

  LambdaStop: 
    DependsOn:
      - ProcessingLambdaPermissionStart
    Type: "AWS::Lambda::Function"
    Properties:
      FunctionName: "dw-trip-stop-ec2"
      Handler: "index.handler"
      Role: !GetAtt LambdaStartStopEC2.Arn
      Runtime: python3.7
      MemorySize: 3008
      Timeout: 900
      Code:
        ZipFile: |
          import boto3
          region = 'us-east-1'
          instances = ['i-05d5fbec4c82956b6']
          ec2 = boto3.client('ec2', region_name=region)
          def lambda_handler(event, context):
              ec2.stop_instances(InstanceIds=instances)
              print('stopping your instances: ' + str(instances))

  ProcessingLambdaPermissionStop:
    Type: AWS::Lambda::Permission
    DependsOn:
      - LambdaStop  
    Properties:
      Action: 'lambda:InvokeFunction'
      FunctionName: !Ref LambdaStop
      Principal: s3.amazonaws.com
      SourceArn:
                Fn::Join: 
                  - ''
                  - - 'arn:aws:s3:::'
                    - !Join ["-",[!Ref "S3BucketBaseName",!Ref "AWS::AccountId"]]
      SourceAccount: !Ref AWS::AccountId

  S3KmsKey:
    Type: AWS::KMS::Key
    DependsOn:
      - ProcessingLambdaPermissionStop
    Properties:
      Description: KMS key for trip S3 bucket.
      Enabled: true
      EnableKeyRotation: true
      KeyPolicy:
        Statement:
          - Sid: Administration
            Effect: Allow
            Principal:
              AWS:
                - Fn::Join:
                    - ''
                    - - 'arn:aws:iam::'
                      - Ref: AWS::AccountId
                      - ':role/DW01-codepipeline-action-us-east-1'              
                - Fn::Join:
                    - ''
                    - - 'arn:aws:iam::'
                      - Ref: AWS::AccountId
                      - ':root'
            Action: 'kms:*'
            Resource: '*'

  S3bucketCreate:
    DependsOn:
      - S3KmsKey
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Join ["-",[!Ref "S3BucketBaseName",!Ref "AWS::AccountId"]]
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              KMSMasterKeyID: !Ref S3KmsKey
              SSEAlgorithm: "aws:kms"
      NotificationConfiguration:
        LambdaConfigurations:
          - Event: s3:ObjectCreated:*
            Function: !GetAtt LambdaStart.Arn
            Filter:
              S3Key:
                Rules:
                - Name: prefix
                  Value: input.txt               
          - Event: s3:ObjectCreated:*
            Function: !GetAtt LambdaStop.Arn
            Filter:
              S3Key:
                Rules:
                - Name: prefix
                  Value: output.txt     

  S3bucketPolicy:
    DependsOn:
      - S3bucketCreate        
    Type: AWS::S3::BucketPolicy
    Properties: 
      Bucket: 
        Ref: 'S3bucketCreate'
      PolicyDocument: 
        Statement: 
          - Sid: AllowEc2AccesstoBucket
            Action: 
              - 's3:GetObject'
              - 's3:PutObject'              
            Effect: Allow
            Principal:
              AWS:
                - Fn::Join:          
                  - ''
                  - - 'arn:aws:iam::'
                    - Ref: AWS::AccountId
                    - ':role/DevDW01-EC2-us-east-1'
            Resource: 
                Fn::Join: 
                  - ''
                  - - 'arn:aws:s3:::'
                    - Ref: 'S3bucketCreate'
                    - '/*'              
  LambdaStartStopEC2:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: 
              - lambda.amazonaws.com
            Action: sts:AssumeRole
      RoleName: Lambda-StartStop-EC2
      MaxSessionDuration: 43200
      Policies:
        - PolicyName: StartStop-EC2
          PolicyDocument:
            Statement:
            - Action:
              - s3:*
              Effect: Allow
              Resource: '*'
            - Action:
              - ec2:*
              Effect: Allow
              Resource: '*'
        - PolicyName: logs
          PolicyDocument:
            Statement:
            - Action:
              - logs:CreateLogGroup
              - logs:CreateLogStream
              - logs:DescribeLogGroups
              - logs:DescribeLogStreams
              - logs:PutLogEvents
              - logs:GetLogEvents
              - logs:FilterLogEvents
              Effect: Allow
              Resource: '*'

Outputs:
  S3bucketCreateName:
    Value:
      Ref: S3bucketCreate
    Export:
      Name: S3bucketCreateName
  S3bucketCreateArn:
    Value:
      Fn::GetAtt: S3bucketCreate.Arn
    Export:
      Name: S3bucketCreateArn
  S3KmsKeyArn:
    Value:
      Fn::GetAtt: S3KmsKey.Arn
    Export:
      Name: S3KmsKeyArn

【问题讨论】：

无法正常工作，cloudformation 堆栈创建失败或 lambda 调用未发生？

云形成堆栈失败，因为它两次不允许前缀。模板格式错误：YAML 格式不正确。

这里没有前缀重叠，多条带前缀的规则只要没有重叠就有效。你能发布整个模板吗？

代码很大，我无法在评论部分添加它，因此我在帖子答案部分添加了它。

不，这不是答案，请使用格式正确的完整模板编辑您的问题

【参考方案1】：

允许以prefix 和suffix 作为名称的多个过滤规则，只要它们不重叠。请参阅here 了解各种示例，说明如何发生重叠以及如何避免重叠。

在这种情况下，错误Template format error: YAML not well-formed 可能是由于不正确的 YAML 格式。使用cfn-lint 验证模板。

添加一个显式指定 S3 对象的预期前缀和后缀的 sn-p。

      NotificationConfiguration:
        LambdaConfigurations:
          - Event: s3:ObjectCreated:*
            Function: !GetAtt LambdaStart.Arn
            Filter:
              S3Key:
                Rules:
                - Name: prefix
                  Value: input
                - Name: suffix
                  Value: txt           
          - Event: s3:ObjectCreated:*
            Function: !GetAtt LambdaStop.Arn
            Filter:
              S3Key:
                Rules:
                - Name: prefix
                  Value: output
                - Name: suffix
                  Value: txt

【参考方案2】：

我实际上不得不这样做，创建多个 LambdaConfigurations。

"NotificationConfiguration": 
    "LambdaConfigurations": [
            "Event": "s3:ObjectCreated:*",
            "Function": 
                "Fn::GetAtt": ["lambdaVodFunction", "Arn"]
            ,
            "Filter": 
                "S3Key": 
                    "Rules": [
                        "Name": "suffix",
                        "Value": ".mp4"
                    ]
                
            
        ,
        
            "Event": "s3:ObjectCreated:*",
            "Function": 
                "Fn::GetAtt": ["lambdaVodFunction", "Arn"]
            ,
            "Filter": 
                "S3Key": 
                    "Rules": [
                        "Name": "suffix",
                        "Value": ".mov"
                    ]
                
            
        
    ]