如何从远程执行的 Fargate 任务访问 S3 对象？

Posted 2023-03-04

【中文标题】如何从远程执行的 Fargate 任务访问 S3 对象？

【英文标题】：How to access an S3 object from a remotely executed fargate task?

【发布时间】：2021-07-03 12:05:54

【问题描述】：

我编写了一个从 S3 存储桶中读取的 java 程序，该程序在 AWS 上的 Fargate 任务中运行，但由于 AccessDeniedException 而失败 该程序位于使用ECS Exec 远程执行的 jar 文件中 Fargate 任务的任务角色具有以下权限，但以下行仍然失败，并出现以下异常

    "Version": "2012-10-17",
    "Statement": [
        
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::*"
        ,
        
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectacl",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::*"
        
    ]


ListObjectsV2Request req = new ListObjectsV2Request().withBucketName("myBucket").withMaxKeys(200);
ListObjectsV2Result result = amazonS3.listObjectsV2(req);

com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 0Z58XY0CF91Q6H4Y; S3 Extended Request ID: sVoHHMEM4+5ti+MlcaHsgvJxyvbFlJrMVZMC4cBOFI1hPg/QtsKExqFiqNECH2ZKoUXuJMYwAt0=), S3 Extended Request ID: sVoHHMEM4+5ti+MlcaHsgvJxyvbFlJrMVZMC4cBOFI1hPg/QtsKExqFiqNECH2ZKoUXuJMYwAt0=
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1395)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1371)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4914)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4860)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4854)
    at com.amazonaws.services.s3.AmazonS3Client.listObjectsV2(AmazonS3Client.java:923)
    at com.mycompany.ReadS3Object.getFilesOnS3(ReadS3Object.java:263)
    at com.mycompany.ReadS3Object.run(ReadS3Object.java:141)
    at picocli.CommandLine.executeUserObject(CommandLine.java:1939)
    at picocli.CommandLine.access$1300(CommandLine.java:145)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
    at picocli.CommandLine.execute(CommandLine.java:2078)
    at com.mycompany.ReadS3Object.main(ReadS3Object.java:132)

需要进行哪些更改才能完成listObjectsV2 调用？

编辑

所以问题是代码试图列出的存储桶不存在！当我在本地运行代码时，它给出了正确的 bucket not found 异常，但远程运行它给出了 AccessDeniedException。创建存储桶解决了问题

【参考方案1】：

您的 S3 V1 代码似乎缺少引用有权调用 S3 操作的 IAM 用户的凭据。作为测试，请为该角色提供完整的 S3 访问权限，并查看此问题是否仍然存在。

此外，作为一般建议，您应该考虑放弃旧 V1 Amazon S3 API 并考虑使用 V2 - 这是 Amazon 在编写 Java 代码以调用 Amazon S3 操作时推荐使用的版本。

当您看到像这样使用 new 运算符时：

ListObjectsV2Request req = new ListObjectsV2Request().withBucketName("myBucket").withMaxKeys(200);

这是 V1 代码。 V2 代码不使用 new 运算符来创建对象，而是使用 builder()：

ListObjectsRequest listObjects = ListObjectsRequest.builder()
                    .bucket(bucketName)
                    .build();

【讨论】：

谢谢，我已将角色的权限添加到问题中

另外，感谢您对 V2 代码的建议，但现在我仅限于 V1 库