使用s3 java sdk 分片文件上传API 报‘SignatureDoesNotMatch’ 异常的定位及解决方案

Posted Freewill

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了使用s3 java sdk 分片文件上传API 报‘SignatureDoesNotMatch’ 异常的定位及解决方案相关的知识,希望对你有一定的参考价值。

import java.io.File;
 
import com.amazonaws.AmazonClientException;

import com.amazonaws.auth.profile.ProfileCredentialsProvider;

import com.amazonaws.services.s3.transfer.TransferManager;

import com.amazonaws.services.s3.transfer.Upload;

 

public class UploadObjectMultipartUploadUsingHighLevelAPI {

 
    public static void main(String[] args) throws Exception {

        String existingBucketName = "*** Provide existing bucket name ***";

        String keyName            = "*** Provide object key ***";

        String filePath           = "*** Path to and name of the file to upload ***"

         

        TransferManager tm = new TransferManager(new ProfileCredentialsProvider());       

        System.out.println("Hello");

        // TransferManager processes all transfers asynchronously,

        // so this call will return immediately.

        Upload upload = tm.upload(

                existingBucketName, keyName, new File(filePath));

        System.out.println("Hello2");

 

        try {

            // Or you can block and wait for the upload to finish

            upload.waitForCompletion();

            System.out.println("Upload complete.");

        catch (AmazonClientException amazonClientException) {

            System.out.println("Unable to upload file, upload was aborted.");

            amazonClientException.printStackTrace();

        }

    }

}
 

问题描述:使用s3 java sdk 的如上代码分片文件上传,报‘SignatureDoesNotMatch’ 异常如下

Unable to abort multipart upload, you may need to manually remove uploaded parts: null (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx000000000000000003ddf-005853b76e-1a56f9b-default)
com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx000000000000000003ddf-005853b76e-1a56f9b-default), S3 Extended Request ID: 1a56f9b-default-default

 

 

 

定位过程:

下载s3 java sdk 源码,并配置好 log4j, 打印aws debug日志; 

# For JBoss: Avoid to setup Log4J outside $JBOSS_HOME/server/default/deploy/log4j.xml!
# For all other servers: Comment out the Log4J listener in web.xml to activate Log4J.
#log4j.rootLogger=INFO, stdout, logfile
log4j.rootLogger=INFO,stdout,logfile
log4j.logger.com.amazonaws = DEBUG
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - <%m>%n

RGW配置文件/etc/ceph/ceph.conf 中的日志级别调整为20,重启RGW进程

debug rgw = 20

 

执行分片上传代码,分别截获客户端和服务端的请求日志。 如下2个图片可以发现uploadId客户端和服务端不一致,导致服务端进行签名校验失败。

 

 

 

  

那么问题来了,到底是RGW服务端,还是java s3 sdk实现有bug呢? 通过查看java 代码和aws文档描述,我认为是RGW签名实现存在bug。 RGW 应该在

获取从客户端url中的参数中值(已经被URLEncode)后,先decode,再根据aws文档描述的步骤进行encode(~特殊字符不encode),得到符合规则的参数字符串。 

.\\src\\rgw\\rgw_rest_s3.cc

 

http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html

 

 

规避方案:

 

因为修改RGW的代码周期比较长,暂时先修改好修改的java代码,编译一个新的jar包。 如下是修改方法,不按照aws的规则来,对参数值只执行URLencode。

提交了一个bugfix到rgw git库, https://github.com/ceph/ceph/pull/12647/commits

 

 https://github.com/BodihTao/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/auth/AbstractAWSSigner.java

 

为了方便大家,我上传修改后编译好的class文件(java 1.8版本),把这2个文件用winrar 放到aws-java-sdk-1.11.69.jar包中com/amazonaws/auth目录下 替换原来文件即可得到新的jar包。

https://github.com/BodihTao/aws-sdk-java/raw/master/aws-java-sdk/AbstractAWSSigner%241.class

https://github.com/BodihTao/aws-sdk-java/raw/master/aws-java-sdk/AbstractAWSSigner.class

 

java 1.7版本的sdk包:

http://s3.yyclouds.com/public/aws-java-sdk-1.11.69rgw.jar

 

以上是关于使用s3 java sdk 分片文件上传API 报‘SignatureDoesNotMatch’ 异常的定位及解决方案的主要内容,如果未能解决你的问题,请参考以下文章

如何使用 aws-sdk 将文件同步上传到 S3?

如何通过 aws Java SDK 公开 S3 对象?

plupload 大文件分片上传与PHP分片合并探索

Javascript AWS SDK S3上传方法与正文流生成空文件

如何在不使用 SDK 的情况下将文件从 Android 上传到 Amazon S3

使用 amazon sdk 和存储桶策略从 ios 应用程序将图像上传到 amazon s3