AmazonClientException：读取的数据长度与预期不同

Posted 2023-04-14

【中文标题】AmazonClientException：读取的数据长度与预期不同

【英文标题】：AmazonClientException: Data read has a different length than the expected

【发布时间】：2015-03-04 03:06:55

【问题描述】：

1.当我使用 AmazonS3Client 在 amazon s3 文件存储上上传文件时。 2.当我尝试一次上传多个文件时，它会给出异常：但同一个文件有多个线程。 我尝试了客户端配置，例如： 1.connectionTimeout=50000 毫秒 2.maxConnections=500 3.socketTimeout=50000 毫秒

Exception stacktrace:
com.amazonaws.AmazonClientException: Data read has a different length than the expected: dataLength=8192; expectedLength=79352; includeSkipped=false; in.getClass()=class com.amazonaws.internal.ResettableInputStream; markedSupported=true; marked=0; resetSinceLastMarked=false; markCount=1; resetCount=0
                    at com.amazonaws.util.LengthCheckInputStream.checkLength(LengthCheckInputStream.java:150)
                    at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:110)
                    at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73)
                    at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151)
                    at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73)
                    at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:98)
                    at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:153)
                    at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
                    at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
                    at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
                    at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
                    at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:197)
                    at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
                    at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:47)
                    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)

                at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:713)
                    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518)
                    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
                    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
                    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:647)
                    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
                    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
                    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3655)
                    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1424)
                    at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:135)
                    at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:127)
                    at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:129)
                    at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:50)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:745)**

【问题讨论】：

我找到了解决这个问题的方法：因为我试图多次发送同一个文件。这就是它给出错误的原因：因为 AmazonS3Client 多部分发送文件。

【参考方案1】：

这个答案是 AWS Hanson 的人写的：

是否可能请求中指定的输入流已经被完全读取？

如果输入流是文件流，是否尝试过在请求中指定原始文件而不是文件的输入流？

【参考方案2】：

用示例代码改进@iucasddaniel 的回答。

AmazonS3Client putObject：没有为流数据指定内容长度。 流内容将缓冲在内存中，并可能导致 内存错误。

解决方案 « 指定 Object Metadata content Length

File tempFile = "D://Test.mp4";
String bucketName = "YashFiles", filePath = "local/mp4/";

FileInputStream sampleStream = new FileInputStream( tempFile );
byte[] byteArray = IOUtils.toByteArray( sampleStream );
Long contentLength = Long.valueOf(byteArray.length);
sampleStream.close();

ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(contentLength);

TransferManager tm = new TransferManager(credentials);

FileInputStream stream = new FileInputStream( tempFile );
PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, filePath, stream,objectMetadata);
Upload myUpload = tm.upload(putObjectRequest);
    if (myUpload.isDone() == false) 
        System.out.println("Transfer: "+ myUpload.getDescription());
        System.out.println("  - State: "+ myUpload.getState());
        System.out.println("  - Progress: "+ myUpload.getProgress().getBytesTransferred());
    
myUpload.waitForCompletion();

tm.shutdownNow();
stream.close();

org.apache.commons.io.FileUtils.forceDelete( tempFile );

---

Amazon S3：检查Key Exists and generating PresignedUrl

【讨论】：

这并不能解决问题。您仍然使用 IOUtils.toByteArray() 将整个文件加载到内存中。这只是将其移动到您的代码中，将整个文件加载到内存中，而不是亚马逊的代码中。

【参考方案3】：

我在尝试执行 S3.putObject(MyObject); 时看到了该错误消息

我必须更新 objectMetadata.setContentLength( [你的内容的长度] );

例如：

String dataset= "Some value you want to add to S3 Bucket"; 
ObjectMetadata objectMetadata= new ObjectMetadata(); 
InputStream content= new ByteArrayInputStream(dataset.getBytes("UTF-8"));
objectMetadata.setContentLength(content.available()); 
objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYTION); 

【讨论】：

这并不能解决问题。您仍然必须将整个数据集加载到内存中才能获得长度。这只是将其移动到您的代码中，将整个数据集加载到内存中，而不是亚马逊的代码中。

我不关注。不是所有的对象在实例化的时候都加载到内存中了吗？

是的，这就是我的观点。这就是为什么您不能在对象中加载数据的原因。如果您将它作为输入或输出流传输，它只是通过而不被加载。如果将数据加载到对象中，则整个文件将加载到内存中。

您在调试器中看到了什么？我不确定为什么您的评论与我的回答错误有关。如果有错，我会删除它。您能否通过电子邮件将您所看到的内容或有关您正在谈论的内容的参考资料发给我？我的邮箱是 [e-mail]

【参考方案4】：

...
byte[] f = IOUtils.toByteArray(inputStream); // This reads all bytes of the input stream
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(f.length);
metadata.setContentType(contentType); //Content type of the uploaded file
metadata.setHeader("filename", fileName);
s3.putObject(bucketName, key, new ByteArrayInputStream(f), metadata); // Here we create a new ByteArrayInputStream so S3 client get happy.