从AWS Lambda向SQS队列发送数据时重置连接
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了从AWS Lambda向SQS队列发送数据时重置连接相关的知识,希望对你有一定的参考价值。
我正在使用AWS SDK for Java,我将数据从AWS Lambda发送到SQS。
我们得到例外:
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:115)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:886)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:857)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160)
at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113)
at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:120)
at org.apache.http.entity.StringEntity.writeTo(StringEntity.java:167)
at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:160)
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:63)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
码:
List<SendMessageBatchRequestEntry> sqsList= new LinkedList<SendMessageBatchRequestEntry>();
int batchId = 0; //To send a unique batchId for each msg in a batch
for (Metadata metadata: metadataList) {
String jsonString = new Gson().toJson(metadata);
sqsList.add(new SendMessageBatchRequestEntry(batchId + "", jsonString));
batchId++;
}
amazonSqs.sendMessageBatch(new SendMessageBatchRequest(queueUrl, sqsList));
背景我们正在尝试做什么:
我们有一个主要的Lambda函数,它可以创建和初始化SQS队列,并包含应该处理的每个记录的详细信息。现在,SQS队列需要设置为从队列创建批量的X个消息,并自动为每个批次调用另一个SQS Lambda函数。
答案
每批邮件的最大邮件数为10.您不能一次用20k填充sqs列表并发送该请求。尝试将其分解为10个。
另一答案
好像你的代码很好,据我记得(我自己已经多次看到这个错误),由于SDK重用HTTP连接的方式,这种情况经常发生在使用SDK时。此错误仅告诉您Lambda,重新安装了HTTP连接,但SDK已经内置了重试失败请求的功能,因此如果您没有在每个请求上看到此错误,那么您应该没问题。
另一答案
我们可以批量大小为10发送它。工作代码:
List<SendMessageBatchRequestEntry> sqsList= new LinkedList<SendMessageBatchRequestEntry>();
int batchId = 1; //To send a unique batchId for each msg in a batch
for (Metadata metadata: metadataList) {
String jsonString = new Gson().toJson(metadata);
if (sqsList.size() == 10) {
amazonSqs.sendMessageBatch(new SendMessageBatchRequest(queueUrl, sqsList));
sqsList.clear();
}
sqsList.add(new SendMessageBatchRequestEntry(batchId + "", jsonString));
batchId++;
}
if(sqsList.size()>0) {
amazonSqs.sendMessageBatch(new SendMessageBatchRequest(queueUrl, sqsList));
}
以上是关于从AWS Lambda向SQS队列发送数据时重置连接的主要内容,如果未能解决你的问题,请参考以下文章
如何从外部 SQS 队列活动触发 AWS Lambda 函数