从AWS Lambda向SQS队列发送数据时重置连接

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了从AWS Lambda向SQS队列发送数据时重置连接相关的知识,希望对你有一定的参考价值。

我正在使用AWS SDK for Java,我将数据从AWS Lambda发送到SQS。

我们得到例外:

Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:115)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:886)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:857)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160)
at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113)
at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:120)
at org.apache.http.entity.StringEntity.writeTo(StringEntity.java:167)
at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:160)
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:63)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)

码:

 List<SendMessageBatchRequestEntry> sqsList= new LinkedList<SendMessageBatchRequestEntry>();
    int batchId = 0; //To send a unique batchId for each msg in a batch
    for (Metadata metadata: metadataList) {
        String jsonString = new Gson().toJson(metadata);
        sqsList.add(new SendMessageBatchRequestEntry(batchId + "", jsonString));
        batchId++;
    }
    amazonSqs.sendMessageBatch(new SendMessageBatchRequest(queueUrl, sqsList));

背景我们正在尝试做什么:

我们有一个主要的Lambda函数,它可以创建和初始化SQS队列,并包含应该处理的每个记录的详细信息。现在,SQS队列需要设置为从队列创建批量的X个消息,并自动为每个批次调用另一个SQS Lambda函数。

答案

每批邮件的最大邮件数为10.您不能一次用20k填充sqs列表并发送该请求。尝试将其分解为10个。

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-limits.html#limits-queues

另一答案

好像你的代码很好,据我记得(我自己已经多次看到这个错误),由于SDK重用HTTP连接的方式,这种情况经常发生在使用SDK时。此错误仅告诉您Lambda,重新安装了HTTP连接,但SDK已经内置了重试失败请求的功能,因此如果您没有在每个请求上看到此错误,那么您应该没问题。

另一答案

我们可以批量大小为10发送它。工作代码:

List<SendMessageBatchRequestEntry> sqsList= new LinkedList<SendMessageBatchRequestEntry>();
    int batchId = 1; //To send a unique batchId for each msg in a batch
    for (Metadata metadata: metadataList) {
        String jsonString = new Gson().toJson(metadata);
        if (sqsList.size() == 10) {
            amazonSqs.sendMessageBatch(new SendMessageBatchRequest(queueUrl, sqsList));
            sqsList.clear();
        } 
        sqsList.add(new SendMessageBatchRequestEntry(batchId + "", jsonString)); 
        batchId++;
    }
    if(sqsList.size()>0) {
        amazonSqs.sendMessageBatch(new SendMessageBatchRequest(queueUrl, sqsList));
    }

以上是关于从AWS Lambda向SQS队列发送数据时重置连接的主要内容,如果未能解决你的问题,请参考以下文章

如何从外部 SQS 队列活动触发 AWS Lambda 函数

消息发布到 SQS 时如何触发 lambda?

当消息存在于 SQS 队列中时触发 AWS 中的 Lambda 函数

通过 SQS lambda 发送 SQS 消息

AWS Lambda 在向 SQS 发送消息之前完成

使用 AWS 无服务器和 NodeJS 在接收器 lambda 处未接收到来自 SQS 的所有消息