node.js request.js 不并行触发请求

Posted

技术标签:

【中文标题】node.js request.js 不并行触发请求【英文标题】:node.js request.js not firing requests in parallel 【发布时间】:2011-12-20 13:50:20 【问题描述】:

(我还在请求的 github 页面上将此问题发布为问题 - link)

我有一个使用请求从内部 API 获取 API 数据的应用。

我正在做一些测试,看起来我们的应用在重负载下扩展性很差,而且 API 响应越来越慢。

我将此归结为我们正在使用的请求调用,看起来请求可能是在队列中触发这些,而不是一次全部触发。这是显示此行为的测试脚本:

#!/usr/bin/env node
var request = require("request"),
    logger = require("./logger.js"),
    argv = require("optimist").argv;
var numRequests = argv.requests || argv.r,
    requestsMade = 0,
    wait = argv.wait || argv.w || 0,
    url = argv.url || argv.u || "http://www.google.com",
    requestApi = function(url,callback)
        var requestTime = new Date().getTime();
        request(
                method: "GET",
                uri: url
            ,function(err, response, body)
                var totalTime = (new Date().getTime()) - requestTime;
                callback(err, response, body, totalTime);
        );
    ,
    doRequest = function()
        requestsMade++;
        if(requestsMade==numRequests) clearInterval(requestMaker);
        var thisRequest = requestsMade;
        logger.info("Firing Request #"+thisRequest);
        requestApi(url,function(err, response, body, totalTime)
            if(err)
                logger.error("error contacting API ", err, "trying to request ",reqUrl," after ", totalTime, "ms");
             else 
                logger.info("Api responded to request #"+thisRequest+" after ", totalTime, "ms");
            
        );
    ;
logger.info("Starting Test with " + numRequests + " Requests.");
var requestMaker = setInterval(doRequest,wait);

(logger.js 只是一个打印时间戳和设置日志级别的日志工具)。

而且,对 google 的一个简单测试显示了增量放缓:

$ node requestTester.js -r 20
[Wed 2011-11-2 11:2:24.575 GMT7] INF: Starting Test with 20 Requests.
[Wed 2011-11-2 11:2:24.580 GMT7] INF: Firing Request #1
[Wed 2011-11-2 11:2:24.654 GMT7] INF: Firing Request #2
[Wed 2011-11-2 11:2:24.661 GMT7] INF: Firing Request #3
[Wed 2011-11-2 11:2:24.664 GMT7] INF: Firing Request #4
[Wed 2011-11-2 11:2:24.667 GMT7] INF: Firing Request #5
[Wed 2011-11-2 11:2:24.672 GMT7] INF: Firing Request #6
[Wed 2011-11-2 11:2:24.673 GMT7] INF: Firing Request #7
[Wed 2011-11-2 11:2:24.674 GMT7] INF: Firing Request #8
[Wed 2011-11-2 11:2:24.675 GMT7] INF: Firing Request #9
[Wed 2011-11-2 11:2:24.675 GMT7] INF: Firing Request #10
[Wed 2011-11-2 11:2:24.676 GMT7] INF: Firing Request #11
[Wed 2011-11-2 11:2:24.677 GMT7] INF: Firing Request #12
[Wed 2011-11-2 11:2:24.678 GMT7] INF: Firing Request #13
[Wed 2011-11-2 11:2:24.679 GMT7] INF: Firing Request #14
[Wed 2011-11-2 11:2:24.680 GMT7] INF: Firing Request #15
[Wed 2011-11-2 11:2:24.681 GMT7] INF: Firing Request #16
[Wed 2011-11-2 11:2:24.682 GMT7] INF: Firing Request #17
[Wed 2011-11-2 11:2:24.683 GMT7] INF: Firing Request #18
[Wed 2011-11-2 11:2:24.684 GMT7] INF: Firing Request #19
[Wed 2011-11-2 11:2:24.685 GMT7] INF: Firing Request #20
[Wed 2011-11-2 11:2:25.257 GMT7] INF: Api responded to request #2 after  602 ms
[Wed 2011-11-2 11:2:25.621 GMT7] INF: Api responded to request #1 after  1041 ms
[Wed 2011-11-2 11:2:25.774 GMT7] INF: Api responded to request #3 after  1113 ms
[Wed 2011-11-2 11:2:25.779 GMT7] INF: Api responded to request #4 after  1115 ms
[Wed 2011-11-2 11:2:25.895 GMT7] INF: Api responded to request #5 after  1228 ms
[Wed 2011-11-2 11:2:26.378 GMT7] INF: Api responded to request #9 after  1703 ms
[Wed 2011-11-2 11:2:26.714 GMT7] INF: Api responded to request #7 after  2041 ms
[Wed 2011-11-2 11:2:26.870 GMT7] INF: Api responded to request #8 after  2196 ms
[Wed 2011-11-2 11:2:27.126 GMT7] INF: Api responded to request #10 after  2449 ms
[Wed 2011-11-2 11:2:27.267 GMT7] INF: Api responded to request #6 after  2595 ms
[Wed 2011-11-2 11:2:27.730 GMT7] INF: Api responded to request #14 after  3051 ms
[Wed 2011-11-2 11:2:28.68 GMT7] INF: Api responded to request #13 after  3389 ms
[Wed 2011-11-2 11:2:28.72 GMT7] INF: Api responded to request #11 after  3395 ms
[Wed 2011-11-2 11:2:28.75 GMT7] INF: Api responded to request #12 after  3398 ms
[Wed 2011-11-2 11:2:28.332 GMT7] INF: Api responded to request #16 after  3650 ms
[Wed 2011-11-2 11:2:28.471 GMT7] INF: Api responded to request #15 after  3791 ms
[Wed 2011-11-2 11:2:29.45 GMT7] INF: Api responded to request #18 after  4362 ms
[Wed 2011-11-2 11:2:29.161 GMT7] INF: Api responded to request #17 after  4479 ms
[Wed 2011-11-2 11:2:29.173 GMT7] INF: Api responded to request #19 after  4488 ms
[Wed 2011-11-2 11:2:29.424 GMT7] INF: Api responded to request #20 after  4738 ms

为什么会这样?是否有我错过的配置选项会使请求立即全部触发?

我不想放弃请求,因为我非常喜欢它,但这是这个项目的一个亮点。

【问题讨论】:

什么节点版本?应该在 0.5.10 中工作得更好。 当然,这是大公司的生产软件,所以我们只使用稳定版。目前我们在0.4.12,但由于我们没有使用流式传输(0.5+),因此无论节点版本如何,请求都应该提供完全相同的功能。 不,错了。节点核心的 HTTP 客户端行为发生了巨大变化,我认为如果你使用代理,0.4.x 真的很糟糕。因此,快速解决方法是禁用代理并为每个请求建立一个新连接。 啊,我不知道! 【参考方案1】:

如果您不想使用 v0.5.10(它已经接近稳定并且是即将到来的节点 v0.6.x 的候选版本),您将不得不修补请求。在请求 main.js 中,替换行

var globalPool = 

var globalPool = false

这应该会取消对并发连接的限制。

【讨论】:

嗯,所以我试过了,它似乎仍然存在减速问题,唯一的区别是请求现在以更随机的顺序解决。然而并没有解决问题,第一个响应是 478 毫秒,最后一个响应是 2663 毫秒。减速似乎没有那么剧烈,但仍然是 20 个请求的 4 倍,这是不可扩展的。 @Jesse:可能是服务器比较慢? 不,我正在测试 google.com 以确保我不只是对服务器进行负载测试。 @Jesse:你能试试它在 v0.5.10 中是否更好用吗?我不得不承认,我不知道还有什么可能会减慢它的速度。 你改变了maxSockets吗? nodejs.org/docs/v0.5.4/api/http.html#agent.maxSockets

以上是关于node.js request.js 不并行触发请求的主要内容,如果未能解决你的问题,请参考以下文章

Node.js - Async.js:并行执行如何工作?

使用 node.js 运行许多并行 http 请求

使用Chrome DevTools直接调试Node.js与JavaScript(并行)

RethinkDB 和 Node.js/Express - 为并行查询打开多个连接?

Node.js Websockets Socket.IO

为啥 Node.js 不连续监听事件?