Cheerio 内存泄漏

Posted 2023-02-19

【中文标题】Cheerio 内存泄漏

【英文标题】：Cheerio memory leak

【发布时间】：2020-01-30 02:05:16

【问题描述】：

我正在使用cheerio 来解析otodom.pl advs 的结果。我遇到了关于记忆的问题。

节点 8.10.0 和cheerio@1.0.0-rc.3

4GB RAM、4GB SWAP、1 个核心、Ubuntu 18.04.3 LTS（GNU/Linux 4.15.0-64-generic x86_64）

let $ = cheerio.load(context.body.toString());

<--- Last few GCs --->

[10154:0x5607b1d35140]   640026 ms: Mark-sweep 1395.4 (1433.4) -> 1395.3 (1433.9) MB, 2037.1 / 0.0 ms  allocation failure GC in old space requested
[10154:0x5607b1d35140]   641229 ms: Mark-sweep 1395.3 (1433.9) -> 1395.3 (1426.9) MB, 1202.9 / 0.0 ms  last resort GC in old space requested
[10154:0x5607b1d35140]   642829 ms: Mark-sweep 1395.3 (1426.9) -> 1395.3 (1426.9) MB, 1599.9 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x26de93518fe1 <JSObject>
    1: toString [buffer.js:~609] [pc=0x4ff3cc3e224](this=0xa4360294ad1 <Uint8Array map = 0x100a82c5b801>,encoding=0x26de93502241 <undefined>,start=0x26de93502241 <undefined>,end=0x26de93502241 <undefined>)
    2: arguments adaptor frame: 0->3
    3: /* anonymous */ [/home/new/crawler.js:152] [bytecode=0x3a0e68d159b1 offset=70](this=0x2209454880a9 <JSGlobal Object>,context=0xa4360294bf1 <Object m...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - javascript heap out of memory
 1: node::Abort() [node]
 2: 0x5607b06b5011 [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [node]
 6: v8::internal::Factory::NewStringFromUtf8(v8::internal::Vector<char const>, v8::internal::PretenureFlag) [node]
 7: v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [node]
 8: node::StringBytes::Encode(v8::Isolate*, char const*, unsigned long, node::encoding, v8::Local<v8::Value>*) [node]
 9: 0x5607b06d793b [node]
10: 0x4ff3c9f6e47
Aborted (core dumped)

【参考方案1】：

对我来说，这是一个发回 522 网络错误的 url。我正在遍历 30 个 url，所以我一次删除一个 url 以确定原因。