Bug解决pymongo.errors.OperationFailure: Executor error during find command :: caused by :: Sort exce

Posted 2023-03-10 zstar-_

问题场景

使用pymongo时，这行代码触发如下报错：

pymongo.errors.OperationFailure: Executor error during find command :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in., full error: ‘ok’: 0.0, ‘errmsg’: ‘Executor error during find command :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.’, ‘code’: 292, ‘codeName’: ‘QueryExceededMemoryLimitNoDiskUseAllowed’

触发代码：

sorted_mongo_doc = mongo.find().sort('Time', -1).limit(2000)

原因

查阅mongo中文文档(https://mongodb.net.cn/manual/reference/method/cursor.sort/#sort-limit-results)，得知此报错的原因是排序内容超越32MB的内存限制。

文档解释：当无法从索引获取排序顺序时，MongoDB将对内存中的结果进行排序，这要求所排序的结果集小于32 MB。

解决方法

找到如下解决方法，参考：https://segmentfault.com/a/1190000040980419

1. 设置排序字段索引

db.getCollection('col').createIndex("updatedAt": 1)

没做尝试，略

2. 提高数据库排序的内存上限

db.adminCommand(setParameter: 1, internalQueryExecMaxBlockingSortBytes: 104857600)

在pymongo中尝试失败

3.更换 aggregate 进行查询
将查询语句替换如下：

sorted_mongo_doc = mongo.aggregate([
    '$project': ,
    '$sort': 'Time': -1,
    '$limit': 2000,
], allowDiskUse=True)

allowDiskUse=True表示使用磁盘存储

该方法运行成功，问题解决。