是否有任何有效的方法可以以最新的一阶对给定的 mongodb 数据进行分页并避免排序超出大数据的内存限制？

Posted 2023-03-11

技术标签:

【中文标题】是否有任何有效的方法可以以最新的一阶对给定的 mongodb 数据进行分页并避免排序超出大数据的内存限制？【英文标题】：Is there any efficient way to paginate given mongodb data in latest first order and avoid sort exceed memory limit for huge data? 【发布时间】：2021-06-23 18:53:52 【问题描述】：

我给出了如下示例数据收集。我正在尝试以最新的第一顺序获得结果，因此我使用了分页并分别对其进行了排序。但是当我试图对大量数据进行排序时，比如 API 命中限制 100 。然后它向我抛出错误

"message": "find command:: 期间执行器错误由:: Sort 引起超过 33554432 字节的内存限制，但未选择加入外部排序。中止操作。通过 allowDiskUse:true 来选择在。”


  "success": true,
  "message": "",
  "status": ,
  "data": 
    "docs": [
 
        "_id": "5d1dbe9cad2de907c4a6c888",
        "requestedBy": 
          "adminId": "1",
          "adminName": "JJ"
        ,
        "reason": "Testing",
        "payload": [
          
            "recipient": 
              "to": "abc@abc.com",
              "from": "abc@abc.com",
              "bcc": "abc@abc.com"
            ,
            "data": 
              "b_name": "Cloud",
              "amount": 100,
              "l_id": 10,
              "date": "10",
              "month": "Jul",
              "year": "2019"
            ,
            "template": 
              "name": "Reminder",
              "type": "EMAIL"
            
          
],
        "createdAt": "2019-07-04T08:53:48.129Z",
        "updatedAt": "2019-07-04T08:53:48.129Z",
        "__v": 0,
        "id": "5d1dbe9cad2de907c4a6c888"
      
    ]

const mongoose = require("mongoose");
const mongoosePaginate = require("mongoose-paginate-v2");

mongoosePaginate.paginate.options = 
  lean: true,
  limit: 200,
  sort:
    createdAt: -1
  
;
const MODEL_NAME = "data";

const dSchema = mongoose.Schema(
  
    requestedBy: 
      adminId: String,
      adminName: String
    ,
    reason: String,
    payload: mongoose.Mixed
  ,
   timestamps: true 
);

// Plugins
dSchema.plugin(mongoosePaginate);

// Methods to interact with models
dSchema.statics.getTasks = (query, options) => 
  return mongoose.model(MODEL_NAME).paginate(query, options);
;

【问题讨论】：

【参考方案1】：

https://mongoosejs.com/docs/guide.html#indexes

https://mongoosejs.com/docs/guide.html#timestamps

https://docs.mongodb.com/manual/tutorial/optimize-query-performance-with-indexes-and-projections/