使用 mongodb 的 CPU 几乎达到 100%
Posted
技术标签:
【中文标题】使用 mongodb 的 CPU 几乎达到 100%【英文标题】：CPU hits almost 100% with mogodb 【发布时间】：2018-02-12 11:57:40 【问题描述】：我正在为 mongodb 使用带有节点 js 的 mongoose 库。
我的页面包含大约 20 个图表。所有图表都与 mongo db 中的同一集合相关。
我必须为同一个集合上的图表调用几乎 15-16 个服务。
当我使用 MongoDB 数据库服务器执行此操作时，cpu 几乎达到 100%。
谁能建议如何优化它。
以下是 mongodb 查询日志。
2017-09-01T23:57:25.164+0000 I COMMAND [conn4] command HyperlocalPortalDB.users command: aggregate aggregate: “coordinates”,
pipeline: [ $match: activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’), t: $gte: new Date(1503810000000), $lt: new Date(1504069140000) ,
$group: _id: mac: “$mac” ,
$lookup: from: “users”, localField: “_id.mac”, foreignField: “mac”, as: “userDetail” ,
$match: userDetail.activations.activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’) ,
$unwind: “$userDetail” , $project: mac: “$_id.mac”, groupField: “$userDetail.city” ,
$group: _id: “$groupField”, count: $sum: 1 ,
$sort: count: -1 ]
planSummary: COLLSCAN keysExamined:0 docsExamined:1373500 hasSortStage:1 numYields:10810 nreturned:0 reslen:18599
locks: Global: acquireCount: r: 26198 , Database: acquireCount: r: 13099 ,
Collection: acquireCount: r: 13098 protocol:op_query 7755ms
2017-09-01T23:57:25.264+0000 I COMMAND [conn3] command HyperlocalPortalDB.users command: aggregate aggregate: “coordinates”,
pipeline: [ $match: activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’), t: $gte: new Date(1503810000000), $lt: new Date(1504069140000) ,
$group: _id: mac: “$mac” ,
$lookup: from: “users”, localField: “_id.mac”, foreignField: “mac”, as: “userDetail” ,
$match: userDetail.activations.activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’) ,
$unwind: “$userDetail” , $project: mac: “$_id.mac”, groupField: “$userDetail.country” ,
$group: _id: “$groupField”, count: $sum: 1 , $sort: count: -1 ]
planSummary: COLLSCAN keysExamined:0 docsExamined:1373500 hasSortStage:1 numYields:10831 nreturned:0 reslen:82
locks: Global: acquireCount: r: 26240 , Database: acquireCount: r: 13120 ,
Collection: acquireCount: r: 13119 protocol:op_query 7776ms
2017-09-01T23:57:25.486+0000 I COMMAND [conn5] command HyperlocalPortalDB.users command: aggregate aggregate: “coordinates”,
pipeline: [ $match: activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’), t: $gte: new Date(1503810000000), $lt: new Date(1504069140000) ,
$group: _id: mac: “$mac” ,
$lookup: from: “users”, localField: “_id.mac”, foreignField: “mac”, as: “userDetail” ,
$match: userDetail.activations.activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’) ,
$unwind: “$userDetail” , $project: mac: “$_id.mac”, groupField: “$userDetail.state” ,
$group: _id: “$groupField”, count: $sum: 1 , $sort: count: -1 ]
planSummary: COLLSCAN keysExamined:0 docsExamined:1373500 hasSortStage:1 numYields:10806 nreturned:0 reslen:2053
locks: Global: acquireCount: r: 26190 , Database: acquireCount: r: 13095 ,
Collection: acquireCount: r: 13094 protocol:op_query 7336ms
2017-09-01T23:57:25.564+0000 I COMMAND [conn6] command HyperlocalPortalDB.users command: aggregate aggregate: “coordinates”,
pipeline: [ $match: activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’), t: $gte: new Date(1503810000000), $lt: new Date(1504069140000) ,
$group: _id: mac: “$mac” ,
$lookup: from: “users”, localField: “_id.mac”, foreignField: “mac”, as: “userDetail” ,
$match: userDetail.activations.activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’) ,
$unwind: “$userDetail” , $project: mac: “$_id.mac”, groupField: “$userDetail.state” ,
$group: _id: “$groupField”, count: $sum: 1 , $sort: count: -1 ]
planSummary: COLLSCAN keysExamined:0 docsExamined:1373500 hasSortStage:1 numYields:10811 nreturned:0 reslen:2053
locks: Global: acquireCount: r: 26200 , Database: acquireCount: r: 13100 ,
Collection: acquireCount: r: 13099 protocol:op_query 8259ms
2017-09-01T23:57:26.018+0000 I COMMAND [conn2] command HyperlocalPortalDB.users command: aggregate aggregate: “coordinates”,
pipeline: [ $match: activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’), t: $gte: new Date(1503810000000), $lt: new Date(1504069140000) ,
$group: _id: mac: “$mac” , $lookup: from: “users”, localField: “_id.mac”, foreignField: “mac”, as: “userDetail” ,
$match: userDetail.activations.activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’) ,
$unwind: “$userDetail” , $project: mac: “$_id.mac”, groupField: “$userDetail.city” ,
$group: _id: “$groupField”, count: $sum: 1 , $sort: count: -1 ]
planSummary: COLLSCAN keysExamined:0 docsExamined:1373500 hasSortStage:1 numYields:10804 nreturned:0 reslen:18599
locks: Global: acquireCount: r: 26186 , Database: acquireCount: r: 13093 ,
Collection: acquireCount: r: 13092 protocol:op_query 7978ms
2017-09-01T23:57:29.030+0000 I COMMAND [conn5] command HyperlocalPortalDB.coordinates command: aggregate aggregate: “coordinates”,
pipeline: [ $match: activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’), t: $gte: new Date(1503810000000), $lt: new Date(1503896340000) ,
$group: _id: “$mac”, minTime: $min: “$t” , maxTime: $max: “$t” ]
planSummary: COLLSCAN keysExamined:0 docsExamined:1373500 numYields:10751 nreturned:0 reslen:26423
locks: Global: acquireCount: r: 21538 , Database: acquireCount: r: 10769 ,
Collection: acquireCount: r: 10768 protocol:op_query 3537ms
2017-09-01T23:57:29.812+0000 I COMMAND [conn3] command HyperlocalPortalDB.coordinates command: aggregate aggregate: “coordinates”,
pipeline: [ $match: activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’), t: $gte: new Date(1503982800000), $lt: new Date(1504069140000) ,
$group: _id: “$mac”, minTime: $min: “$t” , maxTime: $max: “$t” ]
planSummary: COLLSCAN keysExamined:0 docsExamined:1373500 numYields:10755 nreturned:0 reslen:25216
locks: Global: acquireCount: r: 21540 , Database: acquireCount: r: 10770 ,
Collection: acquireCount: r: 10769 protocol:op_query 4540ms
2017-09-01T23:57:30.641+0000 I COMMAND [conn6] command HyperlocalPortalDB.coordinates command: aggregate aggregate: “coordinates”,
pipeline: [ $match: activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’), t: $gte: new Date(1503896400000), $lt: new Date(1503982740000) ,
$group: _id: “$mac”, minTime: $min: “$t” , maxTime: $max: “$t” ]
planSummary: COLLSCAN keysExamined:0 docsExamined:1373500 numYields:10762 nreturned:0 reslen:25145
locks: Global: acquireCount: r: 21558 , Database: acquireCount: r: 10779 ,
Collection: acquireCount: r: 10778 protocol:op_query 5077ms
2017-09-01T23:57:31.664+0000 I COMMAND [conn2] command HyperlocalPortalDB.users command: aggregate aggregate: “coordinates”,
pipeline: [ $match: activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’), t: $gte: new Date(1503810000000), $lt: new Date(1504069140000) ,
$group: _id: mac: “$mac” , $lookup: from: “users”, localField: “_id.mac”, foreignField: “mac”, as: “userDetail” ,
$match: userDetail.activations.activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’) , $unwind: “$userDetail” ,
$project: mac: “$_id.mac”, groupField: “$userDetail.country” , $group: _id: “$groupField”, count: $sum: 1 ,
$sort: count: -1 ] planSummary: COLLSCAN keysExamined:0 docsExamined:1373500 hasSortStage:1 numYields:10774 nreturned:0 reslen:82
locks: Global: acquireCount: r: 26126 , Database: acquireCount: r: 13063 ,
Collection: acquireCount: r: 13062 protocol:op_query 5638ms
2017-09-01T23:57:33.839+0000 I COMMAND [conn5] warning: log line attempted (89kB) over max size (10kB), printing beginning and end ... command HyperlocalPortalDB.coordinates command: aggregate aggregate: “coordinates”,
pipeline: [ $match: activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’), t: $gte: new Date(1503810000000), $lt: new Date(1503896340000) , mac: $in: [ “41:9e:87:01:00:00", “41:9E:87:01:00:00”, “ab:cd:ed”, “41:9e:87:01:09:64”, “41:9E:87:01:09:64" ] ,
$group: _id: $dateToString: format: “%Y-%m-%d %H”, date: $subtract: [ “$t”, 14400000 ] ,
macList: $addToSet: “$mac” , $sort: _id: 1 ]
planSummary: COLLSCAN keysExamined:0 docsExamined:1373500 hasSortStage:1 numYields:10756 nreturned:0 reslen:80098
locks: Global: acquireCount: r: 21548 , Database: acquireCount: r: 10774 ,
Collection: acquireCount: r: 10773 protocol:op_query 4433ms
2017-09-01T23:57:34.270+0000 I COMMAND [conn6] warning: log line attempted (89kB) over max size (10kB), printing beginning and end ... command HyperlocalPortalDB.coordinates command: aggregate aggregate: “coordinates”, pipeline: [ $match: activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’), t: $gte: new Date(1503982800000), $lt: new Date(1504069140000) , mac: $in: [ “41:9e:87:01:00:00”, “41:9E:87:01:00:00", “ab:cd:ed”, “AB:CD:ED”, “41:9e:87:01:09:64", “41:9E:87:01:09:64” ] ,
$group: _id: $dateToString: format: “%Y-%m-%d %H”, date: $subtract: [ “$t”, 14400000 ] ,
macList: $addToSet: “$mac” , $sort: _id: 1 ]
planSummary: COLLSCAN keysExamined:0 docsExamined:1373500 hasSortStage:1 numYields:10749 nreturned:0 reslen:57332
locks: Global: acquireCount: r: 21528 , Database: acquireCount: r: 10764 ,
Collection: acquireCount: r: 10763 protocol:op_query 3619ms
2017-09-01T23:57:34.376+0000 I COMMAND [conn4] warning: log line attempted (89kB) over max size (10kB), printing beginning and end ... command HyperlocalPortalDB.coordinates command: aggregate aggregate: “coordinates”, pipeline: [ $match: activationId: ObjectId(‘59a6e4dad95d240cf4ba37de’), t: $gte: new Date(1503896400000), $lt: new Date(1503982740000) , mac: $in: [ “41:9e:87:01:00:00", “41:9E:87:01:00:00”, “ab:cd:ed”, “AB:CD:ED”, “41:9e:87:01:09:64”, “41:9E:87:01:09:64" ] ,
$group: _id: $dateToString: format: “%Y-%m-%d %H”, date: $subtract: [ “$t”, 14400000 ] , macList: $addToSet: “$mac” ,
$sort: _id: 1 ]
planSummary: COLLSCAN keysExamined:0 docsExamined:1373500 hasSortStage:1 numYields:10758 nreturned:0 reslen:76210
locks: Global: acquireCount: r: 21550 , Database: acquireCount: r: 10775 ,
Collection: acquireCount: r: 10774 protocol:op_query 4973ms
【问题讨论】：
你在activationId字段上有索引吗？
我删除了很长的评论，因为列出的问题太多了。如何显示您拥有的数据以及您需要的结果。还要首先解释如何获取数据。您可能应该预先汇总您提出大量请求的任何数据。
@felix COLLSCAN 在慢查询日志条目中说 NO。不过，索引使用只是记录的管道中明显存在的众多问题之一。
@NeilLunn，我有预先汇总的数据，这给了我大量请求。在每个查询中，您都可以看到我在集合上使用了 match 语句，然后按字段分组以避免获取大量数据。大多数查询首先匹配坐标集合，然后按 mac 分组，然后在该组之后按用户集合的特定字段加入用户集合。
老兄。您正在“加入”，这永远不会有效。这只是一个权宜之计。如果您需要性能，那么所有数据都以最小形式存在于一个集合中。这里的操作远非最佳。因此建议改为解释数据收集和当前可用数据和预期结果。我可以写一整页关于您当前管道中的问题，而不必尽一切可能实际提高性能。所以它可能是学术性的，但它不会解决一般问题。
【参考方案1】：
1 - 禁用控制台显示
2 - 应用 mongodb productions notes.
例如：
对于 WiredTiger 存储引擎，强烈建议使用 XFS 以避免将 EXT4 与 WiredTiger 一起使用时可能出现的性能问题。
禁用透明大页面。 MongoDB 在普通（4096 字节）虚拟内存页面上表现更好。
【讨论】：
我用过mongodb 3.4，它使用WiredTiger存储引擎。如何禁用透明大页面？ 您必须查看 mongodb 文档。 Hugepage section以上是关于使用 mongodb 的 CPU 几乎达到 100%的主要内容，如果未能解决你的问题，请参考以下文章
MongoDB服务器CPU一直很高，最高达到900%，可能是哪些原因？
我电脑CPU的使用率突然达到100%静止不动，程序运行缓慢这是为啥？