Mongodb aggregation 基本操作示例

Posted 2020-10-15 hapjin

MongoDB二个主要的操作：一个是查询，另一个是统计。对于查询而言，主要是find()方法，再配合Filters组合多个查询条件。

对于统计而言，则主要是aggregate操作，比如 group、sum、avg、project、match……

aggregate可以将上述操作组织成 pipeline 形式，依次经过各种操作处理。

 

本文是MongoDB University M101的课程笔记，主要记录：MongoDB aggregate的一些常用操作。

参考资料：sql to aggregation mapping chart

①project

它是个1：1的操作，即一个Document输入给project处理，输出一个新的Document。它主要对Key进行处理（大小写转换、删除原来Document某些Key……）

比如原Document如下：

{
    "city" : "ACMAR",
    "loc" : [
        -86.51557,
        33.584132
    ],
    "pop" : 6055,
    "state" : "AL",
    "_id" : "35004"
}

想把它变成：

{
    "city" : "acmar",
    "pop" : 6055,
    "state" : "AL",
    "zip" : "35004"
}

使用：project操作符进行处理：

db.zips.aggregate([
{$project:{_id:0, city:{$toLower:"$city"}, pop:1, state:1, zip:"$_id"}}
])

_id:0  去掉原Document中的_id字段；city:{$ toLower:"$city"}   对原Document中的 "$city" 的值全部转换成小写，赋给新city字段

pop:1  state:1  表示将原Document中的 pop 字段、state 字段 放到新Document中

zip:"$_id" 将原Document中的 \'_id\'字段值  赋值给 新的 "zip" 字段  

 

②group avg，根据分组求平均值。比如某个Document格式如下：对 state字段进行分组，求每个state的人口(pop)的平均值

{
    "city" : "FISHERS ISLAND",
    "loc" : [
            -72.017834,
            41.263934
    ],
    "pop" : 329,
    "state" : "NY",
    "_id" : "06390"
}

 

db.zips.aggregate([
{"$group":{"_id":"$state", "average_pop":{"$avg":"$pop"}}}
])

$group表示分组操作，执行该操作后会生成一个新Document。

_id:$state 表示对 $state 字段进行分组，生成的新Document的 _id 为 state的值

"$avg":"$pop" 表示对原Document中的 “pop”字段按 $state 分组求平均值。得到的平均值为 "average_pop"字段的值。

最终的结果如下：

{ "_id" : "NY", "average_pop" : 9705.34 }
{ "_id" : "CT", "average_pop" : 13226.48 }
{ "_id" : "CA", "average_pop" : 19067.72 }
{ "_id" : "NJ", "average_pop" : 16949.9 }

 

③match

Document示例如下：想要过滤人口字段(pop)大于100 000 的所有记录。

{
    "city" : "ACMAR",
    "loc" : [
        -86.51557,
        33.584132
    ],
    "pop" : 6055,
    "state" : "AL",
    "_id" : "35004"
}

 

db.zips.aggregate([
{$match:{
    pop:{$gt:100000}
    }
}
])

$match表示对 Document进行过滤

pop:{$gt:100000} 表示根据 pop 字段过滤，过滤的条件为 pop 的值大于100000

 

④sort

Document示例如下，现在需要对 state 和 city 这两个字段进行升序排序。

{
    "city" : "ACMAR",
    "loc" : [
        -86.51557,
        33.584132
    ],
    "pop" : 6055,
    "state" : "AL",
    "_id" : "35004"
}

 

db.zips.aggregate( [ { $sort:{state:1, city:1} } ] )

返回结果如下：

{ "_id" : "95915", "city" : "BELDEN", "loc" : [ -121.325924, 39.921746 ], "pop" : 32, "state" : "CA" }
{ "_id" : "90706", "city" : "BELLFLOWER", "loc" : [ -118.126527, 33.886676 ], "pop" : 61650, "state" : "CA" }
{ "_id" : "93430", "city" : "CAYUCOS", "loc" : [ -120.890791, 35.444606 ], "pop" : 3384, "state" : "CA" }
{ "_id" : "96107", "city" : "COLEVILLE", "loc" : [ -119.482784, 38.502903 ], "pop" : 1370, "state" : "CA" }

 

⑤group、sort、first

Document示例如下，现在要寻找，每个州(state)下的 每个城市(city)人口的最大值。

 

{ "_id" : "07840", "city" : "HACKETTSTOWN", "loc" : [ -74.834315, 40.852891 ], "pop" : 23440, "state" : "NJ" }
{ "_id" : "93254", "city" : "NEW CUYAMA", "loc" : [ -119.823806, 34.996709 ], "pop" : 80, "state" : "CA" }
{ "_id" : "92278", "city" : "TWENTYNINE PALMS", "loc" : [ -116.06041, 34.237969 ], "pop" : 11412, "state" : "CA" }
{ "_id" : "08536", "city" : "PLAINSBORO", "loc" : [ -74.568836, 40.332432 ], "pop" : 13008, "state" : "NJ" }
{ "_id" : "06117", "city" : "W HARTFORD", "loc" : [ -72.745689, 41.790021 ], "pop" : 14774, "state" : "CT" }
{ "_id" : "06071", "city" : "SOMERS", "loc" : [ -72.458266, 41.997813 ], "pop" : 9685, "state" : "CT" }
{ "_id" : "92070", "city" : "SANTA YSABEL", "loc" : [ -116.69635, 33.147579 ], "pop" : 1263, "state" : "CA" }
{ "_id" : "91941", "city" : "LA MESA", "loc" : [ -117.011541, 32.760431 ], "pop" : 42536, "state" : "CA" }
{ "_id" : "06705", "city" : "WATERBURY", "loc" : [ -72.996268, 41.550328 ], "pop" : 25128, "state" : "CT" }
{ "_id" : "07750", "city" : "MONMOUTH BEACH", "loc" : [ -73.98089, 40.333032 ], "pop" : 3329, "state" : "NJ" }
{ "_id" : "06095", "city" : "WINDSOR", "loc" : [ -72.663893, 41.856122 ], "pop" : 27815, "state" : "CT" }
{ "_id" : "06702", "city" : "WATERBURY", "loc" : [ -73.038545, 41.556568 ], "pop" : 4522, "state" : "CT" }
{ "_id" : "13833", "city" : "SANITARIA SPRING", "loc" : [ -75.790978, 42.195735 ], "pop" : 4777, "state" : "NY" }
{ "_id" : "95363", "city" : "PATTERSON", "loc" : [ -121.140732, 37.490592 ], "pop" : 13437, "state" : "CA" }

 

第一步：统计每个州下每个城市的人口总和：

db.zips.aggregate( [ 
{ $group:
{_id:{state:"$state",city:"$city"},
 population:{$sum:"$pop"}} 
} 
] )

得到如下Document：（注意Document的 _id 变化）

{ "_id" : { "state" : "NJ", "city" : "ISLAND HEIGHTS" }, "population" : 1470 }
{ "_id" : { "state" : "NY", "city" : "REDWOOD" }, "population" : 1735 }
{ "_id" : { "state" : "CT", "city" : "TAFTVILLE" }, "population" : 2538 }
{ "_id" : { "state" : "CT", "city" : "ELLINGTON" }, "population" : 9070 }
{ "_id" : { "state" : "CT", "city" : "STORRS MANSFIELD" }, "population" : 16117 }
{ "_id" : { "state" : "NJ", "city" : "MERCHANTVILLE" }, "population" : 22294 }
{ "_id" : { "state" : "NJ", "city" : "STRATHMERE" }, "population" : 163 }
{ "_id" : { "state" : "CA", "city" : "LA JOLLA" }, "population" : 40399 }
{ "_id" : { "state" : "NY", "city" : "HURLEYVILLE" }, "population" : 3303 }
{ "_id" : { "state" : "NJ", "city" : "NORTH BRANCH" }, "population" : 34212 }
{ "_id" : { "state" : "NJ", "city" : "WEEHAWKEN" }, "population" : 69646 }
{ "_id" : { "state" : "NJ", "city" : "MANALAPAN" }, "population" : 28928 }
{ "_id" : { "state" : "NY", "city" : "NEW YORK" }, "population" : 8190 }
{ "_id" : { "state" : "NY", "city" : "DEWITTVILLE" }, "population" : 1159 }
{ "_id" : { "state" : "NY", "city" : "QUEENSBURY" }, "population" : 15023 }
....
....

 

按state 对人口进行排序

db.zips.aggregate( [ 
{ $group:{_id:{state:"$state",city:"$city"}, population:{$sum:"$pop"}} },
{$sort:{"_id.state":1, "population":-1}} 
] )

得到如下Document：

{ "_id" : { "state" : "CA", "city" : "LOS ANGELES" }, "population" : 104702 }
{ "_id" : { "state" : "CA", "city" : "LA MESA" }, "population" : 66480 }
{ "_id" : { "state" : "CA", "city" : "SHORE ACRES" }, "population" : 64053 }
{ "_id" : { "state" : "CA", "city" : "BELLFLOWER" }, "population" : 61650 }
{ "_id" : { "state" : "CA", "city" : "VISALIA" }, "population" : 51620 }
{ "_id" : { "state" : "CA", "city" : "SAN DIEGO" }, "population" : 45487 }
{ "_id" : { "state" : "CA", "city" : "GOLD RIVER" }, "population" : 42461 }
....
....

 

 

然后，再对 state 进行 group by，使用 $first 取第一条记录：就是这个州下 的人口最大的城市(city).

first 一般和 group使用：$group得到一组Document后，使用 $first 获取该组Documents中的第一个Document。

 db.zips.aggregate([ 
{
 $group:{_id:{state:"$state",city:"$city"},
 population:{$sum:"$pop"}
}
},
{$sort:{"$_id.state":1, "population":-1}},
{$group:{_id:"_id.state",
         city:{$first:"$_id.city"},
          population:{$first:"$population"}
    }}
])

得到：

{ "_id" : "NJ", "city" : "JERSEY CITY", "population" : 100756 }
{ "_id" : "NY", "city" : "FLUSHING", "population" : 51947 }
{ "_id" : "CT", "city" : "BRISTOL", "population" : 60670 }
{ "_id" : "CA", "city" : "LOS ANGELES", "population" : 104702 }

最后再对 _id 排序

 db.zips.aggregate( [ 
{ $group:{_id:{state:"$state",city:"$city"},
 population:{$sum:"$pop"}} },
{$sort:{"_id.state":1, "population":-1}},
{$group:{_id:"$_id.state",city:{$first:"$_id.city"},population:{$first:"$population"}}},
{$sort:{"_id":1}}
 ] )

得到：

{ "_id" : "CA", "city" : "LOS ANGELES", "population" : 104702 }
{ "_id" : "CT", "city" : "BRISTOL", "population" : 60670 }
{ "_id" : "NJ", "city" : "JERSEY CITY", "population" : 100756 }
{ "_id" : "NY", "city" : "FLUSHING", "population" : 51947 }