鏈€浣冲疄璺?| MongoDB瀹氫綅oplog蹇呴』鍏ㄨ〃鎵弿鍚楋紵

Posted 2021-05-01 闃块噷浜戞暟鎹簱

MongoDB oplog 锛堢被浼间簬 mysql binlog锛?璁板綍鏁版嵁搴撶殑鎵€鏈変慨鏀规搷浣溿€傞櫎浜嗙敤浜庝富澶囧悓姝ワ紝oplog 杩樿兘鐜╁嚭寰堝鑺辨牱锛屾瘮濡傦細

馃敹鍏ㄩ噺澶囦唤 + 澧為噺澶囦唤鎵€鏈夌殑 oplog锛屽氨鑳藉疄鐜?MongoDB 鎭㈠鍒颁换鎰忔椂闂寸偣鐨勫姛鑳?/span>

馃敺閫氳繃 oplog锛岄櫎浜嗗疄鐜板埌澶囪妭鐐圭殑鍚屾锛屼篃鍙互棰濆鍐嶅線鍗曠嫭鐨勯泦缇ゅ悓姝ユ暟鎹紙鐢氳嚦鏄紓鏋勭殑鏁版嵁搴擄級锛屽疄鐜板鐏俱€佸娲荤瓑鍦烘櫙锛屾瘮濡?/span>闃块噷浜戝紑婧愮殑MongoShake (https://github.com/alibaba/MongoShake) 灏辫兘瀹炵幇鍩轰簬 oplog 鐨勫閲忓悓姝ャ€?/span>

馃敹MongoDB 3.6+ 鐗堟湰瀵?oplog 杩涜浜嗘娊璞★紝鎻愪緵浜?Change Stream 鐨勬帴鍙ｏ紝瀹為檯涓婂氨鏄兘涓嶆柇璁㈤槄鏁版嵁搴撶殑淇敼锛屽熀浜庤繖浜涗慨鏀瑰彲浠ヨЕ鍙戜竴浜涜嚜瀹氫箟鐨勪簨浠躲€?/span>

馃敺......

鎬荤殑鏉ヨ锛孧ongoDB 鍙互閫氳繃 oplog 鏉ヨ窡鐢熸€佸鎺ワ紝鏉ュ疄鐜版暟鎹殑鍚屾銆佽縼绉汇€佹仮澶嶇瓑鑳藉姏銆傝€屽湪鏋勫缓杩欎簺鑳藉姏鐨勬椂鍊欙紝鏈変竴涓€氱敤鐨勯渶姹傦紝灏辨槸宸ュ叿鎴栬€呭簲鐢ㄩ渶瑕佹湁涓嶆柇鎷夊彇 oplog 鐨勮兘鍔涳紝杩欎釜杩囩▼閫氬父鏄細

1锛屾牴鎹笂娆℃媺鍙栫殑浣嶇偣鏋勫缓涓€涓?cursor

2锛屼笉鏂凯浠?cursor 鑾峰彇鏂扮殑 oplog

Q

閭ｄ箞闂鏉ヤ簡锛岀敱浜?MongoDB oplog 鏈韩娌℃湁绱㈠紩鐨勶紝姣忔瀹氫綅 oplog 鐨勮捣鐐归兘闇€瑕佽繘琛屽叏琛ㄦ壂鎻忎箞锛?/span>

oplog 鐨勫疄鐜扮粏鑺?/span>

* 璇峰悜鍙虫粦鍔ㄩ槄瑙堜唬鐮?

{ "ts" : Timestamp(1563950955, 2), "t" : NumberLong(1), "h" : NumberLong("-5936505825938726695"), "v" : 2, "op" : "i", "ns" : "test.coll", "ui" : UUID("020b51b7-15c2-4525-9c35-cd50f4db100d"), "wall" : ISODate("2019-07-24T06:49:15.903Z"), "o" : { "_id" : ObjectId("5d37ff6b204906ac17e28740"), "x" : 0 } }
{ "ts" : Timestamp(1563950955, 3), "t" : NumberLong(1), "h" : NumberLong("-1206874032147642463"), "v" : 2, "op" : "i", "ns" : "test.coll", "ui" : UUID("020b51b7-15c2-4525-9c35-cd50f4db100d"), "wall" : ISODate("2019-07-24T06:49:15.903Z"), "o" : { "_id" : ObjectId("5d37ff6b204906ac17e28741"), "x" : 1 } }
{ "ts" : Timestamp(1563950955, 4), "t" : NumberLong(1), "h" : NumberLong("1059466947856398068"), "v" : 2, "op" : "i", "ns" : "test.coll", "ui" : UUID("020b51b7-15c2-4525-9c35-cd50f4db100d"), "wall" : ISODate("2019-07-24T06:49:15.913Z"), "o" : { "_id" : ObjectId("5d37ff6b204906ac17e28742"), "x" : 2 } }

涓婇潰鏄?MongoDB oplog 鐨勭ず渚嬶紝oplog MongoDB 涔熸槸涓€涓泦鍚堬紝浣嗕笌鏅€氶泦鍚堜笉涓€鏍枫€?/span>

 馃敹oplog 鏄竴涓?capped collection锛屼絾瓒呰繃閰嶇疆澶у皬鍚庯紝灏变細鍒犻櫎鏈€鑰佹彃鍏ョ殑鏁版嵁

 馃敹oplog 闆嗗悎娌℃湁 id 瀛楁锛宼s 鍙互浣滀负 oplog 鐨勫敮涓€鏍囪瘑; oplog 闆嗗悎鐨勬暟鎹湰韬槸鎸?ts 椤哄簭缁勭粐鐨?/span>

 馃敹oplog 娌℃湁浠讳綍绱㈠紩瀛楁锛岄€氬父瑕佹壘鍒版煇鏉?oplog 瑕佽蛋鍏ㄨ〃鎵弿

鎴戜滑鍦ㄦ媺鍙?oplog 鏃讹紝绗竴娆′粠澶村紑濮嬫媺鍙栵紝鐒跺悗姣忔鎷夊彇浣跨敤瀹岋紝浼氳褰曟渶鍚庝竴鏉?oplog 鐨則s瀛楁锛涘鏋滃簲鐢ㄥ彂鐢熼噸鍚紝杩欐椂闇€瑕佹牴鎹笂娆℃媺鍙栫殑 ts 瀛楁锛屽厛鎵惧埌鎷夊彇鐨勮捣鐐癸紝鐒跺悗缁х画閬嶅巻銆?br class="mq-160">

oplogHack 浼樺寲

娉細浠ヤ笅瀹炵幇閽堝 WiredTiger 瀛樺偍寮曟搸锛岄渶瑕?MongoDB 3.0+ 鐗堟湰鎵嶈兘鏀寔銆?/span>

濡傛灉 MongoDB 搴曞眰浣跨敤鐨勬槸 WiredTiger 瀛樺偍寮曟搸锛屽湪瀛樺偍 oplog 鏃讹紝瀹為檯涓婂仛杩囦紭鍖栥€侻ongoDB 浼氬皢 ts 瀛楁浣滀负 key锛宱plog 鐨勫唴瀹逛綔涓?value锛屽皢key-value 瀛樺偍鍒?WiredTiger 寮曟搸閲岋紝WiredTiger 榛樿閰嶇疆浣跨敤 btree 瀛樺偍锛屾墍浠?oplog 鐨勬暟鎹湪 WT 閲屽疄闄呬笂涔熸槸鎸?ts 瀛楁椤哄簭瀛樺偍鐨勶紝鏃㈢劧鏄『搴忓瓨鍌紝閭ｅ氨鏈変簩鍒嗘煡鎵句紭鍖栫殑绌洪棿銆?/span>

MongoDB find 鍛戒护

锛坔ttps://docs.mongodb.com/manual/reference/command/find/锛夋彁渚涗簡涓€涓€夐」锛屼笓闂ㄧ敤浜庝紭鍖?oplog 瀹氫綅銆?/span>

澶ц嚧鎰忔€濇槸锛屽鏋滀綘find鐨勯泦鍚堟槸oplog锛屾煡鎵炬潯浠舵槸閽堝 ts 瀛楁鐨?gte銆乬t銆乪q 锛岄偅涔?MongoDB 瀛楁浼氳繘琛屼紭鍖栵紝閫氳繃浜屽垎鏌ユ壘蹇€熷畾浣嶅埌璧风偣; 澶囪妭鐐瑰悓姝ユ媺鍙杘plog鏃讹紝瀹為檯涓婂氨甯︿簡杩欎釜閫夐」锛岃繖鏍峰鑺傜偣姣忔閲嶅惎锛岄兘鑳芥牴鎹笂娆″悓姝ョ殑浣嶇偣锛屽揩閫熸壘鍒板悓姝ヨ捣鐐癸紝鐒跺悗鎸佺画淇濇寔鍚屾銆?/span>

oplogHack 瀹炵幇

鐢变簬鍜ㄨ闂鐨勫悓瀛﹀鍐呴儴瀹炵幇鎰熷叴瓒ｏ紝杩欓噷绠€鍗曠殑鎶婇噸鐐瑰垪鍑烘潵锛岃娣卞埢鐞嗚В锛岃繕鏄緱娣卞叆鎾哥粏鑺傘€?/span>

* 璇峰悜鍙虫粦鍔ㄩ槄瑙堜唬鐮?

// src/monogo/db/query/get_executor.cpp
StatusWith<unique_ptr<PlanExecutor>> getExecutorFind(OperationContext* txn,
                                                     Collection* collection,
                                                     const NamespaceString& nss,
                                                     unique_ptr<CanonicalQuery> canonicalQuery,
                                                     PlanExecutor::YieldPolicy yieldPolicy) {
    // 鏋勫缓 find 鎵ц璁″垝鏃讹紝濡傛灉鍙戠幇鏈?oplogReplay 閫夐」锛屽垯璧颁紭鍖栬矾寰?/span>
    if (NULL != collection && canonicalQuery->getQueryRequest().isOplogReplay()) {
        return getOplogStartHack(txn, collection, std::move(canonicalQuery));
    }

   ...
   
    return getExecutor(
        txn, collection, std::move(canonicalQuery), PlanExecutor::YIELD_AUTO, options);
}

 StatusWith<unique_ptr<PlanExecutor>> getOplogStartHack(OperationContext* txn,
                                                   Collection* collection,
                                                   unique_ptr<CanonicalQuery> cq) {

    // See if the RecordStore supports the oplogStartHack
    // 濡傛灉搴曞眰寮曟搸鏀寔锛圵T鏀寔锛宮mapv1涓嶆敮鎸侊級锛屾牴鎹煡璇㈢殑ts锛屾壘鍒?startLoc
    const BSONElement tsElem = extractOplogTsOptime(tsExpr);
    if (tsElem.type() == bsonTimestamp) {
        StatusWith<RecordId> goal = oploghack::keyForOptime(tsElem.timestamp());
        if (goal.isOK()) {
            // 鏈€缁堣皟鐢?src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp::oplogStartHack
            startLoc = collection->getRecordStore()->oplogStartHack(txn, goal.getValue());
        }
    }
    
     // Build our collection scan...
     // 鏋勫缓鍏ㄨ〃鎵弿鍙傛暟鏃讹紝甯︿笂 startLoc锛岀湡姝ｆ墽琛屾槸浼氬揩閫熷畾浣嶅埌杩欎釜鐐?/span>
    CollectionScanParams params;
    params.collection = collection;
    params.start = *startLoc;
    params.direction = CollectionScanParams::FORWARD;
    params.tailable = cq->getQueryRequest().isTailable();
}

MongoDB鏂扮敤鎴疯瘯鐢?

棣栨湀浠呴渶59鍏?/strong>

鐐瑰嚮 闃呰鍘熸枃浜嗚В璇︽儏

闃块噷宸村反鏁版嵁搴撴妧鏈?/span>

寰俊锛歛libabadba

鍒嗕韩鏁版嵁搴撳墠娌?nbsp;

瑙ｆ瀯瀹炴垬骞茶揣

闀挎寜浜岀淮鐮佸叧娉?/strong>