MongoDB副本集(一主两从)读写分离故障转移功能环境部署记录
Posted 散尽浮华
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了MongoDB副本集(一主两从)读写分离故障转移功能环境部署记录相关的知识，希望对你有一定的参考价值。
Mongodb是一种非关系数据库(NoSQL)，非关系型数据库的产生就是为了解决大数据量、高扩展性、高性能、灵活数据模型、高可用性。MongoDB官方已经不建议使用主从模式了，替代方案是采用副本集的模式。主从模式其实就是一个单副本的应用，没有很好的扩展性和容错性，而Mongodb副本集具有多个副本保证了容错性，就算一个副本挂掉了还有很多副本存在，主节点挂掉后，整个集群内会实现自动切换。
Mongodb副本集的工作原理
客户端连接到整个Mongodb副本集，不关心具体哪一台节点机是否挂掉。主节点机负责整个副本集的读写，副本集定期同步数据备份，一但主节点挂掉，副本节点就会选举一个新的主服务器，这一切对于应用服务器不需要关心。副本集中的副本节点在主节点挂掉后通过心跳机制检测到后，就会在集群内发起主节点的选举机制，自动选举一位新的主服务器。
看起来Mongodb副本集很牛X的样子，下面就演示下副本集环境部署过程，官方推荐的Mongodb副本集机器数量为至少3个节点，这里我就选择三个节点，一个主节点，两个从节点，暂不使用仲裁节点。
一、环境准备
ip地址 主机名 角色 172.16.60.205 mongodb-master01 副本集主节点 172.16.60.206 mongodb-slave01 副本集副本节点 172.16.60.207 mongodb-slave02 副本集副本节点 三个节点机均设置好各自的主机名，并如下设置好hosts绑定 [[email protected] ~]# cat /etc/hosts ............ 172.16.60.205 mongodb-master01 172.16.60.206 mongodb-slave01 172.16.60.207 mongodb-slave02 三个节点机均关闭selinux，为了测试方便，将iptables也关闭 [[email protected] ~]# setenforce 0 [[email protected] ~]# cat /etc/sysconfig/selinux ........... SELINUX=disabled [[email protected] ~]# iptables -F [[email protected] ~]# /etc/init.d/iptables stop [[email protected] ~]# /etc/init.d/iptables stop iptables: Setting chains to policy ACCEPT: filter [ OK ] iptables: Flushing firewall rules: [ OK ] iptables: Unloading modules: [ OK ]
二、Mongodb安装、副本集配置
1) 在三个节点机上建立mongodb副本集测试文件夹，用于存放整个副本集文件 [[email protected] ~]# mkdir -p /data/mongodb/data/replset/ 2）在三个节点机上安装mongodb 下载地址：https://www.mongodb.org/dl/linux/x86_64-rhel62 [[email protected] ~]# wget http://downloads.mongodb.org/linux/mongodb-linux-x86_64-rhel62-v3.6-latest.tgz [[email protected] ~]# tar -zvxf mongodb-linux-x86_64-rhel62-v3.6-latest.tgz 3）分别在每个节点机上启动mongodb（启动时指明--bind_ip地址，默认是127.0.0.1，需要改成本机ip，否则远程连接时失败） [[email protected] ~]# mv mongodb-linux-x86_64-rhel62-3.6.11-rc0-2-g2151d1d219 /usr/local/mongodb [[email protected] ~]# nohup /usr/local/mongodb/bin/mongod -dbpath /data/mongodb/data/replset -replSet repset --bind_ip=172.16.60.205 --port=27017 & [[email protected] ~]# ps -ef|grep mongodb root 7729 6977 1 15:10 pts/1 00:00:01 /usr/local/mongodb/bin/mongod -dbpath /data/mongodb/data/replset -replSet repset root 7780 6977 0 15:11 pts/1 00:00:00 grep mongodb [[email protected] ~]# lsof -i:27017 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME mongod 7729 root 10u IPv4 6554476 0t0 TCP localhost:27017 (LISTEN) 4）初始化副本集 在三个节点中的任意一个节点机上操作（比如在172.16.60.205节点机） 登陆mongodb [[email protected] ~]# /usr/local/mongodb/bin/mongo 172.16.60.205:27017 ......... #使用admin数据库 > use admin switched to db admin #定义副本集配置变量，这里的 _id:”repset” 和上面命令参数“ –replSet repset” 要保持一样。 > config = { _id:"repset", members:[{_id:0,host:"172.16.60.205:27017"},{_id:1,host:"172.16.60.206:27017"},{_id:2,host:"172.16.60.207:27017"}]} { "_id" : "repset", "members" : [ { "_id" : 0, "host" : "172.16.60.205:27017" }, { "_id" : 1, "host" : "172.16.60.206:27017" }, { "_id" : 2, "host" : "172.16.60.207:27017" } ] } #初始化副本集配置 > rs.initiate(config); { "ok" : 1, "operationTime" : Timestamp(1551166191, 1), "$clusterTime" : { "clusterTime" : Timestamp(1551166191, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } } } #查看集群节点的状态 repset:SECONDARY> rs.status(); { "set" : "repset", "date" : ISODate("2019-02-26T07:31:07.766Z"), "myState" : 1, "term" : NumberLong(1), "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "heartbeatIntervalMillis" : NumberLong(2000), "optimes" : { "lastCommittedOpTime" : { "ts" : Timestamp(1551166263, 1), "t" : NumberLong(1) }, "readConcernMajorityOpTime" : { "ts" : Timestamp(1551166263, 1), "t" : NumberLong(1) }, "appliedOpTime" : { "ts" : Timestamp(1551166263, 1), "t" : NumberLong(1) }, "durableOpTime" : { "ts" : Timestamp(1551166263, 1), "t" : NumberLong(1) } }, "members" : [ { "_id" : 0, "name" : "172.16.60.205:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 270, "optime" : { "ts" : Timestamp(1551166263, 1), "t" : NumberLong(1) }, "optimeDate" : ISODate("2019-02-26T07:31:03Z"), "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "could not find member to sync from", "electionTime" : Timestamp(1551166202, 1), "electionDate" : ISODate("2019-02-26T07:30:02Z"), "configVersion" : 1, "self" : true, "lastHeartbeatMessage" : "" }, { "_id" : 1, "name" : "172.16.60.206:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 76, "optime" : { "ts" : Timestamp(1551166263, 1), "t" : NumberLong(1) }, "optimeDurable" : { "ts" : Timestamp(1551166263, 1), "t" : NumberLong(1) }, "optimeDate" : ISODate("2019-02-26T07:31:03Z"), "optimeDurableDate" : ISODate("2019-02-26T07:31:03Z"), "lastHeartbeat" : ISODate("2019-02-26T07:31:06.590Z"), "lastHeartbeatRecv" : ISODate("2019-02-26T07:31:06.852Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "172.16.60.205:27017", "syncSourceHost" : "172.16.60.205:27017", "syncSourceId" : 0, "infoMessage" : "", "configVersion" : 1 }, { "_id" : 2, "name" : "172.16.60.207:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 76, "optime" : { "ts" : Timestamp(1551166263, 1), "t" : NumberLong(1) }, "optimeDurable" : { "ts" : Timestamp(1551166263, 1), "t" : NumberLong(1) }, "optimeDate" : ISODate("2019-02-26T07:31:03Z"), "optimeDurableDate" : ISODate("2019-02-26T07:31:03Z"), "lastHeartbeat" : ISODate("2019-02-26T07:31:06.589Z"), "lastHeartbeatRecv" : ISODate("2019-02-26T07:31:06.958Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "172.16.60.205:27017", "syncSourceHost" : "172.16.60.205:27017", "syncSourceId" : 0, "infoMessage" : "", "configVersion" : 1 } ], "ok" : 1, "operationTime" : Timestamp(1551166263, 1), "$clusterTime" : { "clusterTime" : Timestamp(1551166263, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } } } 如上信息表明： 副本集配置成功后，172.16.60.205为主节点PRIMARY，172.16.60.206/207为副本节点SECONDARY。 health：1 1表明状态是正常，0表明异常 state:1 值小的是primary节点、值大的是secondary节点
三、测试Mongodb副本集数据复制功能 <mongodb默认是从主节点读写数据的，副本节点上不允许读，需要设置副本节点可以读>
1）在主节点172.16.60.205上连接到终端 [[email protected] ~]# /usr/local/mongodb/bin/mongo 172.16.60.205:27017 ................ #建立test 数据库 repset:PRIMARY> use test; switched to db test #往testdb表插入测试数据 repset:PRIMARY> db.testdb.insert({"test1":"testval1"}) WriteResult({ "nInserted" : 1 }) 2）在副本节点172.16.60.206、172.16.60.207上连接到mongodb查看数据是否复制过来。 这里在172.16.60.206副本节点上进行查看 [[email protected] ~]# /usr/local/mongodb/bin/mongo 172.16.60.206:27017 ................ repset:SECONDARY> use test; switched to db test repset:SECONDARY> show tables; 2019-02-26T15:37:46.446+0800 E QUERY [thread1] Error: listCollections failed: { "operationTime" : Timestamp(1551166663, 1), "ok" : 0, "errmsg" : "not master and slaveOk=false", "code" : 13435, "codeName" : "NotMasterNoSlaveOk", "$clusterTime" : { "clusterTime" : Timestamp(1551166663, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } } } : [email protected]/mongo/shell/utils.js:25:13 [email protected]/mongo/shell/db.js:941:1 [email protected]/mongo/shell/db.js:953:19 [email protected]/mongo/shell/db.js:964:16 [email protected]/mongo/shell/utils.js:853:9 [email protected]/mongo/shell/utils.js:750:15 @(shellhelp2):1:1 上面出现了报错！ 这是因为mongodb默认是从主节点读写数据的，副本节点上不允许读，需要设置副本节点可以读 repset:SECONDARY> db.getMongo().setSlaveOk(); repset:SECONDARY> db.testdb.find(); { "_id" : ObjectId("5c74ec9267d8c3d06506449b"), "test1" : "testval1" } repset:SECONDARY> show tables; testdb 如上发现已经在副本节点上发现了测试数据，即已经从主节点复制过来了。 （在另一个副本节点172.16.60.207也如上操作即可）
四、测试副本集故障转移功能
先停掉主节点172.16.60.205，查看mongodb副本集状态，可以看到经过一系列的投票选择操作，172.16.60.206当选主节点，172.16.60.207从172.16.60.206同步数据过来。
1）停掉原来的主节点172.16.60.205的mongodb，模拟故障 [[email protected] ~]# ps -ef|grep mongodb|grep -v grep|awk ‘{print $2}‘|xargs kill -9 [[email protected] ~]# lsof -i:27017 [[email protected] ~]# 2）接着登录到另外两个正常的从节点172.16.60.206、172.16.60.207中的任意一个节点的mongodb，查看副本集状态 [[email protected] ~]# /usr/local/mongodb/bin/mongo 172.16.60.206:27017 ................. repset:PRIMARY> rs.status(); { "set" : "repset", "date" : ISODate("2019-02-26T08:06:02.996Z"), "myState" : 1, "term" : NumberLong(2), "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "heartbeatIntervalMillis" : NumberLong(2000), "optimes" : { "lastCommittedOpTime" : { "ts" : Timestamp(1551168359, 1), "t" : NumberLong(2) }, "readConcernMajorityOpTime" : { "ts" : Timestamp(1551168359, 1), "t" : NumberLong(2) }, "appliedOpTime" : { "ts" : Timestamp(1551168359, 1), "t" : NumberLong(2) }, "durableOpTime" : { "ts" : Timestamp(1551168359, 1), "t" : NumberLong(2) } }, "members" : [ { "_id" : 0, "name" : "172.16.60.205:27017", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "ts" : Timestamp(0, 0), "t" : NumberLong(-1) }, "optimeDurable" : { "ts" : Timestamp(0, 0), "t" : NumberLong(-1) }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2019-02-26T08:06:02.917Z"), "lastHeartbeatRecv" : ISODate("2019-02-26T08:03:37.492Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "Connection refused", "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "configVersion" : -1 }, { "_id" : 1, "name" : "172.16.60.206:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2246, "optime" : { "ts" : Timestamp(1551168359, 1), "t" : NumberLong(2) }, "optimeDate" : ISODate("2019-02-26T08:05:59Z"), "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "electionTime" : Timestamp(1551168228, 1), "electionDate" : ISODate("2019-02-26T08:03:48Z"), "configVersion" : 1, "self" : true, "lastHeartbeatMessage" : "" }, { "_id" : 2, "name" : "172.16.60.207:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2169, "optime" : { "ts" : Timestamp(1551168359, 1), "t" : NumberLong(2) }, "optimeDurable" : { "ts" : Timestamp(1551168359, 1), "t" : NumberLong(2) }, "optimeDate" : ISODate("2019-02-26T08:05:59Z"), "optimeDurableDate" : ISODate("2019-02-26T08:05:59Z"), "lastHeartbeat" : ISODate("2019-02-26T08:06:02.861Z"), "lastHeartbeatRecv" : ISODate("2019-02-26T08:06:02.991Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "172.16.60.206:27017", "syncSourceHost" : "172.16.60.206:27017", "syncSourceId" : 1, "infoMessage" : "", "configVersion" : 1 } ], "ok" : 1, "operationTime" : Timestamp(1551168359, 1), "$clusterTime" : { "clusterTime" : Timestamp(1551168359, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } } } 发现当原来的主节点172.16.60.205宕掉后，经过选举，原来的从节点172.16.60.206被推举为新的主节点。 3）现在在172.16.60.206新主节点上创建测试数据 repset:PRIMARY> use kevin; switched to db kevin repset:PRIMARY> db.kevin.insert({"shibo":"hahaha"}) WriteResult({ "nInserted" : 1 }) 4）另一个从节点172.16.60.207上登录mongodb查看 [[email protected] ~]# /usr/local/mongodb/bin/mongo 172.16.60.207:27017 ................ repset:SECONDARY> use kevin; switched to db kevin repset:SECONDARY> db.getMongo().setSlaveOk(); repset:SECONDARY> show tables; kevin repset:SECONDARY> db.kevin.find(); { "_id" : ObjectId("5c74f42bb0b339ed6eb68e9c"), "shibo" : "hahaha" } 发现从节点172.16.60.207可以同步新的主节点172.16.60.206的数据 5）再重新启动原来的主节点172.16.60.205的mongodb [[email protected] ~]# nohup /usr/local/mongodb/bin/mongod -dbpath /data/mongodb/data/replset -replSet repset --bind_ip=172.16.60.205 --port=27017 & mongod 9162 root 49u IPv4 6561201 0t0 TCP mongodb-master01:55236->mongodb-slave01:27017 (ESTABLISHED) [[email protected] ~]# ps -ef|grep mongodb root 9162 6977 4 16:14 pts/1 00:00:01 /usr/local/mongodb/bin/mongod -dbpath /data/mongodb/data/replset -replSet repset --bind_ip=172.16.60.205 --port=27017 root 9244 6977 0 16:14 pts/1 00:00:00 grep mongodb 再次登录到三个节点中的任意一个的mongodb，查看副本集状态 [[email protected] ~]# /usr/local/mongodb/bin/mongo 172.16.60.205:27017 .................... repset:SECONDARY> rs.status(); { "set" : "repset", "date" : ISODate("2019-02-26T08:16:11.741Z"), "myState" : 2, "term" : NumberLong(2), "syncingTo" : "172.16.60.206:27017", "syncSourceHost" : "172.16.60.206:27017", "syncSourceId" : 1, "heartbeatIntervalMillis" : NumberLong(2000), "optimes" : { "lastCommittedOpTime" : { "ts" : Timestamp(1551168969, 1), "t" : NumberLong(2) }, "readConcernMajorityOpTime" : { "ts" : Timestamp(1551168969, 1), "t" : NumberLong(2) }, "appliedOpTime" : { "ts" : Timestamp(1551168969, 1), "t" : NumberLong(2) }, "durableOpTime" : { "ts" : Timestamp(1551168969, 1), "t" : NumberLong(2) } }, "members" : [ { "_id" : 0, "name" : "172.16.60.205:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 129, "optime" : { "ts" : Timestamp(1551168969, 1), "t" : NumberLong(2) }, "optimeDate" : ISODate("2019-02-26T08:16:09Z"), "syncingTo" : "172.16.60.206:27017", "syncSourceHost" : "172.16.60.206:27017", "syncSourceId" : 1, "infoMessage" : "", "configVersion" : 1, "self" : true, "lastHeartbeatMessage" : "" }, { "_id" : 1, "name" : "172.16.60.206:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 127, "optime" : { "ts" : Timestamp(1551168969, 1), "t" : NumberLong(2) }, "optimeDurable" : { "ts" : Timestamp(1551168969, 1), "t" : NumberLong(2) }, "optimeDate" : ISODate("2019-02-26T08:16:09Z"), "optimeDurableDate" : ISODate("2019-02-26T08:16:09Z"), "lastHeartbeat" : ISODate("2019-02-26T08:16:10.990Z"), "lastHeartbeatRecv" : ISODate("2019-02-26T08:16:11.518Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "electionTime" : Timestamp(1551168228, 1), "electionDate" : ISODate("2019-02-26T08:03:48Z"), "configVersion" : 1 }, { "_id" : 2, "name" : "172.16.60.207:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 127, "optime" : { "ts" : Timestamp(1551168969, 1), "t" : NumberLong(2) }, "optimeDurable" : { "ts" : Timestamp(1551168969, 1), "t" : NumberLong(2) }, "optimeDate" : ISODate("2019-02-26T08:16:09Z"), "optimeDurableDate" : ISODate("2019-02-26T08:16:09Z"), "lastHeartbeat" : ISODate("2019-02-26T08:16:10.990Z"), "lastHeartbeatRecv" : ISODate("2019-02-26T08:16:11.655Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "172.16.60.206:27017", "syncSourceHost" : "172.16.60.206:27017", "syncSourceId" : 1, "infoMessage" : "", "configVersion" : 1 } ], "ok" : 1, "operationTime" : Timestamp(1551168969, 1), "$clusterTime" : { "clusterTime" : Timestamp(1551168969, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } } } 发现原来的主节点172.16.60.205在故障恢复后，变成了新的主节点172.16.60.206的从节点
五、Mongodb读写分离
目前来看。Mongodb副本集可以完美支持故障转移。至于主节点的读写压力过大如何解决？常见的解决方案是读写分离。
一般情况下，常规写操作来说并没有读操作多，所以在Mongodb副本集中，一台主节点负责写操作，两台副本节点负责读操作。 1）设置读写分离需要先在副本节点SECONDARY 设置 setSlaveOk。 2）在程序中设置副本节点负责读操作，如下代码：
public class TestMongoDBReplSetReadSplit { public static void main(String[] args) { try { List<ServerAddress> addresses = new ArrayList<ServerAddress>(); ServerAddress address1 = new ServerAddress("172.16.60.205" , 27017); ServerAddress address2 = new ServerAddress("172.16.60.206" , 27017); ServerAddress address3 = new ServerAddress("172.16.60.207" , 27017); addresses.add(address1); addresses.add(address2); addresses.add(address3); MongoClient client = new MongoClient(addresses); DB db = client.getDB( "test" ); DBCollection coll = db.getCollection( "testdb" ); BasicDBObject object = new BasicDBObject(); object.append( "test2" , "testval2" ); //读操作从副本节点读取 ReadPreference preference = ReadPreference. secondary(); DBObject dbObject = coll.findOne(object, null , preference); System. out .println(dbObject); } catch (Exception e) { e.printStackTrace(); } } }
读参数除了secondary一共还有五个参数：primary、primaryPreferred、secondary、secondaryPreferred、nearest。
primary：默认参数，只从主节点上进行读取操作；
primaryPreferred：大部分从主节点上读取数据,只有主节点不可用时从secondary节点读取数据。
secondary：只从secondary节点上进行读取操作，存在的问题是secondary节点的数据会比primary节点数据“旧”。
secondaryPreferred：优先从secondary节点进行读取操作，secondary节点不可用时从主节点读取数据；
nearest：不管是主节点、secondary节点，从网络延迟最低的节点上读取数据。
读写分离做好后，就可以进行数据分流，减轻压力，解决了"主节点的读写压力过大如何解决？"这个问题。不过当副本节点增多时，主节点的复制压力会加大有什么办法解决吗？基于这个问题，Mongodb已有了相应的解决方案 - 引用仲裁节点：
在Mongodb副本集中，仲裁节点不存储数据，只是负责故障转移的群体投票，这样就少了数据复制的压力。看起来想的很周到啊，其实不只是主节点、副本节点、仲裁节点，还有Secondary-Only、Hidden、Delayed、Non-Voting，其中：
Secondary-Only：不能成为primary节点，只能作为secondary副本节点，防止一些性能不高的节点成为主节点。
Hidden：这类节点是不能够被客户端制定IP引用，也不能被设置为主节点，但是可以投票，一般用于备份数据。
Delayed：可以指定一个时间延迟从primary节点同步数据。主要用于备份数据，如果实时同步，误删除数据马上同步到从节点，恢复又恢复不了。
Non-Voting：没有选举权的secondary节点，纯粹的备份数据节点。
以上是关于MongoDB副本集(一主两从)读写分离故障转移功能环境部署记录的主要内容，如果未能解决你的问题，请参考以下文章