我们的一个已投产项目的高可用数据库实战 - mongo 副本集的搭建详细过程
Posted Defonds
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了我们的一个已投产项目的高可用数据库实战 - mongo 副本集的搭建详细过程相关的知识，希望对你有一定的参考价值。
我们的 mongo 副本集有三台 mongo 服务器：一台主库两台从库。主库进行写操作，两台从库进行读操作(至于某次读操作究竟路由给了哪台，仲裁决定)，实现了读写分离。这还不止，如果主库宕掉，还能实现不需要用户干预的情况下，将主库自动切换到另外两台从库中的某一台，真正实现了 db 的高可用。1. 背景
1.1 环境
- CPU核数：4
- 内存配置：8G
- 带宽：100MB
- 磁盘：系统盘 40G，数据盘 180G
- 操作系统版本：Ubuntu 14.04 64位
1.2 系统部署结构图
2. MongoDB 副本集环境搭建
2.1 安装包下载
最新 MongoDB 安装包下载地址：https://www.mongodb.org/downloads#production
选择适合我们操作系统的版本下载，最新版本是 mongodb-linux-x86_64-ubuntu1404-3.2.1.tgz，大小 74MB。
2.2 MongoDB 的安装和环境变量设置
解压到当前目录：$ tar zxvf mongodb-linux-x86_64-ubuntu1404-3.2.1.tgz
得到 mongodb-linux-x86_64-ubuntu1404-3.2.1 文件夹，其内有 bin 等文件/目录。
MongoDB 是绿色免安装软件，但我们准备把 MongoDB 作为安装软件放在 /usr/local 目录下：
$ mv mongodb-linux-x86_64-ubuntu1404-3.2.1 /usr/local/mongodb
最后将 MongoDB 加进环境变量并立即生效：
$ echo "export PATH=/usr/local/mongodb/bin:$PATH" >> ~/.bashrc
$ source ~/.bashrc
2.3 MongoDB 实例所需先决条件准备
根据目前的架构，MongoDB 副本集需要四个实例：一个仲裁节点、一个 master 节点、两个 slave 节点。每个实例数据、日志文件都需要有一个专门的目录存放，我们分别将其放在数据盘下的 rs1- 端口号 目录下。每个节点之间访问需要授权，因此需要同一个授权文件。
每个节点数据、日志目录创建：
$ cd /opt/
$ mkdir mongodb
$ cd mongodb/
$ mkdir arb-30000
$ mkdir rs1-27017
$ mkdir rs1-27018
$ mkdir rs1-27019
授权文件创建并修改其访问权限：
$ openssl rand -base64 512 > rs1.key
$ chmod 600 rs1.key
2.4 用户实例创建
2.4.1 副本集四个实例的首次启动
因为用户还没创建，所以首次启动 MongoDB 副本集，不要带上 auth 参数，不然无法创建用户。master 节点启动(端口号 27017)：
$ mongod --port=27017 --fork --dbpath=/opt/mongodb/rs1-27017 --logpath=/opt/mongodb/rs1-27017/mongod.log --replSet=rs1
slave 节点之一启动(端口号 27018)：
$ mongod --port=27018 --fork --dbpath=/opt/mongodb/rs1-27018 --logpath=/opt/mongodb/rs1-27018/mongod.log --replSet=rs1
slave 节点之二启动(端口号 27019)：
$ mongod --port=27019 --fork --dbpath=/opt/mongodb/rs1-27019 --logpath=/opt/mongodb/rs1-27019/mongod.log --replSet=rs1
输出如下证明启动成功：
about to fork child process, waiting until server is ready for connections.
forked process: 21947
child process started successfully, parent exiting
最后是仲裁节点的启动(端口号 30000)：
$ mongod --port=30000 --fork --dbpath=/opt/mongodb/arb-30000 --logpath=/opt/mongodb/arb-30000/mongod.log --replSet=rs1
2.4.2 副本集的创建
现在每个节点都已经启动成功了，现在我们要创建一个副本集，并把这些节点加进来。先登录主节点：$ mongo -port 27017
进入后执行副本集初始化：
rs.initiate()
建好以后查看本节点是否 master：
rs1:OTHER> rs.isMaster()
输出：
{
"hosts" : [
"somehost:27017"
],
"setName" : "rs1",
"setVersion" : 1,
"ismaster" : true,
"secondary" : false,
"primary" : "somehost:27017",
"me" : "somehost:27017",
"electionId" : ObjectId("56b175570000000000000001"),
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 1000,
"localTime" : ISODate("2016-02-03T03:35:26.532Z"),
"maxWireVersion" : 4,
"minWireVersion" : 0,
"ok" : 1
}
之后依次把两个 slave 节点、仲裁节点加进来：
rs1:PRIMARY> rs.add(‘somehost:27018‘)
{ "ok" : 1 }
rs1:PRIMARY> rs.add(‘somehost:27019‘)
{ "ok" : 1 }
rs1:PRIMARY> rs.addArb(‘somehost:30000‘)
{ "ok" : 1 }
注意 somehost 是每个节点的主机名(可以使用 hostname 命令查看)。
验证副本集状态：
rs1:PRIMARY> rs.status()
{
"set" : "rs1",
"date" : ISODate("2016-02-03T03:38:56.519Z"),
"myState" : 1,
"term" : NumberLong(1),
"heartbeatIntervalMillis" : NumberLong(2000),
"members" : [
{
"_id" : 0,
"name" : "somehost:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 560,
"optime" : {
"ts" : Timestamp(1454470707, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2016-02-03T03:38:27Z"),
"electionTime" : Timestamp(1454470487, 2),
"electionDate" : ISODate("2016-02-03T03:34:47Z"),
"configVersion" : 4,
"self" : true
},
{
"_id" : 1,
"name" : "somehost:27018",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 102,
"optime" : {
"ts" : Timestamp(1454470707, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2016-02-03T03:38:27Z"),
"lastHeartbeat" : ISODate("2016-02-03T03:38:55.923Z"),
"lastHeartbeatRecv" : ISODate("2016-02-03T03:38:55.927Z"
),
"pingMs" : NumberLong(0),
"syncingTo" : "somehost:27017",
"configVersion" : 4
},
{
"_id" : 2,
"name" : "somehost:27019",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 85,
"optime" : {
"ts" : Timestamp(1454470707, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2016-02-03T03:38:27Z"),
"lastHeartbeat" : ISODate("2016-02-03T03:38:55.923Z"),
"lastHeartbeatRecv" : ISODate("2016-02-03T03:38:54.924Z"
),
"pingMs" : NumberLong(0),
"syncingTo" : "somehost:27018",
"configVersion" : 4
},
{
"_id" : 3,
"name" : "somehost:30000",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 28,
"lastHeartbeat" : ISODate("2016-02-03T03:38:55.923Z"),
"lastHeartbeatRecv" : ISODate("2016-02-03T03:38:52.940Z"
),
"pingMs" : NumberLong(0),
"configVersion" : 4
}
],
"ok" : 1
}
3.4.3 管理员用户的创建
rs1:PRIMARY> use adminswitched to db admin
rs1:PRIMARY> db.createUser({user:"admin",pwd:"adminpwd",roles:["root"]})
Successfully added user: { "user" : "admin", "roles" : [ "root" ] }
rs1:PRIMARY> exit
bye
用户名 admin 密码 adminpwd 的数据库管理员用户已创建。
验证所创建的用户是否已同步到其他几个节点：
$ mongo -port 27018
rs1:SECONDARY> use admin
switched to db admin
rs1:SECONDARY> db.auth("admin","adminpwd")
1
3.4.4 生产库、生产用户的创建
先把原来启动的实例依次关闭，顺序是两个 slave 节点、主节点、仲裁节点：rs1:SECONDARY> db.shutdownServer()
server should be down...
2016-02-03T11:46:36.234+0800 I NETWORK [thread1] trying reconnect to 127.0.0.1:27018 (127.0.0.1) failed
2016-02-03T11:46:36.235+0800 W NETWORK [thread1] Failed to connect to 127.0.0.1:27018, reason: errno:111 Connection refused
2016-02-03T11:46:36.235+0800 I NETWORK [thread1] reconnect 127.0.0.1:27018 (127.0.0.1) failed failed
2016-02-03T11:46:36.237+0800 I NETWORK [thread1] trying reconnect to 127.0.0.1:27018 (127.0.0.1) failed
2016-02-03T11:46:36.237+0800 W NETWORK [thread1] Failed to connect to 127.0.0.1:27018, reason: errno:111 Connection refused
2016-02-03T11:46:36.237+0800 I NETWORK [thread1] reconnect 127.0.0.1:27018 (127.0.0.1) failed failed
$ mongo -port 27019
rs1:SECONDARY> use admin
switched to db admin
rs1:SECONDARY> db.auth("admin","adminpwd")
1
rs1:SECONDARY> db.shutdownServer()
server should be down...
2016-02-03T11:48:01.191+0800 I NETWORK [thread1] trying reconnect to 127.0.0.1:27019 (127.0.0.1) failed
2016-02-03T11:48:01.191+0800 W NETWORK [thread1] Failed to connect to 127.0.0.:27019, reason: errno:111 Connection refused
2016-02-03T11:48:01.191+0800 I NETWORK [thread1] reconnect 127.0.0.1:27019 (12.0.0.1) failed failed
2016-02-03T11:48:01.193+0800 I NETWORK [thread1] trying reconnect to 127.0.0.1:27019 (127.0.0.1) failed
2016-02-03T11:48:01.193+0800 W NETWORK [thread1] Failed to connect to 127.0.0.1:27019, reason: errno:111 Connection refused
2016-02-03T11:48:01.193+0800 I NETWORK [thread1] reconnect 127.0.0.1:27019 (12.0.0.1) failed failed
同理关掉 master 节点和仲裁节点。
然后再按顺序依次启动，注意这次要加上 auth 参数了：
$ mongod --port=27017 --fork --dbpath=/opt/mongodb/rs1-27017 --logpath=/opt/mongodb/rs1-27017/mongod.log --replSet=rs1 --auth --keyFile=/opt/mongodb/rs1.key
about to fork child process, waiting until server is ready for connections.
forked process: 22803
child process started successfully, parent exiting
$ mongod --port=27018 --fork --dbpath=/opt/mongodb/rs1-27018 --logpath=/opt/mongodb/rs1-27018/mongod.log --replSet=rs1 --auth --keyFile=/opt/mongodb/rs1.key
about to fork child process, waiting until server is ready for connections.
forked process: 22803
child process started successfully, parent exiting
$ mongod --port=27019 --fork --dbpath=/opt/mongodb/rs1-27019 --logpath=/opt/mongodb/rs1-27019/mongod.log --replSet=rs1 --auth --keyFile=/opt/mongodb/rs1.key
about to fork child process, waiting until server is ready for connections.
forked process: 23068
child process started successfully, parent exiting
$ mongod --port=30000 --fork --dbpath=/opt/mongodb/arb-30000 --logpath=/opt/mongodb/arb-30000/mongod.log --replSet=rs1 --keyFile=/opt/mongodb/rs1.key
about to fork child process, waiting until server is ready for connections.
forked process: 23068
child process started successfully, parent exiting
每个节点启动以后，在主节点 test 库里插入一条记录测试是否同步：
$ mongo -port 27017
MongoDB shell version: 3.2.1
connecting to: 127.0.0.1:27017/test
rs1:PRIMARY> db.auth(‘admin‘,‘adminpwd‘)
1
rs1:PRIMARY> use test
switched to db test
rs1:PRIMARY> db.test.insert({test:"test"})
WriteResult({ "nInserted" : 1 })
rs1:PRIMARY> exit
bye
去从库看看是否同步：
$ mongo -port 27018
MongoDB shell version: 3.2.1
connecting to: 127.0.0.1:27018/test
rs1:SECONDARY> use test
switched to db test
rs1:SECONDARY> use admin
switched to db admin
rs1:SECONDARY> db.auth(‘admin‘,‘adminpwd‘)
1
rs1:SECONDARY> use test
switched to db test
rs1:SECONDARY> rs.slaveOk()
rs1:SECONDARY> db.test.find()
{ "_id" : ObjectId("56b17b7cafc9b56181378550"), "test" : "test" }
同步成功。
接下来创建生产数据库、生产用户，先登录 master 节点：
$ mongo -port 27017
rs1:PRIMARY> db.auth("admin","adminpwd")
rs1:PRIMARY> use quicktest
rs1:PRIMARY> db.createUser({user:"quicktest",pwd:"quicktest",roles:[{ role: "readWrite", db: "quicktest" }]})
Successfully added user: {
"user" : "quicktest",
"roles" : [
{
"role" : "readWrite",
"db" : "quicktest"
}
]
}
quicktest 生产用户创建成功。
rs1:PRIMARY> db.justtest.insert({test:"test"})
WriteResult({ "nInserted" : 1 })
rs1:PRIMARY> show dbs
admin 0.000GB
local 0.000GB
quicktest 0.000GB
test 0.000GB
quicktest 生产数据库创建成功。
3.5 app 应用 admin 用户的创建
一些基础数据表如用户表等需要先行创建，至于交易表等非基础数据则在交易产生的时候会自行创建。$ mongo -port 27017
MongoDB shell version: 3.2.1
connecting to: 127.0.0.1:27017/test
rs1:PRIMARY> use quicktest
switched to db quicktest
rs1:PRIMARY> db.auth(‘quicktest‘,‘quicktest‘)
1
rs1:PRIMARY> show collections
justtest
masterlog
task
rs1:PRIMARY> db.user.insert({ "_id" : ObjectId("55ef883acb91f093af390689"), "userName" : "admin", "nickName" : "admin", "password" : "c6203112cd2a39dc956feac05ba676e1096c0e9d", "mail" : "[email protected]", "phoneNum" : "13148143761", "userType" : "admin", "agentCode" : "", "subAgentCode" : "", "groupCode" : "", "merId" : "", "areaCode" : "222", "updateTime" : "2015-11-24 17:26:19", "loginTime" : "", "lockTime" : "" })
WriteResult({ "nInserted" : 1 })
以上是关于我们的一个已投产项目的高可用数据库实战 - mongo 副本集的搭建详细过程的主要内容，如果未能解决你的问题，请参考以下文章