技术学习Mongo Administration

Posted 2021-02-27 tben

Latest: Support Plan and Cloud Manager

Access Cloud Manager: 

https://cloud.mongodb.com/

Access Mongo Support

https://support.mongodb.com

Tuning

https://www.percona.com/blog/2016/08/12/tuning-linux-for-mongodb/

ALSO:

Copy collections across MongoDB databases and servers

 

NOTE:

In AWS we started using Mongo3.0.  This created a problem for authentication from clients not supporting the new authentication mechanism (meteor can‘t connect).

Therefore, if using 3.0 we must downgrade the schema

// Use MongoDB 2.6 auth schema (default auth mechanism: MONGODB-CR) use admin; db.system.version.save({ "_id" : "authSchema", "currentVersion" : 3 })

 

Administration Tasks

Increasing Replica Timeout

Due to vmware (veeam) backups, it can hang our servers for a bit. We tried to increase the heartbeat by 10 seconds to prevent a mongo switch.  Here is how you increase the timeout for > 3.2 mongo

cfg = rs.config()
cfg.settings.electionTimeoutMillis=20000
#default is 10000
rs.reconfig(cfg)

 

Creating a replica set for single instance Mongo (AQA and Collector Fix)

Per Shiming‘s request, here is a short instruction how oplog can be
enabled on stand-alone Mongo system:
http://tuttlem.github.io/2014/06/13/how-to-setup-an-oplog-on-a-single-mongodb-instance.html
This is year 2014 so it may not be very fresh but I hope it still works.
Here is another reference:
http://stackoverflow.com/questions/20487002/oplog-enable-on-standalone-mongod-not-for-replicaset.
Accepted answer is outdated but take a look at comments and second
answer. I believe it goes in line with the first link above.
-Sergey

Adding Authentication to MongoDB Replica Set – Full Example

https://docs.mongodb.org/manual/tutorial/enable-authentication/

1) Login to the PRIMARY replica and create users.

# FULL root privilage use admin db.createUser({user: "root", pwd: "root", roles: ["root"] } )   # Create admin user for sxa database use sxa db.createUser({user: "sxaAdmin", pwd: "sxaAdmin", roles: [ { role: "userAdmin", db: "sxa" }, { role: "readWrite", db: "sxa" }  ] } )   # readonly user for sxa db use sxa db.createUser({user: "readonlysxa", pwd: "readonlysxa", roles: [ { role: "read", db:"sxa" } ] } )   # readonly user for anything use admin db.createUser({user: "readonlyadmin", pwd: "readonlyadmin", roles: [ { role: "readAnyDatabase", db:"admin" } ] } )

2) We add additional security. Use keyFile authentication for Intra-Replica communication (amongst themselves). Create a keyfile.

Reference: Mongo KeyFile Instructions
http://docs.mongodb.org/manual/tutorial/generate-key-file/

openssl rand -base64 741 > /etc/mongodb-keyfile chmod 600 /etc/mongodb-keyfile chown mongodb:mongodb /etc/mongodb-keyfile

3) Copy this /etc/mongodb-keyfile to every replica instance. Don‘t forget to chmod and chown as above or mongod won‘t start.

4) Edit the /etc/mongod.conf file on every replica.  Enable these lines

vi /etc/mongod.conf # Enable user authentication auth=true #Enable keyFile for intra-replica authentication keyFile=/etc/mongodb-keyfile

 

PAUSE: We‘re at the point where we want this all to take effect. We can do this without downtime by keeping a PRIMARY replica up at all times.

Reference: Mongo Maintenance Procedure
http://docs.mongodb.org/manual/tutorial/perform-maintence-on-replica-set-members/

5) Now we need to restart the replicas for these changes to take effect.  Begin by logging into a SECONDARY replica and restart the mongod.

restart mongod

6) Continue for all secondaries

7) Last, login to the PRIMARY.  Tell it to give up mastership and become a SECONDARY

rs.stepDown()

8) Now we‘re ok to restart mongod on the last replica without downtime.

restart mongod

 

10) Extra Cred: The mongo URL that is needed for other applications such as meteor is in a very particular format when you use authentication.

MONGO_URL="mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]"   Example: MONGO=URL="mongodb://sxaAdmin:[e-mail]:27017,ec2-52-24-148-47.us-west-2.compute.amazonaws.com:27017,ec2-52-26-60-142.us-west-2.compute.amazonaws.com:27017/sxa?replicaSet=mongodb&readPreference=primaryPreferred"

Get/Drop Users

use admin db.getUsers()    use sxa db.getUsers() db.dropUser("sxaAdmin")    use admin db.dropAllUsers() #!

Add Roles

use sxa db.grantRolesToUser("sxaAdmin", [{ role: "readWrite", db: "local" } , { role: "userAdmin", db:"local" } ] )

Revoke Roles

use sxa
db.revokeRolesFromUser("sxaAdmin",[{ role: "readWrite", db: "local" } , { role: "userAdmin", db:"local" } ])

 

Running Mongo Stand-Alone for debugging purposes.

We use --auth if we want to enable logging.

screen mongod --port 37017 --dbpath /data --auth ctrl-A-D

Drop down Schema to 2.6 due to Authentication errors with 3.0

Support for the new authentication in MongoDB 3.0 has caused problems with meteor, and with tools like RoboMongo. Work around is to drop the schema down.

// Use MongoDB 2.6 auth schema (default auth mechanism: MONGODB-CR) use admin; db.system.version.save({ "_id" : "authSchema", "currentVersion" : 3 })

This may not be that simple.  I found I needed to start mongod in stand-alone mode for every replica memeber with authentication disabled (remove the --auth), then set the authSchema value.

Resync MongoDB Replica Set

• Check MongoDB Replica Set status - if the secondary MongoDB replica member stays in "RECOVERING" state, it will require resync to get it back to correct state.
  • login one of the MongoDB VM  instance (199.71.143.62 or 199.71.143.63)
  • Run: "mongo 199.71.143.62/sxa"

  • Run rs.status command in MongoDB shell. The following will be example of result

     

    cmdctr:PRIMARY> rs.status() {         "set" : "cmdctr",         "date" : ISODate("2015-06-08T01:00:51Z"),         "myState" : 1,         "members" : [                 {                         "_id" : 0,                         "name" : "199.71.142.249:27017",                         "health" : 1,                         "state" : 7,                         "stateStr" : "ARBITER",                         "uptime" : 3818114,                         "lastHeartbeat" : ISODate("2015-06-08T01:00:49Z"),                         "lastHeartbeatRecv" : ISODate("2015-06-08T01:00:49Z"),                         "pingMs" : 0                 },                 {                         "_id" : 1,                         "name" : "199.71.143.62:27017",                         "health" : 1,                         "state" : 1,                         "stateStr" : "PRIMARY",                         "uptime" : 20854145,                         "optime" : Timestamp(1433725251, 3),                         "optimeDate" : ISODate("2015-06-08T01:00:51Z"),                         "electionTime" : Timestamp(1430122091, 1),                         "electionDate" : ISODate("2015-04-27T08:08:11Z"),                         "self" : true                 },                 {                         "_id" : 2,                         "name" : "199.71.143.63:27017",                         "health" : 1,                         "state" : 3,                         "stateStr" : "RECOVERING",                         "uptime" : 1633250,                         "optime" : Timestamp(1433184037, 6),                         "optimeDate" : ISODate("2015-06-01T18:40:37Z"),                         "lastHeartbeat" : ISODate("2015-06-08T01:00:50Z"),                         "lastHeartbeatRecv" : ISODate("2015-06-08T01:00:50Z"),                         "pingMs" : 3,                         "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 5574e47c:a2"                 }         ],         "ok" : 1 }

• Steps to resync MongoDB replica set - recommend to perform this procedure during light MongoDB traffic.
  • Login the secondary MongoDB VM instance - 199.71.143.63 in the above example.
  • Stop MongoDB process
    • supervisorctl
    • stop mongo
    • exit
  • back up and delete MongoDB data folder
    • cd /var/lib
    •  mv mongo mongo_bak_<timestamp> (replace <timestamp> to string like "06072015")
    • mkdir mongo
    • chown mongod:mongod mongo
  • Start MongoDB process
    • supervisorctl
    • start mongo
    • exit
  • Wait for sync to complete by checking rs.status() or db.printSlaveReplicationInfo()

    • The result of db.printSlaveReplicationInfo() should show something like the following when sync is complete:

       

      cmdctr:PRIMARY> db.printSlaveReplicationInfo() source: 199.71.143.63:27017         syncedTo: Thu May 28 2015 12:02:25 GMT-0700 (PDT)         0 secs (0 hrs) behind the primary

       

       

Renaming a Mongo Replica

 

Example: Mongo replica set is using the name "compass-dev-2" instead of "compass-dev-2.xx.com" causing DNS name resolution errors. We can rename it as follows.  Note: it is also the current primary server.

 

ssh compass-dev-2.xx.com

 

mongo

 

rs.stepDown()

 

 

 

Login to new primary

 

cfg = rs.config()

 

cfg.members[0].host="compass-dev-2.xx.com:27017”

 

rs.reconfig(cfg)

 

Maintenance

http://docs.mongodb.org/manual/tutorial/perform-maintence-on-replica-set-members/

For command center deployments, you must disable cron jobs on JBOSS, and the CC+ backend during the mongo restore or it can conflict with the restore process.

 

We should even shut down these components as well during the restore.

 

Backup/Restore

mongodump --out <backup_file>   Example: This dumps the "sxa" database from the mongo server 199.71.142.151.  It can be run anywhere, though it is faster to run locally and then tar the results and scp it over to your destination.    mongodump -h 199.71.142.151 --out /tmp/goldbackup --db sxa           Example: Note: You can restore to an alternate DB if desired. mongorestore --drop -d new_db_name mongodump/old_db_name    mongorestore --drop --host 199.71.142.151 -d sxa gold-backup-080216/sxa

Restoring from production DB for testing and staging systems

Note: We copy the backup NFS to the local disk.  It seemed to me I could just restore directly from the NFS location and save time and a step, but in one case restoring from NFS failed on me, and I thought it might have been due to some NFS instability. I did not pursue that problem, I worked around it by copying the data to the local disk before the restore.

Example: Restore prod DB to gold system.
 
#Identify gold system primary. 
ssh -p 1035 root@199.71.142.151
 
#Ensure there is enough local disk to hold the backup. Copy your backup over to the local disk.
# Recommend you use screen in case you lose the ssh connection
screen
mkdir /root/restores
cp -frp /backup/command-center-db/mongo/2017-02-21_020001/199.71.143.62 /root/restores
cd /root/restores
 
# WARNING: ENSURE THE HOST IP IS YOUR GOLD SYSTEM AND NOT PRODUCTION.  It is possible to overwrite production remotely if you accidentally provide a production mongo IP.
 
mongorestore --drop --host 199.71.142.151 --db sxa dump/sxa

 

 

 

Reference: http://docs.mongodb.org/manual/tutorial/backup-with-mongodump/

Have not watched this yet, but should be good: https://www.mongodb.com/presentations/webinar-backups-disaster-recovery

Fail Over Explanation

Tuning and/or Performance Monitoring

Script provided by David Jansen

 

General

Mongo URI URL syntax

For other applications, such as meteor.

%vmlnx-djansen<225> /opt/script/devops/mongo/dump_system_profile.py -h usage: dump query info from system.profile with optional date query        [-h] [--mongo_url MONGO_URL] [--after AFTER] [--before BEFORE]    optional arguments:   -h, --help            show this help message and exit   --mongo_url MONGO_URL                         Mongo Url (default mongodb://localhost:27017/sxa)   --after AFTER         DateTime of format like "2016-04-12T08:00:00Z"   --before BEFORE       DateTime of format like "2016-04-12T09:00:00Z"    %vmlnx-djansen<226> /opt/script/devops/mongo/dump_system_profile.py --after 2016-04-13T16Z --before 2016-04-13T16:01Z before : 2016-04-13T16:01Z ==> 2016-04-13 16:01:00+00:00 after : 2016-04-13T16Z ==> 2016-04-13 16:00:00+00:00 using query: "{"ts": {"$lte": {"$date": 1460563260000}, "$gte": {"$date": 1460563200000}}}" ----------------------------------

 

Installing 3.0 version

Installing mongodb-org-tools for 3.0 instructions are located here:

https://docs.mongodb.org/v3.0/tutorial/install-mongodb-on-ubuntu/

Reference

Installing a Specific Version of mongodb

http://knowledgepia.com/en/k-blog/database/how-to-install-mongodb-2-6-on-ubuntu-14-10-14-04-and-12-04-lts

Enabling Authentication in 2.6

https://docs.mongodb.org/v2.6/tutorial/enable-authentication/

 

Upgrade Mongo 2.6.5 to 3.2

Overview - Production Migration/Upgrade Plan

 

1) Launch 2 brand new centos7 servers.

2) Install mongo 2.6.5 on them ahead of time.  Before the migration, like 1-2 hours before, add them to the replica set so the data syncs.  We could set priority to zero to prevent these from becoming a primary, but this is not tested yet, so I‘m not going to include that step at this point.

3) Once all replicas are synced then you can begin upgrading the standbys.

4) Switch IP for meteor1 and meteor2 to new mongo servers mongo1 and mongo2

5) Assign NEW IPs for meteor1 and meteor2 and update nginx to reflect this.

 

Migrating 2.6.5 to stand-alone Servers

Create new centos 7.2 servers.

Install Mongo 2.6.5. 

https://docs.mongodb.com/v2.6/tutorial/install-mongodb-on-red-hat/

 

[root@GOLD-MONGO32-2-DEV ~]# vi /etc/yum.repos.d/mongodb.repo
[mongodb]
name=MongoDB Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/
gpgcheck=0
enabled=1

[root@GOLD-MONGO32-2-DEV ~]# sudo yum install -y mongodb-org-2.6.5 mongodb-org-server-2.6.5 mongodb-org-shell-2.6.5 mongodb-org-mongos-2.6.5 mongodb-org-tools-2.6.5

 

Setup supervisord /etc/supervisor/conf.d/mongo.conf

Had to change this to use sudo otherwise if I try to run as mongod with supervisord the process limits are not set to our specs.

[program:mongo]
#command=mongod --smallfiles -f /etc/mongod.conf
command=sudo -u mongod bash -c ‘mongod --smallfiles -f /etc/mongod.conf‘
numprocs=1
#directory=/var/lib/mongo
stdout_logfile=/var/log/mongodb/[e-mail]
stderr_logfile=/var/log/mongodb/[e-mail]
autostart=true
autorestart=true
stopasgroup=true
user=root
~

Create /etc/mongod.conf

Create path:

mkdir -p /home/mongo/mongodb
chmod 777 /home/mongo/mongodb
chown -R mongod:mongod /home/mongo
chown -R mongod:mongod /home/mongo/mongodb

# mongod.conf
#where to log
logpath=/var/log/mongodb/mongod.log
logappend=true
# fork and run in background
#fork=true
#port=27017
dbpath=/home/mongo/mongodb
# location of pidfile
pidfilepath=/home/mongo/mongod.pid
# Listen to local interface only. Comment out to listen on all interfaces.
bind_ip=0.0.0.0
# Disables write-ahead journaling
# nojournal=true
# Enables periodic logging of CPU utilization and I/O wait
#cpu=true
# Turn on/off security.  Off is currently the default
#noauth=true
#auth=true
# Verbose logging output.
#verbose=true
# Inspect all client data for validity on receipt (useful for
# developing drivers)
#objcheck=true
# Enable db quota management
#quota=true
# Set oplogging level where n is
#   0=off (default)
#   1=W
#   2=R
#   3=both
#   7=W+some reads
#diaglog=0
# Ignore query hints
#nohints=true
# Enable the HTTP interface (Defaults to port 28017).
httpinterface=false
# Turns off server-side scripting.  This will result in greatly limited
# functionality
#noscripting=true
# Turns off table scans.  Any query that would do a table scan fails.
#notablescan=true
# Disable data file preallocation.
#noprealloc=true
# Specify .ns file size for new databases.
# nssize=<size>
# Replication Options
# in replicated mongo databases, specify the replica set name here
replSet=gold
# maximum size in megabytes for replication operation log
#oplogSize=1024
# path to a key file storing authentication info for connections
# between replica set members
#keyFile=/path/to/keyfile
 

Sudoers File

Need to disable need for tty, or supervisor process won‘t start.

vi /etc/sudoers

#
#Defaults    requiretty

 

Start mongo

supervisorctl update mongo

 

Add this new mongo server to the replica set.

Login to the primary server (assuming you know how to determine this)

> rs.add("192.35.85.16:27017")

 Access Analyze

Note, you can use this format: <replica>host:port,host:port,host:port, and you don‘t need to determine the primary node.

 

We use mongoexport to get AA organization list. As following
mongoexport -h 199.71.143.62 -d sxa -c sxacms_organizations -f shortName,name,orgId,email -o $MONGOEXPORT --csv

Mongo to Redshift / Spark Aggregation

/home/ubuntu/sxadp-config-file

 

 

 

Routing Rules (Optional) - CURRENTLY NOT WORKING, and not sure if this is a good idea because two IPs would be going to the same mongo service.

iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j DNAT --to 192.168.1.2:8080
iptables -A FORWARD -p tcp -d 192.168.1.2 --dport 8080 -j ACCEPT

 

Upgrading Mongo 2.6 to 3.0

This is required on the path to 3.2.  You can‘t go from 2.6 to 3.2.

https://docs.mongodb.com/manual/release-notes/3.0-upgrade/

New centos servers must have this mtu setting or aws storm will get hung up.

sudo ip link set dev <interface> mtu 1300
Must be set permanently. Not discussed here, it is located if you search the wiki for mtu.

 

https://docs.mongodb.com/v3.0/tutorial/install-mongodb-on-red-hat/

vi  /etc/yum.repos.d/mongodb-org-3.0.repo

[mongodb-org-3.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.0/x86_64/
gpgcheck=0
enabled=1

 

supervisorctl stop mongo
sudo yum install -y mongodb-org-3.0.12 mongodb-org-server-3.0.12 mongodb-org-shell-3.0.12 mongodb-org-mongos-3.0.12 mongodb-org-tools-3.0.12

 

Upgrading Mongo 3.0 to 3.2

New centos servers must have this mtu setting or aws storm will get hung up.

sudo ip link set dev <interface> mtu 1300
Must be set permanently. Not discussed here, it is located if you search the wiki for mtu.

Centos

Create file

vi /etc/yum.repos.d/mongodb-org-3.2.repo
 
[mongodb-org-3.2]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.2/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.2.asc

 

Do this for all the secondaries, then step-down the primary as the last step and upgrade that one last.

supervisorctl stop mongo
sudo yum install -y mongodb-org-3.2.8 mongodb-org-server-3.2.8 mongodb-org-shell-3.2.8 mongodb-org-mongos-3.2.8 mongodb-org-tools-3.2.8
 
supervisorctl start mongo

 

Ubuntu 14.04

Reference:https://docs.mongodb.com/manual/release-notes/3.2-upgrade/

Reference: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/ 

Begin with a standby member of the replica set.

• Login to a Standby mongo instance
• Shutdown mongo on the server

Add repo key

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927

echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
 
apt-get update

Install specific version of mongo.  This happens to be the latest stable as well, but might be useful to make sure we‘re explict in all our upgrades.

sudo apt-get install -y mongodb-org=3.2.8 mongodb-org-server=3.2.8 mongodb-org-shell=3.2.8 mongodb-org-mongos=3.2.8 mongodb-org-tools=3.2.8

• Repeat for the other Standby.

Finally, let‘s do the primary.

Login to the primary

rs.stepDown()

Wait until this system no longer reports itself as Primary.  Keep using rs.status() to check. When it finally is a standby then proceed.

Perform the usual upgrade of the binaries like you did before.

echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list

Install specific version of mongo.  This happens to be the latest stable as well, but might be useful to make sure we‘re explict in all our upgrades.

sudo apt-get install -y mongodb-org=3.2.8 mongodb-org-server=3.2.8 mongodb-org-shell=3.2.8 mongodb-org-mongos=3.2.8 mongodb-org-tools=3.2.8

All servers are upgraded to 3.2. Lastly, connect to the current primary server and run this.

cfg = rs.conf();
cfg.protocolVersion=1;
rs.reconfig(cfg);

DONE

 

On ubuntu you can pin the versions down to prevent accidental upgrade. Not sure on centos.

echo "mongodb-org hold" | sudo dpkg --set-selections
echo "mongodb-org-server hold" | sudo dpkg --set-selections
echo "mongodb-org-shell hold" | sudo dpkg --set-selections
echo "mongodb-org-mongos hold" | sudo dpkg --set-selections
echo "mongodb-org-tools hold" | sudo dpkg --set-selections

Downgrade

https://docs.mongodb.com/manual/release-notes/3.2-downgrade/#downgrade-replica-set

Note, for downgrade, you need to set a variable in the replica set called protocolVersion first,  Then you can do the usual downgrade of each standby.

sudo apt-get install  mongodb-org=3.0.1 mongodb-org-server=3.0.1 mongodb-org-shell=3.0.1 mongodb-org-mongos=3.0.1 mongodb-org-tools=3.0.1

 

Testing Upgrade

1) Restart storm topology

2) Possibly need to restart the collector if FA+/RT doesn‘t come back entirely. Do that after the storm topo has been restarted

Centos

reference: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-red-hat/

 

 

Mongo Cloud Manager

Fix for centos7 and automation agent

To the best of my knowledge, Fedora 20 is equivalent to RHEL 7 which is not yet supported by MongoDB. There is work planned for this latter OS, but I cannot guarantee a timetable for that release. In the meantime, I believe it is possible to work around the issue with the following:

cd /lib64
sudo ln -s libsasl2.so.3.0.0 libsasl2.so.2

Hopefully this helps, and allows you to start the agent.