分布式存储之MogileFS基于Nginx实现负载均衡（Nginx+MogileFS）
Posted 明天、未来
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了分布式存储之MogileFS基于Nginx实现负载均衡（Nginx+MogileFS）相关的知识，希望对你有一定的参考价值。
MogileFS分布式文件系统特点：
1.具有raid的性能
2.不存在单点故障
3.简单的命名空间： 每个文件对应一个key：用于domain定义名称空间
4.不共享任何数据
5.传输中立，无特殊协议：可以通过NFS或HTTP进行通信
6.自动文件复制：复制的最小单位不是文件，而class
7.应用层： 用户空间文件系统，无须特殊的核心组件
Nginx+MogileFS的好处：
1、将请求代理至后端MogileFS服务器集群中，能实现负载均衡的效果。
2、能对后端的tracker节点进行健康检测。
3、将第三方模块“nginx_mogilefs_module”编译进Nginx中，能实现直接使用key访问对应的文件，如下：
使用nginx做代理之前：http://192.168.80.137:7500/dev2/0/000/000/0000000007.fid
使用nginx做代理之后：http://192.168.80.132/image/1.jpg
MogileFS是由三个组件组成的：
1、tracker：MogileFS的核心，是一个调度器，服务进程为mogilefsd，职责：删除数据、复制数据、监控、查询等。
2、database：为tracker存储元素据
3、数据存储的位置，通常是一个HTTP（webDAV）服务器，用来做数据的创建（put）、删除（delete）、获取（get），监听端口7500, storage节点使用http进行数据传输, 依赖于perbal, 进程为mogstored。
理想中模型：
实验架构：
实验中用到的rpm包：http://pan.baidu.com/s/1c1S0dJM
MariaDB节点配置：
对各mogilefs节点的主机名进行解析：
[root@www ~]# vim /etc/hosts 192.168.80.136 mog1.daixiang.com 192.168.80.137 mog2.daixiang.com 192.168.80.138 mog3.daixiang.com
二进制包安装MariaDB:
[root@mariadb mysql]# useradd -r -s /sbin/nologin mysql [root@mariadb ~]# tar xf mariadb-10.1.14-linux-x86_64.tar.gz -C /usr/local/ [root@mariadb ~]# ln -sv /usr/local/mariadb-10.1.14-linux-x86_64 /usr/local/mysql [root@mariadb mysql]# chown -R mysql.mysql /usr/local/mysql/ [root@mariadb mysql]# ./scripts/mysql_install_db --user=mysql --datadir=/data [root@mariadb mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld [root@mariadb mysql]# chmod +x /etc/rc.d/init.d/mysqld [root@mariadb mysql]# cp support-files/my-large.cnf /etc/my.cnf [root@mariadb mysql]# vim /etc/my.cnf datadir = /data [root@mariadb mysql]# ln -sv /usr/local/mysql/include/ /usr/include/mysql/ [root@mariadb mysql]# vim /etc/ld.so.conf.d/mysql.conf /usr/local/mysql/lib [root@mariadb mysql]# vim /etc/profile.d/mysql.sh export PATH=/usr/local/mysql/bin:$PATH [root@mariadb mysql]# ldconfig [root@mariadb mysql]# ldconfig -p | grep mysql libmysqld.so.18 (libc6,x86-64) => /usr/local/mysql/lib/libmysqld.so.18 libmysqld.so (libc6,x86-64) => /usr/local/mysql/lib/libmysqld.so libmysqlclient_r.so.16 (libc6,x86-64) => /usr/lib64/mysql/libmysqlclient_r.so.16 libmysqlclient.so.18 (libc6,x86-64) => /usr/local/mysql/lib/libmysqlclient.so.18 libmysqlclient.so.16 (libc6,x86-64) => /usr/lib64/mysql/libmysqlclient.so.16 libmysqlclient.so (libc6,x86-64) => /usr/local/mysql/lib/libmysqlclient.so libgalera_smm.so (libc6,x86-64) => /usr/local/mysql/lib/libgalera_smm.so
对用户进行授权：
MariaDB [(none)]> grant all on *.* to \'root\'@\'192.168.80.%\' identified by \'rootpass\'; Query OK, 0 rows affected (0.06 sec) MariaDB [(none)]> grant all on mogilefs.* to \'moguser\'@\'192.168.80.%\' identified by \'mogpass\'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> flush privileges; Query OK, 0 rows affected (0.00 sec)
MogileFS配置：
[root@mog1 ~]# yum install *.rpm -y [root@mog1 ~]# yum install perl-IO-AIO -y [root@mog2 ~]# yum install *.rpm -y [root@mog2 ~]# yum install perl-IO-AIO -y [root@mog3 ~]# yum install *.rpm -y [root@mog3 ~]# yum install perl-IO-AIO -y
修改mogilefsd进程的配置文件：
1 [root@mog1 ~]# vim /etc/mogilefs/mogilefsd.conf 2 3 # Enable daemon mode to work in background and use syslog 4 daemonize = 1 #以守护进程的形式运行 5 # Where to store the pid of the daemon (must be the same in the init script) 6 pidfile = /var/run/mogilefsd/mogilefsd.pid 7 # Database connection information 8 db_dsn = DBI:mysql:mogilefs:host=192.168.80.135 #定义数据库名为mogilefs和数据库服务器地址 9 db_user = moguser #定义管理此数据库的用户名 10 db_pass = mogpass #定义密码 11 # IP:PORT to listen on for mogilefs client requests 12 listen = 0.0.0.0:7001 #定义监听的地址和端口 13 # Optional, if you don\'t define the port above. 14 conf_port = 7001 15 # Number of query workers to start by default. 16 query_jobs = 10 #定义启动查询线程个数 17 # Number of delete workers to start by default. 18 delete_jobs = 1 #定义启动删除线程个数 19 # Number of replicate workers to start by default. 20 replicate_jobs = 5 #定义启动复制线程个数 21 # Number of reaper workers to start by default. 22 # (you don\'t usually need to increase this) 23 reaper_jobs = 1 #响应客户端请求，在磁盘失败后将请求重新放到队列中 24 # Number of fsck workers to start by default. 25 # (these can cause a lot of load when fsck\'ing) 26 #fsck_jobs = 1 #对磁盘进行检测，默认没有启动 27 # Minimum amount of space to reserve in megabytes 28 # default: 100 #默认保留100M空间 29 # Consider setting this to be larger than the largest file you 30 # would normally be uploading. 31 #min_free_space = 200 #最小空闲空间为200M 32 # Number of seconds to wait for a storage node to respond. 33 # default: 2 #等待存储节点相应的时间，默认为2s 34 # Keep this low, so busy storage nodes are quickly ignored. 35 #node_timeout = 2 #等待节点的超时时长 36 # Number of seconds to wait to connect to a storage node. 37 # default: 2 #连接存储节点的超时时长 38 # Keep this low so overloaded nodes get skipped. 39 #conn_timeout = 2 40 # Allow replication to use the secondary node get port, 41 # if you have apache or similar configured for GET\'s 42 #repl_use_get_port = 1
设置tracker对应的数据库，生成mogilefs库：
[root@mog1 ~]# mogdbsetup --dbhost=192.168.80.135 --dbport=3306 --dbname=mogilefs --dbrootuser=root --dbrootpass=rootpass --dbuser=moguser --dbpass=mogpass --yes #多个tracker共享一个库，所以其他tracker节点上就不需要在做此操作，直接修改配置文件/etc/mogilefs/mogilefs.conf就可以
注意：此处有一bug，本人也不知道是什么原因造成的：在生成mogilefs库的时候会出现错误，说root用户没对mogilefs库的访问权限（“Failed to grant privileges: Access denied for user \'root\'@\'192.168.80.%\' to database \'mogilefs\'”），但是再一次执行上面的设置数据库的命令时就会成功。前提是确保mariadb对用户授权是正确的。如果有哪位大神知道是什么原因造成的，求指点。。
到MariaDB节点上查看mogilefs库是否成功生成：
[root@Mariadb ~]# mysql
1 MariaDB [(none)]> show databases; 2 +--------------------+ 3 | Database | 4 +--------------------+ 5 | information_schema | 6 | mogilefs | 7 | mysql | 8 | performance_schema | 9 | test | 10 +--------------------+ 11 5 rows in set (0.00 sec) 12 13 MariaDB [(none)]> use mogilefs; 14 Database changed 15 MariaDB [mogilefs]> show tables; 16 +----------------------+ 17 | Tables_in_mogilefs | 18 +----------------------+ 19 | checksum | 20 | class | 21 | device | 22 | domain | 23 | file | 24 | file_on | 25 | file_on_corrupt | 26 | file_to_delete | 27 | file_to_delete2 | 28 | file_to_delete_later | 29 | file_to_queue | 30 | file_to_replicate | 31 | fsck_log | 32 | host | 33 | server_settings | 34 | tempfile | 35 | unreachable_fids | 36 +----------------------+ 37 17 rows in set (0.00 sec)
启动mogilefsd进程：
[root@mog1 ~]# service mogilefsd start
1 [root@mog1 ~]# ss -tnl | grep \'7001\' 2 LISTEN 0 128 *:7001 *:*
对mogstored进程进行配置：
[root@mog1 ~]# vim /etc/mogilefs/mogstored.conf maxconns = 10000 httplisten = 0.0.0.0:7500 mgmtlisten = 0.0.0.0:7501 docroot = /dfs/mogdata
将mogilefsd进程和mogstored进程的配置文件都同步到其他mog节点上去：
[root@mog1 ~]# scp /etc/mogilefs/mogilefsd.conf 192.168.80.137:/etc/mogilefs/ [root@mog1 ~]# scp /etc/mogilefs/mogilefsd.conf 192.168.80.138:/etc/mogilefs/ [root@mog1 ~]# scp /etc/mogilefs/mogstored.conf 192.168.80.137:/etc/mogilefs/ [root@mog1 ~]# scp /etc/mogilefs/mogstored.conf 192.168.80.138:/etc/mogilefs/
在存储节点上创建挂载点：
[root@mog1 ~]# mkdir /dfs/mogdata/dev1
[root@mog1 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/
[root@mog2 ~]# mkdir /dfs/mogdata/dev2 -pv
[root@mog2 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/
[root@mog3 ~]# mkdir /dfs/mogdata/dev3 -pv
[root@mog3 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/
注意：在生产环境中，需要将存储磁盘挂载在/dfs/mogdata目录下，当块磁盘挂载成功之后，需要在磁盘上创建共享块设备,即这里博主就不挂载磁盘了，直接创建共享块设备。
挂载磁盘的配置方法：
[root@mog1 ~]# mkdir /dfs/mogdata/
[root@mog1 ~]# mount -t ext4 /dev/sdb1 /dfs/mogdata/
[root@mog1 ~]# mkdir /dfs/mogdata/dev1
[root@mog1 ~]# chown -R mogilefs,mogilefs /dfs/mogdata/
启动mogstored进程：
[root@mog1 ~]# service mogstored start [root@mog2 ~]# service mogstored start [root@mog3 ~]# service mogstored start
[root@mog1 ~]# ss -tnlp | grep \'mogstored\' LISTEN 0 128 *:7500 *:* users:(("mogstored",2288,4)) LISTEN 0 128 *:7501 *:* users:(("mogstored",2288,9)) [root@mog2 ~]# ss -tnlp | grep \'mogstored\' LISTEN 0 128 *:7500 *:* users:(("mogstored",2288,4)) LISTEN 0 128 *:7501 *:* users:(("mogstored",2288,9)) [root@mog3 ~]# ss -tnlp | grep \'mogstored\' LISTEN 0 128 *:7500 *:* users:(("mogstored",2288,4)) LISTEN 0 128 *:7501 *:* users:(("mogstored",2288,9))
对各节点进行管理：
[root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog1 --ip=192.168.80.136 --status=alive [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog1 001 --status=alive [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog2 --ip=192.168.80.137 --status=alive [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog2 002 --status=alive [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog3 --ip=192.168.80.138 --status=alive [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog3 003 --status=alive [root@mog1 ~]# mogadm domain add linux1 [root@mog1 ~]# mogadm domain add python1 [root@mog1 ~]# mogadm class add linux1 class1 --mindevcount=3 [root@mog1 ~]# mogadm class add linux1 class2 --mindevcount=2 [root@mog1 ~]# mogadm class add python1 dx1 --mindevcount=2
[root@mog1 ~]# mogadm check Checking trackers... 127.0.0.1:7001 ... OK Checking hosts... [ 1] mog1 ... OK [ 2] mog2 ... OK [ 3] mog3 ... OK Checking devices... host device size(G) used(G) free(G) use% ob state I/O% ---- ------------ ---------- ---------- ---------- ------ ---------- ----- [ 1] dev1 16.509 3.901 12.608 23.63% writeable 4.4 [ 2] dev2 16.509 3.901 12.608 23.63% writeable 0.3 [ 3] dev3 16.509 3.897 12.612 23.60% writeable 0.0 ---- ------------ ---------- ---------- ---------- ------ total: 49.527 11.699 37.828 23.62% [root@mog1 ~]# mogadm domain list domain class mindevcount replpolicy hashtype -------------------- -------------------- ------------- ------------ ------- linux1 class1 3 MultipleHosts() NONE linux1 class2 2 MultipleHosts() NONE linux1 default 2 MultipleHosts() NONE python1 default 2 MultipleHosts() NONE python1 dx1 2 MultipleHosts() NONE
详细介绍其查看上一篇博客："分布式存储之MogileFS分布式文件系统简单应用"
上传文件进行测试：
[root@mog1 ~]# mogupload --trackers=192.168.80.136:7001 --domain=linux1 --key=\'/1.jpg\' --file=\'/root/centos.jpg\' [root@mog1 ~]# mogfileinfo --tracker=192.168.80.137:7001 --domain=linux1 --key=\'/1.jpg\' [root@mog1 ~]# mogupload --trackers=192.168.80.136:7001 --domain=python1 --key=\'/fstab.html\' --file=\'/etc/fstab\' [root@mog1 ~]# mogfileinfo --tracker=192.168.80.137:7001 --domain=python1 --key=\'/fstab.html\'
[root@mog1 ~]# mogfileinfo --tracker=192.168.80.137:7001 --domain=linux1 --key=\'1.jpg\' - file: 1.jpg class: default devcount: 2 domain: linux1 fid: 5 key: 1.jpg length: 134783 - http://192.168.80.137:7500/dev2/0/000/000/0000000005.fid - http://192.168.80.136:7500/dev1/0/000/000/0000000005.fid
配置Nginx:
[root@nginx ~]# # yum groupinstall "Development Tools" "Server Platform Deveopment" -y [root@nginx ~]# # yum install openssl-devel pcre-devel -y [root@nginx ~]# useradd -r nginx [root@nginx ~]# tar xf nginx_mogilefs_module-1.0.4.tar.gz [root@nginx ~]# tar xf nginx-1.10.0.tar.gz [root@nginx ~]# cd nginx-1.10.0 [root@nginx nginx-1.10.0]# ./configure \\ --prefix=/usr/local/nginx \\ --sbin-path=/usr/sbin/nginx \\ --conf-path=/etc/nginx/nginx.conf \\ --error-log-path=/var/log/nginx/error.log \\ --http-log-path=/var/log/nginx/access.log \\ --pid-path=/var/run/nginx/nginx.pid \\ --lock-path=/var/lock/nginx.lock \\ --user=nginx \\ --group=nginx \\ --with-http_ssl_module \\ --with-http_flv_module \\ --with-http_stub_status_module \\ --with-http_gzip_static_module \\ --http-client-body-temp-path=/var/tmp/nginx/client/ \\ --http-proxy-temp-path=/var/tmp/nginx/proxy/ \\ --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \\ --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \\ --http-scgi-temp-path=/var/tmp/nginx/scgi \\ --with-pcre \\ --with-debug \\ --add-module=/root/nginx_mogilefs_module-1.0.4 #添加的第三方模块，实现直接使用key作为uri进行访问 [root@nginx nginx-1.10.0]# make && make install
提供nginx启动脚本：
[root@nginx ~]# vim /etc/rc.d/init.d/nginx #!/bin/sh # # nginx - this script starts and stops the nginx daemon # # chkconfig: - 85 15 # description: Nginx is an HTTP(S) server, HTTP(S) reverse \\ # proxy and IMAP/POP3 proxy server # processname: nginx # config: /etc/nginx/nginx.conf # config: /etc/sysconfig/nginx # pidfile: /var/run/nginx.pid # Source function library. . /etc/rc.d/init.d/functions # Source networking configuration. . /etc/sysconfig/network # Check that networking is up. [ "$NETWORKING" = "no" ] && exit 0 nginx="/usr/sbin/nginx" prog=$(basename $nginx) NGINX_CONF_FILE="/etc/nginx/nginx.conf" [ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx lockfile=/var/lock/subsys/nginx make_dirs() { # make required directories user=`nginx -V 2>&1 | grep "configure arguments:" | sed \'s/[^*]*--user=\\([^ ]*\\).*/\\1/g\' -` options=`$nginx -V 2>&1 | grep \'configure arguments:\'` for opt in $options; do if [ `echo $opt | grep \'.*-temp-path\'` ]; then value=`echo $opt | cut -d "=" -f 2` if [ ! -d "$value" ]; then # echo "creating" $value mkdir -p $value && chown -R $user $value fi fi done } start() { [ -x $nginx ] || exit 5 [ -f $NGINX_CONF_FILE ] || exit 6 make_dirs echo -n $"Starting $prog: " daemon $nginx -c $NGINX_CONF_FILE retval=$? echo [ $retval -eq 0 ] && touch $lockfile return $retval } stop() { echo -n $"Stopping $prog: " killproc $prog -QUIT retval=$? echo [ $retval -eq 0 ] && rm -f $lockfile return $retval } restart() { configtest || return $? stop sleep 1 start } reload() { configtest || return $? echo -n $"Reloading $prog: " killproc $nginx -HUP RETVAL=$? echo } force_reload() { restart } configtest() { $nginx -t -c $NGINX_CONF_FILE } rh_status() { status $prog } rh_status_q() { rh_status >/dev/null 2>&1 } case "$1" in start) rh_status_q && exit 0 $1 ;; stop) rh_status_q || exit 0 $1 ;; restart|configtest) $1 ;; reload) rh_status_q || exit 7 $1 ;; force-reload) force_reload ;; status) rh_status ;; condrestart|try-restart) rh_status_q || exit 0 ;; *) echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}" exit 2 esac
[root@nginx ~]# chmod +x /etc/rc.d/init.d/nginx
修改nginx配置文件如下：
http { upstream images { server 192.168.80.136:7001; server 192.168.80.137:7001; server 192.168.80.138:7001; } server { listen 80; server_name localhost; location /image { mogilefs_tracker images; mogilefs_domain linux1; mogilefs_methods GET; mogilefs_noverify on; mogilefs_pass { proxy_pass $mogilefs_path; proxy_hide_header Content-Type; proxy_buffering off; } } location /files { mogilefs_tracker images; mogilefs_domain python1; mogilefs_methods GET; mogilefs_noverify on; mogilefs_pass { proxy_pass $mogilefs_path; proxy_hide_header Content-Type; proxy_buffering off; } } } }
启动nginx：
[root@nginx ~]# service nginx start
访问测试：
以上是关于分布式存储之MogileFS基于Nginx实现负载均衡（Nginx+MogileFS）的主要内容，如果未能解决你的问题，请参考以下文章
FastDFS + Nginx实现基于CentOS7平台的分布式文件存储与访问