分布式存储之MogileFS基于Nginx实现负载均衡(Nginx+MogileFS)

Posted 明天、未来

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了分布式存储之MogileFS基于Nginx实现负载均衡(Nginx+MogileFS)相关的知识,希望对你有一定的参考价值。

MogileFS分布式文件系统特点:

  1.具有raid的性能

  2.不存在单点故障

  3.简单的命名空间: 每个文件对应一个key:用于domain定义名称空间

      4.不共享任何数据

      5.传输中立,无特殊协议:可以通过NFS或HTTP进行通信

      6.自动文件复制:复制的最小单位不是文件,而class

  7.应用层: 用户空间文件系统,无须特殊的核心组件

 

Nginx+MogileFS的好处:

  1、将请求代理至后端MogileFS服务器集群中,能实现负载均衡的效果。

   2、能对后端的tracker节点进行健康检测。

   3、将第三方模块“nginx_mogilefs_module”编译进Nginx中,能实现直接使用key访问对应的文件,如下:

        使用nginx做代理之前:http://192.168.80.137:7500/dev2/0/000/000/0000000007.fid

        使用nginx做代理之后:http://192.168.80.132/image/1.jpg

 

MogileFS是由三个组件组成的:

  1、tracker:MogileFS的核心,是一个调度器,服务进程为mogilefsd,职责:删除数据、复制数据、监控、查询等。

  2、database:为tracker存储元素据

  3、数据存储的位置,通常是一个HTTP(webDAV)服务器,用来做数据的创建(put)、删除(delete)、获取(get),监听端口7500, storage节点使用http进行数据传输, 依赖于perbal, 进程为mogstored。

 

理想中模型:

 

 

 

实验架构:

 

 

实验中用到的rpm包:http://pan.baidu.com/s/1c1S0dJM 

 

MariaDB节点配置:

对各mogilefs节点的主机名进行解析:

[root@www ~]# vim /etc/hosts
     192.168.80.136 mog1.daixiang.com
     192.168.80.137 mog2.daixiang.com
     192.168.80.138 mog3.daixiang.com
                           
View Code

 

二进制包安装MariaDB:

[root@mariadb mysql]# useradd -r -s /sbin/nologin mysql
[root@mariadb ~]# tar xf mariadb-10.1.14-linux-x86_64.tar.gz -C /usr/local/
[root@mariadb ~]# ln -sv /usr/local/mariadb-10.1.14-linux-x86_64 /usr/local/mysql
[root@mariadb mysql]# chown -R mysql.mysql /usr/local/mysql/
[root@mariadb mysql]# ./scripts/mysql_install_db --user=mysql --datadir=/data
[root@mariadb mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@mariadb mysql]# chmod +x /etc/rc.d/init.d/mysqld
[root@mariadb mysql]# cp support-files/my-large.cnf /etc/my.cnf 
[root@mariadb mysql]# vim /etc/my.cnf
         datadir = /data
[root@mariadb mysql]# ln -sv /usr/local/mysql/include/ /usr/include/mysql/
[root@mariadb mysql]# vim /etc/ld.so.conf.d/mysql.conf
         /usr/local/mysql/lib
[root@mariadb mysql]# vim /etc/profile.d/mysql.sh 
         export PATH=/usr/local/mysql/bin:$PATH
[root@mariadb mysql]# ldconfig 
[root@mariadb mysql]# ldconfig -p | grep mysql
	libmysqld.so.18 (libc6,x86-64) => /usr/local/mysql/lib/libmysqld.so.18
	libmysqld.so (libc6,x86-64) => /usr/local/mysql/lib/libmysqld.so
	libmysqlclient_r.so.16 (libc6,x86-64) => /usr/lib64/mysql/libmysqlclient_r.so.16
	libmysqlclient.so.18 (libc6,x86-64) => /usr/local/mysql/lib/libmysqlclient.so.18
	libmysqlclient.so.16 (libc6,x86-64) => /usr/lib64/mysql/libmysqlclient.so.16
	libmysqlclient.so (libc6,x86-64) => /usr/local/mysql/lib/libmysqlclient.so
	libgalera_smm.so (libc6,x86-64) => /usr/local/mysql/lib/libgalera_smm.so

   

对用户进行授权:

MariaDB [(none)]> grant all on *.* to \'root\'@\'192.168.80.%\' identified by \'rootpass\';
Query OK, 0 rows affected (0.06 sec)

MariaDB [(none)]> grant all on mogilefs.* to \'moguser\'@\'192.168.80.%\' identified by \'mogpass\';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

  

MogileFS配置:

[root@mog1 ~]# yum install *.rpm -y
[root@mog1 ~]# yum install perl-IO-AIO -y

[root@mog2 ~]# yum install *.rpm -y
[root@mog2 ~]# yum install perl-IO-AIO -y

[root@mog3 ~]# yum install *.rpm -y
[root@mog3 ~]# yum install perl-IO-AIO -y 

 

修改mogilefsd进程的配置文件:

 1 [root@mog1 ~]# vim /etc/mogilefs/mogilefsd.conf
 2 
 3 # Enable daemon mode to work in background and use syslog
 4 daemonize = 1                                           #以守护进程的形式运行
 5 # Where to store the pid of the daemon (must be the same in the init script)
 6 pidfile = /var/run/mogilefsd/mogilefsd.pid
 7 # Database connection information
 8 db_dsn = DBI:mysql:mogilefs:host=192.168.80.135      #定义数据库名为mogilefs和数据库服务器地址
 9 db_user = moguser                                     #定义管理此数据库的用户名
10 db_pass = mogpass                                     #定义密码
11 # IP:PORT to listen on for mogilefs client requests
12 listen = 0.0.0.0:7001                                  #定义监听的地址和端口
13 # Optional, if you don\'t define the port above.
14 conf_port = 7001                                                            
15 # Number of query workers to start by default.
16 query_jobs = 10                                         #定义启动查询线程个数
17 # Number of delete workers to start by default.
18 delete_jobs = 1                                          #定义启动删除线程个数
19 # Number of replicate workers to start by default.
20 replicate_jobs = 5                                       #定义启动复制线程个数
21 # Number of reaper workers to start by default.
22 # (you don\'t usually need to increase this)
23 reaper_jobs = 1                            #响应客户端请求,在磁盘失败后将请求重新放到队列中
24 # Number of fsck workers to start by default.
25 # (these can cause a lot of load when fsck\'ing)
26 #fsck_jobs = 1                              #对磁盘进行检测,默认没有启动
27 # Minimum amount of space to reserve in megabytes
28 # default: 100                               #默认保留100M空间
29 # Consider setting this to be larger than the largest file you
30 # would normally be uploading.
31 #min_free_space = 200                #最小空闲空间为200M
32 # Number of seconds to wait for a storage node to respond.
33 # default: 2                                   #等待存储节点相应的时间,默认为2s
34 # Keep this low, so busy storage nodes are quickly ignored.
35 #node_timeout = 2                        #等待节点的超时时长
36 # Number of seconds to wait to connect to a storage node.
37 # default: 2                                     #连接存储节点的超时时长
38 # Keep this low so overloaded nodes get skipped.
39 #conn_timeout = 2                         
40 # Allow replication to use the secondary node get port,
41 # if you have apache or similar configured for GET\'s
42 #repl_use_get_port = 1
View Code

 

设置tracker对应的数据库,生成mogilefs库: 

[root@mog1 ~]# mogdbsetup --dbhost=192.168.80.135 --dbport=3306 --dbname=mogilefs --dbrootuser=root --dbrootpass=rootpass --dbuser=moguser --dbpass=mogpass --yes

#多个tracker共享一个库,所以其他tracker节点上就不需要在做此操作,直接修改配置文件/etc/mogilefs/mogilefs.conf就可以

    注意:此处有一bug,本人也不知道是什么原因造成的:在生成mogilefs库的时候会出现错误,说root用户没对mogilefs库的访问权限(“Failed to grant privileges: Access denied for user \'root\'@\'192.168.80.%\' to database \'mogilefs\'”),但是再一次执行上面的设置数据库的命令时就会成功。前提是确保mariadb对用户授权是正确的。如果有哪位大神知道是什么原因造成的,求指点。。  

 

 到MariaDB节点上查看mogilefs库是否成功生成:

[root@Mariadb ~]# mysql  
 1 MariaDB [(none)]> show databases;
 2 +--------------------+
 3 | Database           |
 4 +--------------------+
 5 | information_schema |
 6 | mogilefs           |
 7 | mysql              |
 8 | performance_schema |
 9 | test               |
10 +--------------------+
11 5 rows in set (0.00 sec)
12 
13 MariaDB [(none)]> use mogilefs;
14 Database changed
15 MariaDB [mogilefs]> show tables;
16 +----------------------+
17 | Tables_in_mogilefs   |
18 +----------------------+
19 | checksum             |
20 | class                |
21 | device               |
22 | domain               |
23 | file                 |
24 | file_on              |
25 | file_on_corrupt      |
26 | file_to_delete       |
27 | file_to_delete2      |
28 | file_to_delete_later |
29 | file_to_queue        |
30 | file_to_replicate    |
31 | fsck_log             |
32 | host                 |
33 | server_settings      |
34 | tempfile             |
35 | unreachable_fids     |
36 +----------------------+
37 17 rows in set (0.00 sec)
View Code

 

启动mogilefsd进程:

[root@mog1 ~]# service mogilefsd start
1 [root@mog1 ~]# ss -tnl | grep \'7001\'
2 LISTEN     0      128                       *:7001                     *:*  
View Code

 

对mogstored进程进行配置:

[root@mog1 ~]# vim /etc/mogilefs/mogstored.conf

maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /dfs/mogdata
View Code

 

将mogilefsd进程和mogstored进程的配置文件都同步到其他mog节点上去:

[root@mog1 ~]# scp /etc/mogilefs/mogilefsd.conf 192.168.80.137:/etc/mogilefs/
[root@mog1 ~]# scp /etc/mogilefs/mogilefsd.conf 192.168.80.138:/etc/mogilefs/

[root@mog1 ~]# scp /etc/mogilefs/mogstored.conf 192.168.80.137:/etc/mogilefs/
[root@mog1 ~]# scp /etc/mogilefs/mogstored.conf 192.168.80.138:/etc/mogilefs/

 

在存储节点上创建挂载点

[root@mog1 ~]# mkdir /dfs/mogdata/dev1
[root@mog1 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/

[root@mog2 ~]# mkdir /dfs/mogdata/dev2 -pv
[root@mog2 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/

[root@mog3 ~]# mkdir /dfs/mogdata/dev3 -pv
[root@mog3 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/


    注意:在生产环境中,需要将存储磁盘挂载在/dfs/mogdata目录下,当块磁盘挂载成功之后,需要在磁盘上创建共享块设备,即这里博主就不挂载磁盘了,直接创建共享块设备。 
            挂载磁盘的配置方法:
                          [root@mog1 ~]# mkdir /dfs/mogdata/
                          [root@mog1 ~]# mount -t ext4 /dev/sdb1 /dfs/mogdata/
                          [root@mog1 ~]# mkdir /dfs/mogdata/dev1
                          [root@mog1 ~]# chown -R mogilefs,mogilefs /dfs/mogdata/
                          

 

启动mogstored进程:

[root@mog1 ~]# service mogstored start
[root@mog2 ~]# service mogstored start
[root@mog3 ~]# service mogstored start
[root@mog1 ~]# ss -tnlp | grep \'mogstored\'
LISTEN     0      128                       *:7500                     *:*      users:(("mogstored",2288,4))
LISTEN     0      128                       *:7501                     *:*      users:(("mogstored",2288,9))

[root@mog2 ~]# ss -tnlp | grep \'mogstored\'
LISTEN     0      128                       *:7500                     *:*      users:(("mogstored",2288,4))
LISTEN     0      128                       *:7501                     *:*      users:(("mogstored",2288,9))

[root@mog3 ~]# ss -tnlp | grep \'mogstored\'
LISTEN     0      128                       *:7500                     *:*      users:(("mogstored",2288,4))
LISTEN     0      128                       *:7501                     *:*      users:(("mogstored",2288,9))
View Code

  

对各节点进行管理:

[root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog1 --ip=192.168.80.136 --status=alive
[root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog1 001 --status=alive

[root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog2 --ip=192.168.80.137 --status=alive
[root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog2 002 --status=alive

[root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog3 --ip=192.168.80.138 --status=alive
[root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog3 003 --status=alive

[root@mog1 ~]# mogadm domain add linux1 
[root@mog1 ~]# mogadm domain add python1

[root@mog1 ~]# mogadm class add linux1 class1 --mindevcount=3
[root@mog1 ~]# mogadm class add linux1 class2 --mindevcount=2
[root@mog1 ~]# mogadm class add python1 dx1 --mindevcount=2
[root@mog1 ~]# mogadm check

Checking trackers...
  127.0.0.1:7001 ... OK

Checking hosts...
  [ 1] mog1 ... OK
  [ 2] mog2 ... OK
  [ 3] mog3 ... OK

Checking devices...
  host device         size(G)    used(G)    free(G)   use%   ob state   I/O%
  ---- ------------ ---------- ---------- ---------- ------ ---------- -----
  [ 1] dev1            16.509      3.901     12.608  23.63%  writeable   4.4
  [ 2] dev2            16.509      3.901     12.608  23.63%  writeable   0.3
  [ 3] dev3            16.509      3.897     12.612  23.60%  writeable   0.0
  ---- ------------ ---------- ---------- ---------- ------
             total:    49.527     11.699     37.828  23.62%



[root@mog1 ~]# mogadm domain list 
 domain               class                mindevcount   replpolicy   hashtype
-------------------- -------------------- ------------- ------------ -------
 linux1               class1                    3        MultipleHosts() NONE   
 linux1               class2                    2        MultipleHosts() NONE   
 linux1               default                   2        MultipleHosts() NONE   

 python1              default                   2        MultipleHosts() NONE   
 python1              dx1                       2        MultipleHosts() NONE   
View Code

  详细介绍其查看上一篇博客:"分布式存储之MogileFS分布式文件系统简单应用"

 

上传文件进行测试:

[root@mog1 ~]# mogupload --trackers=192.168.80.136:7001 --domain=linux1 --key=\'/1.jpg\' --file=\'/root/centos.jpg\'
[root@mog1 ~]# mogfileinfo --tracker=192.168.80.137:7001 --domain=linux1 --key=\'/1.jpg\'

[root@mog1 ~]# mogupload --trackers=192.168.80.136:7001 --domain=python1 --key=\'/fstab.html\' --file=\'/etc/fstab\'
[root@mog1 ~]# mogfileinfo --tracker=192.168.80.137:7001 --domain=python1 --key=\'/fstab.html\'  
[root@mog1 ~]# mogfileinfo --tracker=192.168.80.137:7001 --domain=linux1 --key=\'1.jpg\'
- file: 1.jpg
     class:              default
  devcount:                    2
    domain:               linux1
       fid:                    5
       key:                1.jpg
    length:               134783
 - http://192.168.80.137:7500/dev2/0/000/000/0000000005.fid
 - http://192.168.80.136:7500/dev1/0/000/000/0000000005.fid
View Code

 

配置Nginx:

[root@nginx ~]# # yum groupinstall "Development Tools" "Server Platform Deveopment" -y
[root@nginx ~]# # yum install openssl-devel pcre-devel -y
[root@nginx ~]# useradd -r nginx
[root@nginx ~]# tar xf nginx_mogilefs_module-1.0.4.tar.gz 
[root@nginx ~]# tar xf nginx-1.10.0.tar.gz 
[root@nginx ~]# cd nginx-1.10.0
[root@nginx nginx-1.10.0]#  ./configure \\
  --prefix=/usr/local/nginx \\
  --sbin-path=/usr/sbin/nginx \\
  --conf-path=/etc/nginx/nginx.conf \\
  --error-log-path=/var/log/nginx/error.log \\
  --http-log-path=/var/log/nginx/access.log \\
  --pid-path=/var/run/nginx/nginx.pid  \\
  --lock-path=/var/lock/nginx.lock \\
  --user=nginx \\
  --group=nginx \\
  --with-http_ssl_module \\
  --with-http_flv_module \\
  --with-http_stub_status_module \\
  --with-http_gzip_static_module \\
  --http-client-body-temp-path=/var/tmp/nginx/client/ \\
  --http-proxy-temp-path=/var/tmp/nginx/proxy/ \\
  --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \\
  --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \\
  --http-scgi-temp-path=/var/tmp/nginx/scgi \\
  --with-pcre \\
  --with-debug \\
  --add-module=/root/nginx_mogilefs_module-1.0.4    #添加的第三方模块,实现直接使用key作为uri进行访问

[root@nginx nginx-1.10.0]#  make && make install

  

提供nginx启动脚本:

[root@nginx ~]# vim /etc/rc.d/init.d/nginx

#!/bin/sh
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig:   - 85 15 
# description:  Nginx is an HTTP(S) server, HTTP(S) reverse \\
#               proxy and IMAP/POP3 proxy server
# processname: nginx
# config:      /etc/nginx/nginx.conf
# config:      /etc/sysconfig/nginx
# pidfile:     /var/run/nginx.pid
 
# Source function library.
. /etc/rc.d/init.d/functions
 
# Source networking configuration.
. /etc/sysconfig/network
 
# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0
 
nginx="/usr/sbin/nginx"
prog=$(basename $nginx)
 
NGINX_CONF_FILE="/etc/nginx/nginx.conf"
 
[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx
 
lockfile=/var/lock/subsys/nginx
 
make_dirs() {
   # make required directories
   user=`nginx -V 2>&1 | grep "configure arguments:" | sed \'s/[^*]*--user=\\([^ ]*\\).*/\\1/g\' -`
   options=`$nginx -V 2>&1 | grep \'configure arguments:\'`
   for opt in $options; do
       if [ `echo $opt | grep \'.*-temp-path\'` ]; then
           value=`echo $opt | cut -d "=" -f 2`
           if [ ! -d "$value" ]; then
               # echo "creating" $value
               mkdir -p $value && chown -R $user $value
           fi
       fi
   done
}
 
start() {
    [ -x $nginx ] || exit 5
    [ -f $NGINX_CONF_FILE ] || exit 6
    make_dirs
    echo -n $"Starting $prog: "
    daemon $nginx -c $NGINX_CONF_FILE
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}
 
stop() {
    echo -n $"Stopping $prog: "
    killproc $prog -QUIT
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}
 
restart() {
    configtest || return $?
    stop
    sleep 1
    start
}
 
reload() {
    configtest || return $?
    echo -n $"Reloading $prog: "
    killproc $nginx -HUP
    RETVAL=$?
    echo
}
 
force_reload() {
    restart
}
 
configtest() {
  $nginx -t -c $NGINX_CONF_FILE
}
 
rh_status() {
    status $prog
}
 
rh_status_q() {
    rh_status >/dev/null 2>&1
}
 
case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart|configtest)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
            ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
        exit 2
esac
View Code
[root@nginx ~]# chmod +x /etc/rc.d/init.d/nginx  

 

修改nginx配置文件如下:

http {

    upstream images {
        server 192.168.80.136:7001;
        server 192.168.80.137:7001;
        server 192.168.80.138:7001;
    }

    server {
        listen       80;
        server_name  localhost;
        location /image {
            mogilefs_tracker images;
            mogilefs_domain linux1;
            mogilefs_methods GET;
            mogilefs_noverify on;
            mogilefs_pass {
                proxy_pass $mogilefs_path;
                proxy_hide_header Content-Type;
                proxy_buffering off;
            }
        }

        location /files {
            mogilefs_tracker images;
            mogilefs_domain python1;
            mogilefs_methods GET;
            mogilefs_noverify on;
            mogilefs_pass {
                proxy_pass $mogilefs_path;
                proxy_hide_header Content-Type;
                proxy_buffering off;
            }
        }
    }
}   
View Code

 

启动nginx:

[root@nginx ~]# service nginx start

 

访问测试:

 

 

 

 

  

  

 

 

 

以上是关于分布式存储之MogileFS基于Nginx实现负载均衡(Nginx+MogileFS)的主要内容,如果未能解决你的问题,请参考以下文章

MogileFS基于Nginx反向代理实现分布式存储与访问

FastDFS + Nginx实现基于CentOS7平台的分布式文件存储与访问

Linux之使用MogileFS分布式文件系统并使用nginx实现反向代理

mogilefs-企业级分布式存储应用与实战

新手必读:Linux分布式存储之——Mogilefs

Nginx反代Mogilefs分布式储存示例