Nginx+MogileFS分布式文件系统部署实验

Posted 2020-09-02

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Nginx+MogileFS分布式文件系统部署实验相关的知识，希望对你有一定的参考价值。

实验要求

1、部署mogilefs分布式文件系统

2、 nginx代理

实验环境

1、拓扑

2、网络地址规划

node1:172.16.76.10 nginx （centos6.8）

node2:172.16.76.20 Trackers 、storage Node （centos7.2）

node3:172.16.76.30 storage Node （centos7.2）

node4:172.16.76.40 storage Node (Mariadb) （centos7.2）

实验准备

同步时间、关闭iptables、selinux等。同时自己配置/etc/hots

基本配置

node1

1、nginx配置

a、解决依赖关系

# yum groupinstall "DevelopmentTools" "Server Platform Deveopment"

# yum install openssl-devel pcre-devel

b、安装

首先添加用户nginx，实现以之运行nginx服务进程：

# groupadd -r nginx

# useradd -r -g nginx nginx

c、开始编译和安装：

[[email protected] ~]# tar -xvfnginx-1.10.3.tar.gz
[[email protected] ~]# tar -xvfnginx_mogilefs_module-1.0.4.tar.gz
[[email protected] ~]# cd nginx-1.10.3/
[[email protected] nginx-1.10.3]#  ./configure  --prefix=/usr --sbin-path=/usr/sbin/nginx  --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log  --pid-path=/var/run/nginx/nginx.pid   --lock-path=/var/lock/nginx.lock  --user=nginx   --group=nginx  --with-http_ssl_module  --with-http_flv_module  --with-http_stub_status_module --with-http_gzip_static_module --http-client-body-temp-path=/var/tmp/nginx/client/ --http-proxy-temp-path=/var/tmp/nginx/proxy/ --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi --http-scgi-temp-path=/var/tmp/nginx/scgi  --with-pcre  --with-debug  --add-module=/root/nginx_mogilefs_module-1.0.4/
[[email protected] nginx-1.10.3]# make &&make install
[[email protected] ~]#vim /etc/rc.d/init.d/nginx
         #脚本内容见文章后附件。
[[email protected] ~]#chmod +x /etc/rc.d/init.d/nginx
[[email protected] ~]#chkconfig –add nginx
[[email protected] ~]#service nginx start

node1为centos6

node2

[[email protected] ~]# ls mogilefs/

MogileFS-Server-2.46-2.el6.noarch.rpm MogileFS-Utils-2.19-1.el6.noarch.rpm perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm

MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm Perlbal-1.78-1.el6.noarch.rpm perl-MogileFS-Client-1.14-1.el6.noarch.rpm

MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm Perlbal-doc-1.78-1.el6.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm

[[email protected] ~]# for i in {2,3,4} ;do scp -r mogilefs node$i:/root/;done

#将mogilefs拷贝至其余各节点

[[email protected] ~]# for i in {2,3,4};do sshnode$i ‘yum install perl-Net-Netmask perl-IO-stringy perl-Sys-Syslogperl-IO-AIO -y‘;done #各节点安装

[[email protected] ~]# for i in {2,3,4};do sshnode$i ‘yum install ./mogilefs/*.rpm -y‘;done #各节点安装

node4:

[[email protected] ~]# yum install mariadb-server –y

2、初始化数据

a、数据库授权

[[email protected] ~]#vim /etc/my.cnf

skip_name_resolve=ON

innodb_file_per_table=ON

[[email protected] ~]# systemctl start mariadb

[[email protected] ~]# mysql

MariaDB [mogilefs]> grant all ON *.* [email protected]‘172.16.%.%‘ identified by ‘mogilepass‘

MariaDB [mogilefs]> GRANT ALL PRIVILEGES ON mogilefs.*TO [email protected]‘172.16.%.%‘ IDENTIFIED BY ‘mogile’ WITH GRANT OPTION;

MariaDB [mogilefs]> flush privileges;

b、初始化数据

[[email protected] ~]# mogdbsetup --dbhost=172.16.76.40 --dbname=mogilefs--dbrootuser=mogilefs -- dbrootpass=mogilefs --dbuser=mogile --dbpass=mogile --yes

3、设定连接数据库

[[email protected] ~]# vim /etc/mogilefs/mogilefsd.conf

db_dsn =DBI:mysql:mogilefs:host=172.16.76.40

db_user = mogile

db_pass =mogile

# IP:PORT tolisten on for mogilefs client requests

listen = 0.0.0.0:7001

[[email protected]~]# service mogilefsd start

存储节点配置

node2

[[email protected] ~]# vim/etc/mogilefs/mogstored.conf

maxconns = 10000

httplisten = 0.0.0.0:7500

mgmtlisten = 0.0.0.0:7501

docroot = /apps/data/mogilefs

[[email protected] ~]#mkdir -pv/apps/data/mogilefs

[[email protected] ~]# chown -R mogilefs.mogilefs /apps/data/mogilefs/

启动服务报错

#此处是因为启动脚本问题

[[email protected] ~]# vim /etc/init.d/mogstored

图中红线标记处为添加内容，修改保存后再次启动服务正常

[[email protected] ~]# systemctl daemon-reload

文章后会将修改后脚本上传为附件。

[[email protected] ~]# for i in {3,4}; do scp/etc/mogilefs/mogstored.conf node$i:/etc/mogilefs/;done

#将此文件传递到node3、node4节点（配置相同）

[[email protected] ~]#for i in {3,4}; do scp/etc/init.d/mogstored node$i:/etc/init.d/ ;done

#修改后的启动脚本

[[email protected] ~]# for i in {3.4}; do sshnode$i ‘mkdir -pv /apps/data/mogilefs‘ ;done

#node3、node4节点创建存储目录

[[email protected]~]# for i in {3,4}; do ssh node$i ‘chown -R mogilefs.mogilefs /apps/data/mogilefs/‘ ;done#权限

[[email protected]~]# for i in {3,4}; do ssh node$i ‘systemctl daemon-reload‘ ;done

[[email protected]~]# for i in {3,4}; do ssh node$i ‘ service mogstored start‘ ;done

#启动各节点 mostored服务。

添加存储节点主机

[[email protected] ~]# mogadm host add172.16.76.20 --ip=172.16.76.20 --port=7500 --status=alive

[[email protected] ~]# mogadm host add172.16.76.30 --ip=172.16.76.30 --port=7500 --status=alive

[[email protected] ~]# mogadm host add172.16.76.40 --ip=172.16.76.40 --port=7500 --status=alive

添加存储设备

node2:

[[email protected] ~]# cd /apps/data/mogilefs/

[[email protected] mogilefs]# mkdir dev1

[[email protected] mogilefs]# chown -R mogilefs.mogilefs /apps/data/mogilefs/

node3

[[email protected] ~]# cd /apps/data/mogilefs/

[[email protected] mogilefs]# mkdir dev2

[[email protected] mogilefs]# chown -R mogilefs.mogilefs /apps/data/mogilefs/

node4

[[email protected] ~]# cd /apps/data/mogilefs/

[[email protected] mogilefs]# mkdir dev3

[[email protected] mogilefs]# chown -R mogilefs.mogilefs /apps/data/mogilefs/

node2:

[[email protected] ~]# mogadm device add172.16.76.20 1

[[email protected] ~]# mogadm device add172.16.76.30 2

[[email protected] ~]# mogadm device add172.16.76.40 3

域（名称空间）、类添加

添加域

node2

[[email protected]~]# mogadm domain add file #添加域 file

[[email protected] ~]#mogadm domain add imgs #添加域imgs

[[email protected] ~]#mogadm domain list #域列表

添加类

node2

[[email protected]~]# mogadm class add imgs jpg --hashtype=MD5 --mindevcount=2

#添加类jpg ，MD5算法最小文件分数为2

[[email protected] ~]#mogadm class add imgs png --hashtype=MD5 --mindevcount=2

[[email protected] ~]#mogadm class list #类列表

测试存储

上传文件

[[email protected]~]# mogupload --trackers=172.16.76.20:7001 --domain=imgs--class=jpg --key=‘/default.jpg‘ --file=‘/root/test.jpg‘

[[email protected]~]# mogupload --trackers=172.16.76.20 --domain=file --key=‘t1.html‘--file=‘/root/anaconda-ks.cfg‘

查询文件

[[email protected] ~]# mogfileinfo--trackers=172.16.76.20:7001 --domain=imgs --key=‘default.jpg‘

#注意此处显示devcount这是有问题，文章最后有说明

查看详细信息

[[email protected]~]# mogfiledebug --trackers=172.16.76.20:7001 --domain=imgs --key=‘default.jpg‘

浏览器测试查看

http://172.16.76.20:7500/dev1/0/000/000/0000000041.fid

此时说明mogilefs已经可以正常使用。

配置Nginx代理

[[email protected] ~]vim /etc/nginx/nginx.conf

#user nobody;
worker_processes  1;
 
#error_log logs/error.log;
#error_log logs/error.log  notice;
#error_log logs/error.log  info;
#pid       logs/nginx.pid;
events {
   worker_connections  1024;
}
http {
   include       mime.types;
   default_type application/octet-stream;
 
   #log_format  main  ‘$remote_addr - $remote_user [$time_local]"$request" ‘
   #                  ‘$status$body_bytes_sent "$http_referer" ‘
   #                 ‘"$http_user_agent" "$http_x_forwarded_for"‘;
 
   #access_log  logs/access.log  main;
 
   sendfile        on;
   #tcp_nopush     on;
 
   #keepalive_timeout  0;
   keepalive_timeout  65;
 
   #gzip  on;
 
         server{
             listen       80; 
             server_name yourdomain.com default;
 
             location /imgs  { 
             mogilefs_tracker 172.16.76.20:7001;  #代理trackers
             mogilefs_domain imgs;  #imgs域
             mogilefs_methods get;  #请求方法
 
             mogilefs_pass {
                  proxy_pass $mogilefs_path;  #mogilefs模块内置变量
                  proxy_hide_headerContent-Type;
                  proxy_buffering off;
             }   
             expires 1h; 
             }   
     location /file/ { 
             mogilefs_tracker 172.16.76.20:7001;  
             mogilefs_domain file;  #file域
             mogilefs_methods get; 
 
             mogilefs_pass {
                       proxy_pass $mogilefs_path;  
                      proxy_hide_headerContent-Type;
                       proxy_buffering off;
             }   
             expires 1h; 
             }
             error_page  500 502 503 504  /50x.html;
             location = /50x.html {
                root   html;
             }  
         }
}

[[email protected] ~] nginx -t

nginx: the configuration file/etc/nginx/nginx.conf syntax is ok

nginx: configuration file/etc/nginx/nginx.conf test is successful

[[email protected] ~]nginx -s reload

测试