fastdfs集群

Posted 2020-08-28

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了fastdfs集群相关的知识，希望对你有一定的参考价值。

基础环境说明 6台服务器两台tracker 4台storages

192.168.56.131	tracker 服务器	edu-dfs-tracker-1
192.168.56.132	tracker 服务器	edu-dfs-tracker-2
192.168.56.135	storage服务器	edu-dfs-storage-group1-1
192.168.56.136	storage服务器	edu-dfs-storage-group1-2
192.168.56.137	storage服务器	edu-dfs-storage-group2-1
192.168.56.138	storage服务器	edu-dfs-storage-group2-2

[[email protected] ~]# cat  /etc/hosts
192.168.56.131 edu-dfs-tracker-1
192.168.56.132 edu-dfs-tracker-2
192.168.56.135 edu-dfs-storage-group1-1
192.168.56.136 edu-dfs-storage-group1-2
192.168.56.137 edu-dfs-storage-group2-1
192.168.56.138 edu-dfs-storage-group2-2
[[email protected] ~]#

2.安装依赖（所有服务器）

yum  -y install perl perl-devel  make  gcc-c++  gcc  cmake

3.安装fastdfs所需要的common库（所有的服务器）

cd  /usr/local/src
 unzip  libfastcommon-master.zip 
 cd libfastcommon-master
 ./make.sh 
 ./make.sh  install

4.做软连接（所有机器）

ln -s  /usr/lib64/libfastcommon.so   /usr/local/lib/libfastcommon.so
ln -s  /usr/lib64/libfastcommon.so   /usr/lib/libfastcommon.so
ln  -s  /usr/lib64/libfdfsclient.so  /usr/local/lib/libfdfsclient.so
ln  -s  /usr/lib64/libfdfsclient.so  /usr/lib/libfdfsclient.so

5.安装fastdfs（所有机器）

cd  /usr/local/src
tar xf fastdfs-5.05.tar.gz
cd fastdfs-5.05
./make.sh
./make.sh install

6.修改启动脚本（启动脚本是在/usr/local/bin 实际是在/usr/bin）

 tracker服务器两台
 sed -i ‘s#/usr/local/bin#/usr/bin#g‘  /etc/init.d/fdfs_trackerd
 storage服务器四台
 sed -i ‘s#/usr/local/bin#/usr/bin#g‘  /etc/init.d/fdfs_storaged

7.下面来配置tracker服务器两台（tracker两台服务器全都要做）

cd  /etc/fdfs
cp storage.conf.sample  storage.conf
vi  storage.conf

disabled=false
port=22122
base_path=/fastdfs/tracker 

保存以后
mkdir  -p  /fastdfs/tracker
/etc/init.d/fdfs_trackerd   start
chkconfig fdfs_trackerd  on

启动以后的结果,会生成data  logs 这两个文件
[[email protected] fdfs]# cd  /fastdfs/tracker/
[[email protected] tracker]# ls
data  logs
[[email protected] tracker]# 
查看监听的端口22122
[[email protected] tracker]# netstat -lntup 
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22122           0.0.0.0:*               LISTEN      19522/fdfs_trackerd 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      786/sshd            
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      870/master          
tcp6       0      0 :::22                   :::*                    LISTEN      786/sshd            
tcp6       0      0 ::1:25                  :::*                    LISTEN      870/master          
[[email protected] tracker]# 
查看启动日志：
[[email protected] logs]# tail -f /fastdfs/tracker/logs/trackerd.log 
[2017-01-22 05:20:51] INFO - FastDFS v5.05, base_path=/fastdfs/tracker, run_by_grou

8.下面来配置storage服务器两台（storage四台服务器全都要做）（同一组的做的不一样）

cd  /etc/fdfs
cp storage.conf.sample  storage.conf

group1 的服务器 group_name=group1
group2 的服务器 group_name=group2  其余的不变
mdkir  -p  /fastdfs/storage
监听日志
cd /fastdfs/storage/logs
[[email protected] logs]# tail -f storaged.log 
mkdir data path: FD ...
mkdir data path: FE ...
mkdir data path: FF ...
data path: /fastdfs/storage/data, mkdir sub dir done.
[2017-01-22 06:07:17] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0
[2017-01-22 06:07:17] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.56.137, my_server_id_str: 192.168.56.137, g_server_id_in_filename: -1992775488
[2017-01-22 06:07:19] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.56.132:22122, as a tracker client, my ip is 192.168.56.137
[2017-01-22 06:07:19] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.56.131:22122, as a tracker client, my ip is 192.168.56.137
[2017-01-22 06:07:19] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.56.132:22122, set tracker leader: 192.168.56.132:22122
[2017-01-22 06:07:19] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.56.138:23000

特别说明：一台storage只会选择一个tracker服务器，一台tracker服务器挂掉，会接着选择另外一台tracker服务器

加入开机自启，以上所有的端口都要在防火墙允许放行

echo "/etc/init.d/fdfs_storaged  start " >>/etc/rc.local

9查看storage服务器的具体信心，可以用于监控，任意一台storage服务器

[[email protected] logs]# /usr/bin/fdfs_monitor /etc/fdfs/storage.conf
[2017-01-22 06:32:36] DEBUG - base_path=/fastdfs/storage, connect_timeout=30, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=2, server_index=0

tracker server is 192.168.56.131:22122

group count: 2

Group 1:
group name = group1
disk total space = 38345 MB
disk free space = 36848 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

	Storage 1:
		id = 192.168.56.135
		ip_addr = 192.168.56.135 (edu-dfs-storage-group1-1)  ACTIVE
		http domain = 
		version = 5.05
		join time = 2017-01-22 05:59:38
		up time = 2017-01-22 06:15:24
		total storage = 38345 MB
		free storage = 36849 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8888
		current_write_path = 0
		source storage id = 192.168.56.136
		if_trunk_server = 0
		connection.alloc_count = 256
		connection.current_count = 1
		connection.max_count = 1
		total_upload_count = 0
		success_upload_count = 0
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 0
		success_upload_bytes = 0
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 0
		success_sync_in_bytes = 0
		total_sync_out_bytes = 0
		success_sync_out_bytes = 0
		total_file_open_count = 0
		success_file_open_count = 0
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 0
		success_file_write_count = 0
		last_heart_beat_time = 2017-01-22 06:32:26
		last_source_update = 1970-01-01 08:00:00
		last_sync_update = 1970-01-01 08:00:00
		last_synced_timestamp = 1970-01-01 08:00:00 
	Storage 2:
		id = 192.168.56.136
		ip_addr = 192.168.56.136 (edu-dfs-storage-group1-2)  ACTIVE
		http domain = 
		version = 5.05
		join time = 2017-01-22 06:00:33
		up time = 2017-01-22 06:15:17
		total storage = 38345 MB
		free storage = 36848 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8888
		current_write_path = 0
		source storage id = 
		if_trunk_server = 0
		connection.alloc_count = 256
		connection.current_count = 1
		connection.max_count = 1
		total_upload_count = 0
		success_upload_count = 0
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 0
		success_upload_bytes = 0
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 0
		success_sync_in_bytes = 0
		total_sync_out_bytes = 0
		success_sync_out_bytes = 0
		total_file_open_count = 0
		success_file_open_count = 0
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 0
		success_file_write_count = 0
		last_heart_beat_time = 2017-01-22 06:32:18
		last_source_update = 1970-01-01 08:00:00
		last_sync_update = 1970-01-01 08:00:00
		last_synced_timestamp = 1970-01-01 08:00:00 

Group 2:
group name = group2
disk total space = 38345 MB
disk free space = 36849 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

	Storage 1:
		id = 192.168.56.137
		ip_addr = 192.168.56.137 (edu-dfs-storage-group2-1)  ACTIVE
		http domain = 
		version = 5.05
		join time = 2017-01-22 06:07:15
		up time = 2017-01-22 06:07:15
		total storage = 38345 MB
		free storage = 36849 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8888
		current_write_path = 0
		source storage id = 192.168.56.138
		if_trunk_server = 0
		connection.alloc_count = 256
		connection.current_count = 1
		connection.max_count = 1
		total_upload_count = 0
		success_upload_count = 0
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 0
		success_upload_bytes = 0
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 0
		success_sync_in_bytes = 0
		total_sync_out_bytes = 0
		success_sync_out_bytes = 0
		total_file_open_count = 0
		success_file_open_count = 0
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 0
		success_file_write_count = 0
		last_heart_beat_time = 2017-01-22 06:32:20
		last_source_update = 1970-01-01 08:00:00
		last_sync_update = 1970-01-01 08:00:00
		last_synced_timestamp = 1970-01-01 08:00:00 
	Storage 2:
		id = 192.168.56.138
		ip_addr = 192.168.56.138 (edu-dfs-storage-group2-2)  ACTIVE
		http domain = 
		version = 5.05
		join time = 2017-01-22 06:06:09
		up time = 2017-01-22 06:06:09
		total storage = 38345 MB
		free storage = 36849 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8888
		current_write_path = 0
		source storage id = 
		if_trunk_server = 0
		connection.alloc_count = 256
		connection.current_count = 1
		connection.max_count = 1
		total_upload_count = 0
		success_upload_count = 0
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 0
		success_upload_bytes = 0
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 0
		success_sync_in_bytes = 0
		total_sync_out_bytes = 0
		success_sync_out_bytes = 0
		total_file_open_count = 0
		success_file_open_count = 0
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 0
		success_file_write_count = 0
		last_heart_beat_time = 2017-01-22 06:32:11
		last_source_update = 1970-01-01 08:00:00
		last_sync_update = 1970-01-01 08:00:00
		last_synced_timestamp = 1970-01-01 08:00:00

10.上传文件测试：

在tracker服务器上测试（任意一台）
cd  /etc/fdfs
[[email protected] fdfs]# pwd
/etc/fdfs
[[email protected] fdfs]# ls
client.conf  client.conf.sample  storage.conf.sample  tracker.conf  tracker.conf.sample
[[email protected] fdfs]# cp client.conf.sample  client.conf
[[email protected] fdfs]#vi  client.conf
base_path=/fastdfs/tracker
tracker_server=192.168.56.131:22122
tracker_server=192.168.56.132:22122


[[email protected] fdfs]# /usr/bin/fdfs_upload_file   /etc/fdfs/client.conf  /usr/local/src/fastdfs-5.05.tar.gz 
group2/M00/00/00/wKg4iViD5pGASNhIAAUggSnIHZU.tar.gz
[[email protected] fdfs]# /usr/bin/fdfs_upload_file   /etc/fdfs/client.conf  /usr/local/src/libfastcommon-master.zip 
group1/M00/00/00/wKg4h1iD5rGATHRxAAKe64wyZIg888.zip

storage服务器检测一下：
[[email protected] logs]# cd  /fastdfs/storage/data/00/00
[[email protected] 00]# ls
wKg4h1iD5rGATHRxAAKe64wyZIg888.zip
[[email protected] 00]#

11.在每个storage服务器安装nginx

fastdfs-nginx-module 安装

cd  /usr/local/src/
tar xf fastdfs-nginx-module_v1.16.tar.gz
cd fastdfs-nginx-module/src/
vi config 
CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/"  #改成如下

12.nginx的安装，每个storage服务器都得安装

yum -y  install openssl openssl-devel  pcre* zlib libtool autoconf  automke
 cd  /usr/local/src/
 tar xf  nginx-1.6.2.tar.gz 
 cd  nginx-1.6.2
 ./configure  --add-module=/usr/local/src/fastdfs-nginx-module/src/
 make &&  make  install

13.拷贝nginx-module配置文件 storage服务器

 cp /usr/local/src/fastdfs-nginx-module/src/mod_fastdfs.conf   /etc/fdfs/
vi /etc/fdfs/mod_fastdfs.conf 

group1的配置文件
[[email protected] fdfs]# grep  -r  "^[a-Z]" mod_fastdfs.conf 
connect_timeout=2
network_timeout=30
base_path=/tmp
load_fdfs_parameters_from_tracker=true
storage_sync_file_max_delay = 86400
use_storage_id = false
storage_ids_filename = storage_ids.conf
tracker_server=192.168.56.131:22122
tracker_server=192.168.56.132:22122
storage_server_port=23000
group_name=group1    #group1 的配置文件为group1  group2的配置文件为group2  其余不变
url_have_group_name = true
store_path_count=1
store_path0=/fastdfs/storage
log_level=info
log_filename=
response_mode=proxy
if_alias_prefix=
flv_support = true
flv_extension = flv
group_count = 2
[group1]
group_name=group1
storage_server_port=23000
store_path_count=1
store_path0=/fastdfs/storage
[group2]
group_name=group2
storage_server_port=23000
store_path_count=1
store_path0=/fastdfs/storage
[[email protected] fdfs]# 


group2的配置文件
[[email protected] fdfs]# grep  -r "^[a-Z]" mod_fastdfs.conf 
connect_timeout=2
network_timeout=30
base_path=/tmp
load_fdfs_parameters_from_tracker=true
storage_sync_file_max_delay = 86400
use_storage_id = false
storage_ids_filename = storage_ids.conf
tracker_server=192.168.56.131:22122
tracker_server=192.168.56.132:22122
storage_server_port=23000
group_name=group2
url_have_group_name = true
store_path_count=1
store_path0=/fastdfs/storage
log_level=info
log_filename=
response_mode=proxy
if_alias_prefix=
flv_support = true
flv_extension = flv
group_count = 2
[group1]
group_name=group1
storage_server_port=23
000
store_path_count=1
store_path0=/fastdfs/storage
[group2]
group_name=group2
storage_server_port=23000
store_path_count=1
store_path0=/fastdfs/storage
[[email protected] fdfs]# 


所有的storage服务器的nginx加入开机自启
 echo "/usr/local/nginx/sbin/nginx" >>/etc/rc.local

复制fastdfs的配置文件到/etc/fdfs下

[[email protected] conf]# pwd
/usr/local/src/fastdfs-5.05/conf
[[email protected] conf]# 
[[email protected] conf]# \cp http.conf  /etc/fdfs/
[[email protected] conf]# \cp  mime.types   /etc/fdfs/

做软连接：

ln -s  /fastdfs/storage/data   /fastdfs/storage/data/M00

修改nginx的配置文件

[[email protected] conf]# pwd
/usr/local/nginx/conf
 
vim  nginx.conf  修改如下位置
 
user  root;
 listen       8888;
    location ~/group([0-9])/M00 {
           ngx_fastdfs_module;
        }

启动nginx

[[email protected] conf]# /usr/local/nginx/sbin/nginx   -t 
ngx_http_fastdfs_set pid=19562
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

/usr/local/nginx/sbin/nginx   启动nginx
echo "/usr/local/nginx/sbin/nginx" >>/etc/rc.local

测试：刚才上传文件的路径看上面

http://192.168.56.138:8888/group2/M00/00/00/wKg4iViD5pGASNhIAAUggSnIHZU.tar.gz

http://192.168.56.135:8888/group1/M00/00/00/wKg4h1iD5rGATHRxAAKe64wyZIg888.zip

可以看出两组文件都可以下载