fastdfs能存储啥东西
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了fastdfs能存储啥东西相关的知识，希望对你有一定的参考价值。
参考技术A FastDFS是一个开源的轻量级分布式文件系统，它对文件进行管理，功能包括：文件存储、文件同步、文件访问（文件上传、文件下载）等，解决了大容量存储和负载均衡的问题。特别适合以文件为载体的在线服务，如相册网站、视频网站等等。FastdFS实现分布式存储
FastDFS是一个轻量级的开源分布式文件系统，主要解决了大容量的文件存储和高并发访问问题，文件存取时实现了负载均衡。
FastDFS的特性
分组存储，灵活简洁，对等结构，不存在单点。
文件ID由FastDFS生成，作为文件访问凭证。FastDFS不需要传统的name server
和流行的web server无缝衔接，FastDFS已提供apache和nginx扩展模块。
大，中，小文件均可以很好支持，支持海量小文件存储。
支持多块磁盘，支持单盘数据恢复。
支持相同文件内容只保存一份，节省存储空间。
存储服务器上可以保存文件附加属性
下载文件支持多线程方式，支持断点续传。
只有两个角色，tracker server和storage server,不需要存储文件的索引信息。
所有服务器都是对等的，不存在master-slave关系
存储服务器采用分组方式，同组内存储服务器上的文件完全相同。
不同组的storage server之间不会相互通信。
由storage server主动向tracker server报告状态信息，tracker server之间通常不会相互通信。
FastDFS上传文件的流程
client询问tracker上传的storage
tracker返回一台可用的storage
client直接和storage通信完成文件上传，storage返回文件的ID。
FastDFS下载文件流程
client询问tracker下载文件的storage,参数为文件ID（组名和文件名）
tracker返回一台可用的storage
client直接和storage通信完成文件下载。
FastDFS同步机制
采用binlog文件记录更新操作，根据binlog进行文件同步
同一组内的storage server之间是对等的，文件上传，删除等操作可以在任意一台storage server上进行。
文件同步只在同组内的storage server之间进行，采用push方式，即源服务器同步给目标服务器。
当新增一台storage server时，由已有的一台storage server将已有的所有数据（包括源头数据和备份数据）同步给新增服务器。
FastDFS的核心组件介绍
tracker:调度器，负责维持集群的消息。
storage server:以group为单位进行组织，任何一个storage server都应该属于某个group，一个group应该包含多个storage server；在同一个group内部，各storage server的数据互相冗余.
FastDFS实现分布式存储
分别在tracker server 和storage server的节点上安装如下软件包。
fastdfs-5.0.11-1.el7.centos.x86_64.rpm
fastdfs-debuginfo-5.0.11-1.el7.centos.x86_64.rpm
fastdfs-server-5.0.11-1.el7.centos.x86_64.rpm
fastdfs-tool-5.0.11-1.el7.centos.x86_64.rpm
libfastcommon-1.0.36-1.el7.centos.x86_64.rpm
libfastcommon-devel-1.0.36-1.el7.centos.x86_64.rpm
libfdfsclient-5.0.11-1.el7.centos.x86_64.rpm
libfdfsclient-devel-5.0.11-1.el7.centos.x86_64.rpm
配置tracker节点的文件：主要配置内容如下
[[email protected] ~]#mkdir -p /data/fastdfs/tracker
[[email protected] ~]# cd /etc/fdfs
disabled=false（默认为false，表示是否无效） port=22122（默认为22122） base_path=/data/fastdfs/tracker [[email protected] fdfs]# /etc/init.d/fdfs_trackerd start Starting fdfs_trackerd (via systemctl): [ OK ] [[email protected] fastdfs]# ps -ef |grep fdfs root 5573 1 0 17:42 ? 00:00:00 /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
配置storage节点的文件：配置内容大致如下
[[email protected] ~]#mkdir -p /data/fastdfs/storage
disabled=false（默认为false，表示是否无效） group_name=group1(定义组） port=23000（默认为23000） base_path=/data/fastdfs/storage tracker_server=172.16.250.97:22122 store_path0=/data/fastdfs/storage（真正存储数据的路径） http.server_port=8888（默认为8888，nginx中配置的监听端口那一致） [[email protected] fdfs]# /etc/init.d/fdfs_storaged start Starting fdfs_storaged (via systemctl): [ OK ] [[email protected] fdfs]# ps -ef |grep fdfs root 3431 1 0 18:01 ? 00:00:00 /usr/bin/fdfs_storaged /etc/fdfs/storage.conf
配置tracker节点的client.conf文件
base_path=/data/fastdfs/tracker tracker_server=172.16.250.97:22122 查看存储节点状态 [[email protected] fdfs]# fdfs_monitor /etc/fdfs/client.conf [2017-12-16 20:08:46] DEBUG - base_path=/data/fastdfs/tracker, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0 server_count=1, server_index=0 tracker server is 172.16.250.97:22122 group count: 2 Group 1: group name = group1 disk total space = 99951 MB disk free space = 93419 MB trunk free space = 0 MB storage server count = 1 active server count = 1 storage server port = 23000 storage HTTP port = 8888 store path count = 1 subdir count per path = 256 current write server index = 0 current trunk file id = 0 Storage 1: id = 172.16.252.16 ip_addr = 172.16.252.16 ACTIVE http domain = version = 5.11 join time = 2017-12-16 19:25:52 up time = 2017-12-16 18:01:23 total storage = 99951 MB free storage = 93419 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 0 connection.max_count = 0 total_upload_count = 0 success_upload_count = 0 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 0 success_upload_bytes = 0 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 0 success_sync_in_bytes = 0 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 0 success_file_open_count = 0 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 0 success_file_write_count = 0 last_heart_beat_time = 2017-12-16 20:08:37 last_source_update = 1970-01-01 08:00:00 last_sync_update = 1970-01-01 08:00:00 last_synced_timestamp = 1970-01-01 08:00:00 Group 2: group name = group2 disk total space = 99951 MB disk free space = 93027 MB trunk free space = 0 MB storage server count = 2 active server count = 2 storage server port = 2399 storage HTTP port = 8888 store path count = 1 subdir count per path = 256 current write server index = 0 current trunk file id = 0 Storage 1: id = 172.16.252.15 ip_addr = 172.16.252.15 ACTIVE http domain = version = 5.11 join time = 2017-12-16 19:31:27 up time = 2017-12-16 19:31:27 total storage = 99951 MB free storage = 93027 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 2399 storage_http_port = 8888 current_write_path = 0 source storage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 0 success_upload_count = 0 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 0 success_upload_bytes = 0 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 0 success_sync_in_bytes = 0 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 0 success_file_open_count = 0 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 0 success_file_write_count = 0 last_heart_beat_time = 2017-12-16 20:08:32 last_source_update = 1970-01-01 08:00:00 last_sync_update = 1970-01-01 08:00:00 last_synced_timestamp = 1970-01-01 08:00:00 Storage 2: id = 172.16.252.16 ip_addr = 172.16.252.16 ACTIVE http domain = version = 5.11 join time = 2017-12-16 19:25:52 up time = 2017-12-16 19:35:46 total storage = 99951 MB free storage = 93419 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 2399 storage_http_port = 8888 current_write_path = 0 source storage id = 172.16.252.15 if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 0 success_upload_count = 0 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 0 success_upload_bytes = 0 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 0 success_sync_in_bytes = 0 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 0 success_file_open_count = 0 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 0 success_file_write_count = 0 last_heart_beat_time = 2017-12-16 20:08:37 last_source_update = 1970-01-01 08:00:00 last_sync_update = 1970-01-01 08:00:00 last_synced_timestamp = 1970-01-01 08:00:00
上传文件
[[email protected] ~]# fdfs_upload_file /etc/fdfs/client.conf all.sh group1/M00/00/00/rBD8EFo1EKGAXX8nAAAANoKhiHA5839.sh
查看文件
[[email protected] fdfs]# fdfs_file_info /etc/fdfs/client.conf group1/M00/00/00/rBD8EFo1EKGAXX8nAAAANoKhiHA5839.sh source storage id: 0 source ip address: 172.16.252.16 file create timestamp: 2017-12-16 20:25:05 file size: 54 file crc32: 2191624304 (0x82A18870)
文件下载
[[email protected] fdfs]# fdfs_download_file /etc/fdfs/client.conf group1/M00/00/00/rBD8EFo1EKGAXX8nAAAANoKhiHA5839.sh
以上是关于fastdfs能存储啥东西的主要内容，如果未能解决你的问题，请参考以下文章