实战学习开源项目学习之FastDFS部署

Posted 黑黑白白君

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了实战学习开源项目学习之FastDFS部署相关的知识,希望对你有一定的参考价值。



涉及开源项目(dailyfresh-B2C)的部署架构如下:
在这里插入图片描述


1)FastDFS是什么?

fastDFS 是以C语言开发的一项开源轻量级分布式文件系统,对文件进行管理。主要功能有:文件存储、文件同步、文件访问(文件上传/下载),特别适合以文件为载体的在线服务,如图片网站、视频网站等。

FastDFS由跟踪服务器(Tracker Server)、存储服务器(Storage Server)和客户端(Client)构成。

fastDFS三方交互

在这里插入图片描述

在这里插入图片描述

1.1 Tracker server 追踪服务器

跟踪服务器主要做调度工作,起负载均衡的作用。不论是上传还是下载都是通过tracker来分配资源,是客户端和数据服务器交互的枢纽

Tracker是FastDFS的协调者,负责管理所有的storage server和group,每个storage在启动后会连接Tracker,告知自己所属的group等信息,并保持周期性的心跳,tracker根据storage的心跳信息,建立group==[storage server list]的映射表。

  • Tracker需要管理的元信息很少,会全部存储在内存中。
  • tracker上的元信息都是由storage汇报的信息生成的,本身不需要持久化任何数据,这样使得tracker非常容易扩展,直接增加tracker机器即可扩展为tracker cluster来服务
  • cluster里每个tracker之间是完全对等的,所有的tracker都接受stroage的心跳信息,生成元数据信息来提供读写服务

客户端访问集群的时候会随机分配一个Tracker来和客户端交互。

1.2 Storage server 储存服务器

实际存储数据,分成若干个组(group),实际traker就是管理的storage中的组,而组内机器中则存储数据,group可以隔离不同应用的数据,不同的应用的数据放在不同group里面。

  • group:

    组, 也可称为卷。 同组内服务器上的文件是完全相同的 ,同一组内的storage server之间是对等的, 文件上传、 删除等操作可以在任意一台storage server上进行 。卷内服务器文件相互同步备份,以达到容灾的目的。

  • 优点:

    • 海量的存储:主从型分布式存储,存储空间方便拓展
    • fastDFS对文件内容做hash处理,避免出现重复文件
    • fastDFS结合nginx集成, 提供网站效率

1.3 客户端Client

主要是上传下载数据的服务器,也就是自己的项目所部署在的服务器。客户端一般可以使用ngnix等静态服务器来调用或者做一部分的缓存。

作为业务请求的发起方,通过专有接口,使用TCP/IP协议与跟踪器服务器或存储节点进行数据交互。FastDFS向使用者提供基本文件访问接口,比如upload、download、append、delete等,以客户端库的方式提供给用户使用。

2)上传机制

在这里插入图片描述
1、客户端请求Tracker服务,获取到存储服务器的ip地址和端口
2、客户端根据返回的IP地址和端口号请求上传文件
3、存储服务器接收到请求后生产文件,并且将文件内容写入磁盘并返回给客户端file_id、路径信息、文件名等信息
4、客户端保存相关信息,上传完毕

2.1 内部机制:

2.1.1 选择tracker server

当集群中不止一个tracker server时,由于tracker之间是完全对等的关系,客户端在upload文件时可以任意选择一个trakcer。

2.1.2 选择存储的group

当tracker接收到upload file的请求时,会为该文件分配一个可以存储该文件的group,支持如下选择group的规则:

  • Round robin,所有的group间轮询
  • Specified group,指定某一个确定的group
  • Load balance,剩余存储空间多多group优先

2.1.3 选择storage server

当选定group后,tracker会在group内选择一个storage server给客户端,支持如下选择storage的规则:

  • Round robin,在group内的所有storage间轮询
  • First server ordered by ip,按ip排序
  • First server ordered by priority,按优先级排序(优先级在storage上配置)

2.1.4 选择storage path

当分配好storage server后,客户端将向storage发送写文件请求,storage将会为文件分配一个数据存储目录,支持如下规则:

  • Round robin,多个存储目录间轮询
  • 剩余存储空间最多的优先

2.1.5 生成Fileid

选定存储目录之后,storage会为文件生一个Fileid,由storage server ip、文件创建时间、文件大小、文件crc32和一个随机数拼接而成,然后将这个二进制串进行base64编码,转换为可打印的字符串。 选择两级目录 当选定存储目录之后,storage会为文件分配一个fileid,每个存储目录下有两级256*256的子目录,storage会按文件fileid进行两次hash(猜测),路由到其中一个子目录,然后将文件以fileid为文件名存储到该子目录下。

2.1.6 生成文件名

当文件存储到某个子目录后,即认为该文件存储成功,接下来会为该文件生成一个文件名,文件名由group、存储目录、两级子目录、fileid、文件后缀名(由客户端指定,主要用于区分文件类型)拼接而成。

3)下载机制

在这里插入图片描述

1、客户端带上文件名信息请求Tracker服务,获取到存储服务器的ip地址和端口
2、客户端根据返回的IP地址和端口号请求下载文件
3、存储服务器接收到请求后返回文件给客户端

  • 跟upload file一样,在download file时客户端可以选择任意tracker server。
  • client发送download请求给某个tracker,必须带上文件名信息,tracke从文件名中解析出文件的group、大小、创建时间等信息,然后为该请求选择一个storage用来服务读请求。

由于group内的文件同步是在后台异步进行的,所以有可能出现在读到时候,文件还没有同步到某些storage server上,为了尽量避免访问到这样的storage,tracker按照特定的规则选择group内可读的storage,具体在这边不作详述。

4)FastDFS搭建

可以使用一台PC,只有一个Tracker、一个Storage服务。配置nginx访问图片。

在这里插入图片描述

安装FastDFS之前要先安装它的依赖库libfastcommon。

4.1 安装fastDFS依赖包libfastcommon

  • https://github.com/happyfish100/libfastcommon/releases
he@he-ThinkPad-X200:~$ wget https://github.com/happyfish100/libfastcommon/archive/master.zip  # 下载安装包
--2021-05-01 18:40:50--  https://github.com/happyfish100/libfastcommon/archive/master.zip
Resolving github.com (github.com)... 13.250.177.223
Connecting to github.com (github.com)|13.250.177.223|:443... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2021-05-01 18:42:51--  (try: 2)  https://github.com/happyfish100/libfastcommon/archive/master.zip
Connecting to github.com (github.com)|13.250.177.223|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/happyfish100/libfastcommon/zip/master [following]
--2021-05-01 18:42:51--  https://codeload.github.com/happyfish100/libfastcommon/zip/master
Resolving codeload.github.com (codeload.github.com)... 54.251.140.56
Connecting to codeload.github.com (codeload.github.com)|54.251.140.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘master.zip’

master.zip                           [    <=>                                                    ] 303.49K   476KB/s    in 0.6s    

2021-05-01 18:42:58 (476 KB/s) - ‘master.zip’ saved [310774]
he@he-ThinkPad-X200:~$ ls
Desktop    Downloads  github-learning  Music     Public          Python-3.7.9      redis-6.2.2         snap       test-uwsgi.py
Documents  dump.rdb   master.zip       Pictures  pycharm-2021.1  Python-3.7.9.tgz  redis-6.2.2.tar.gz  Templates  Videos
he@he-ThinkPad-X200:~$ unzip master.zip 
Archive:  master.zip
5650e87665df067061d11ab7553901edc813fedc
   creating: libfastcommon-master/
  inflating: libfastcommon-master/.gitignore  
  inflating: libfastcommon-master/HISTORY  
...

he@he-ThinkPad-X200:~$ ls
Desktop    dump.rdb              master.zip  Public          Python-3.7.9.tgz    snap           Videos
Documents  github-learning       Music       pycharm-2021.1  redis-6.2.2         Templates
Downloads  libfastcommon-master  Pictures    Python-3.7.9    redis-6.2.2.tar.gz  test-uwsgi.py
he@he-ThinkPad-X200:~$ cd libfastcommon-master/
he@he-ThinkPad-X200:~/libfastcommon-master$ ls
doc  HISTORY  INSTALL  libfastcommon.spec  LICENSE  make.sh  php-fastcommon  README  src
he@he-ThinkPad-X200:~/libfastcommon-master$ ./make.sh  # 执行编译
gcc -Wall -Wformat-truncation=0 -Wformat-overflow=0 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -g -O3 -c -o hash.o hash.c  
gcc -Wall -Wformat-truncation=0 -Wformat-overflow=0 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -g -O3 -c -o chain.o chain.c  
...

he@he-ThinkPad-X200:~/libfastcommon-master$ sudo ./make.sh install  # 安装
[sudo] password for he: 
mkdir -p /usr/lib64
mkdir -p /usr/lib
mkdir -p /usr/include/fastcommon
install -m 755 libfastcommon.so /usr/lib64
install -m 644 common_define.h hash.h chain.h logger.h base64.h shared_func.h pthread_func.h ini_file_reader.h _os_define.h sockopt.h sched_thread.h http_func.h md5.h local_ip_func.h avl_tree.h ioevent.h ioevent_loop.h fast_task_queue.h fast_timer.h locked_timer.h process_ctrl.h fast_mblock.h connection_pool.h fast_mpool.h fast_allocator.h fast_buffer.h skiplist.h multi_skiplist.h flat_skiplist.h skiplist_common.h system_info.h fast_blocked_queue.h php7_ext_wrapper.h id_generator.h char_converter.h char_convert_loader.h common_blocked_queue.h multi_socket_client.h skiplist_set.h uniq_skiplist.h fc_list.h locked_list.h json_parser.h buffered_file_writer.h server_id_func.h fc_queue.h fc_memory.h shared_buffer.h thread_pool.h fc_atomic.h /usr/include/fastcommon

4.2 安装fastDFS

  • https://github.com/happyfish100/FastDFS
he@he-ThinkPad-X200:~$ wget https://github.com/happyfish100/fastdfs/archive/V5.12.zip
--2021-05-01 19:18:07--  https://github.com/happyfish100/fastdfs/archive/V5.12.zip
Resolving github.com (github.com)... 13.250.177.223
Connecting to github.com (github.com)|13.250.177.223|:443... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2021-05-01 19:18:23--  (try: 2)  https://github.com/happyfish100/fastdfs/archive/V5.12.zip
Connecting to github.com (github.com)|13.250.177.223|:443... ^C
he@he-ThinkPad-X200:~$ 
he@he-ThinkPad-X200:~$ 
he@he-ThinkPad-X200:~$ wget https://github.com/happyfish100/fastdfs/archive/V5.12.tar.gz
--2021-05-01 19:18:48--  https://github.com/happyfish100/fastdfs/archive/V5.12.tar.gz
Resolving github.com (github.com)... 13.250.177.223
Connecting to github.com (github.com)|13.250.177.223|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/happyfish100/fastdfs/tar.gz/V5.12 [following]
--2021-05-01 19:18:49--  https://codeload.github.com/happyfish100/fastdfs/tar.gz/V5.12
Resolving codeload.github.com (codeload.github.com)... 13.229.189.0
Connecting to codeload.github.com (codeload.github.com)|13.229.189.0|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘V5.12.tar.gz’

V5.12.tar.gz                         [    <=>                                                    ] 369.74K   584KB/s    in 0.6s    

2021-05-01 19:18:50 (584 KB/s) - ‘V5.12.tar.gz’ saved [378615]

he@he-ThinkPad-X200:~$ ls
Desktop    dump.rdb              master.zip  Public          Python-3.7.9.tgz    snap           V5.12.tar.gz
Documents  github-learning       Music       pycharm-2021.1  redis-6.2.2         Templates      Videos
Downloads  libfastcommon-master  Pictures    Python-3.7.9    redis-6.2.2.tar.gz  test-uwsgi.py
he@he-ThinkPad-X200:~$ tar -zxvf V5.12.tar.gz 
fastdfs-5.12/
fastdfs-5.12/COPYING-3_0.txt
fastdfs-5.12/HISTORY
...

he@he-ThinkPad-X200:~/fastdfs-5.12$ ./make.sh
cc -Wall -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -g -O1 -DDEBUG_FLAG -c -o ../common/fdfs_global.o ../common/fdfs_global.c  -I../common -I/usr/local/include
cc -Wall -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -g -O1 -DDEBUG_FLAG -c -o tracker_proto.o tracker_proto.c  -I../common -I/usr/local/include
cc -Wall -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -g -O1 -DDEBUG_FLAG -c -o tracker_mem.o tracker_mem.c  -I../common -I/usr/local/include
...

he@he-ThinkPad-X200:~/fastdfs-5.12$ sudo ./make.sh install
[sudo] password for he: 
mkdir -p /usr/bin
mkdir -p /etc/fdfs
cp -f fdfs_trackerd /usr/bin
...

4.3 部署Tracker服务

4.3.1 配置

he@he-ThinkPad-X200:~/fastdfs-5.12$ cd /etc/fdfs  # 进入/etc/fdfs目录
he@he-ThinkPad-X200:/etc/fdfs$ ls  # 有四个.sample后缀的文件(自动生成的fdfs模板配置文件)
client.conf.sample  storage.conf.sample  storage_ids.conf.sample  tracker.conf.sample
he@he-ThinkPad-X200:/etc/fdfs$ sudo cp tracker.conf.sample tracker.conf  # 通过cp命令拷贝tracker.conf.sample
he@he-ThinkPad-X200:/etc/fdfs$ ls
client.conf.sample  storage.conf.sample  storage_ids.conf.sample  tracker.conf  tracker.conf.sample
he@he-ThinkPad-X200:/etc/fdfs$ sudo vim tracker.conf  # 编辑tracker.conf,修改相关参数

在这里插入图片描述

  • base_path:tracker存储data和log的跟路径,必须提前创建好
  • port:tracker默认23000
  • http.server_port=80 #http端口,需要和nginx相同

4.3.2 启动和关闭

he@he-ThinkPad-X200:~$ /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start  # 启动tracker(支持start|stop|restart)
he@he-ThinkPad-X200:~$ netstat -apn | grep fdfs  # 查看端口情况
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:22122           0.0.0.0:*               LISTEN      221572/fdfs_tracker 
he@he-ThinkPad-X200:~$ ls ./fastdfs/tracker/
data  logs
he@he-ThinkPad-X200:~$ vi ./fastdfs/tracker/logs/  # 查看tracker.log文件
he@he-ThinkPad-X200:~$ /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf stop  # 停止运行
waiting for pid [221572] exit ...
pid [221572] exit.
he@he-ThinkPad-X200:~$ netstat -apn | grep fdfs
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
he@he-ThinkPad-X200:~$ 

4.4 部署Storage服务

4.4.1 配置

1、进入/etc/fdfs目录,有cp命令拷贝storage.conf.sample
2、编辑storage.conf,修改相关参数:
在这里插入图片描述

  • base_path:storage存储data和log的跟路径,必须提前创建好
  • port=23000 #storge默认23000,同一个组的storage端口号必须一致
  • group_name=group1 #默认组名,根据实际情况修改

在这里插入图片描述

  • store_path_count=1 #存储路径个数,需要和store_path个数匹配
  • store_path0:如果为空,则使用base_path
  • tracker_server:配置该storage监听的tracker的ip和port

4.4.2 启动和关闭

he@he-ThinkPad-X200:~$ /usr/bin/fdfs_storaged /etc/fdfs/storage.conf start
he@he-ThinkPad-X200:~$ netstat -apn|grep fdfs
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
he@he-ThinkPad-X200:~$ /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start
he@he-ThinkPad-X200:~$ netstat -apn | grep fdfs
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:22122           0.0.0.0:*               LISTEN      221674/fdfs_tracker 
he@he-ThinkPad-X200:~$ /usr/bin/fdfs_storaged /etc/fdfs/storage.conf start  # 启动storage(支持start|stop|restart)
process /usr/bin/fdfs_storaged already running, pid: 221668
he@he-ThinkPad-X200:~$ netstat -apn | grep fdfs  # 查看端口情况
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:22122           0.0.0.0:*               LISTEN      221674/fdfs_tracker 
tcp        0      0 0.0.0.0:23000           0.0.0.0:*               LISTEN      221668/fdfs_storage 
tcp        0      0 192.168.0.100:49076     192.168.0.100:22122     ESTABLISHED 221668/fdfs_storage 
tcp        0      0 192.168.0.100:22122     192.168.0.100:49076     ESTABLISHED 221674/fdfs_tracker 
he@he-ThinkPad-X200:~$ vi ./fastdfs/storage/logs/

4.5 测试Tracker和Storage是否部署成功

1、进入/etc/fdfs目录,有cp命令拷贝client.conf.sample
2、修改client.conf相关配置:

he@he-ThinkPad-X200:~$ sudo cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf
he@he-ThinkPad-X200:~$ sudo vim /etc/fdfs/client.conf  # 编辑/etc/fdfs/client.conf配置文件

在这里插入图片描述

  • base_path=tracker服务器文件路径
  • tracker_server=tracker服务器IP地址和端口号
  • http.tracker_server_port=80 # tracker服务器的http端口号,必须和tracker的设置对应起来

3、新建一个test.txt,内容为abc,并上传文件:

he@he-ThinkPad-X200:~$ cd Downloads/
he@he-ThinkPad-X200:~/Downloads$ vim test.txt  # 创建一个test.txt文件
he@he-ThinkPad-X200:~/Downloads$ /usr/bin/fdfs_upload_file  /etc/fdfs/client.conf /home/he/Downloads/test.txt  # 上传
group1/M00/00/00/wKgAZGCNRu6AYR6pAAAABEeIgU4549.txt
# 组名:group1
# 磁盘:M00
# 目录:00/00
# 文件名称:wKgAZGCNRu6AYR6pAAAABEeIgU4549.txt

he@he-ThinkPad-X200:~/Downloads$ cd /home/he/fastdfs/storage/data/00/00  # 查看结果,进入storage的data目录
he@he-ThinkPad-X200:~/fastdfs/storage/data/00/00$ ls
wKgAZGCNRu6AYR6pAAAABEeIgU4549.txt
he@he-ThinkPad-X200:~/fastdfs/storage/data/00/00$ 

  • 上传命令:/usr/bin/fdfs_upload_file <config_file> <local_filename>

4.6 安装Nginx和fastdfs-nginx-module模块

4.6.1 安装Nginx

关于Nginx安装可见本栏的《【实战学习(四)】开源项目学习之Nginx介绍及安装部署》。(https://blog.csdn.net/m0_37621024/article/details/116074201)

4.6.2 安装fastdfs-nginx-module模块

he@he-ThinkPad-X200:~$ wget https://github.com/happyfish100/fastdfs-nginx-module/archive/V1.20.tar.gz  # 下载fastdfs-nginx-module安装包
--2021-05-01 20:44:22--  https://github.com/happyfish100/fastdfs-nginx-module/archive/V1.20.tar.gz
Resolving github.com (github.com)... 13.229.188.59
Connecting to github.com (github.com)|13.229.188.59|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/happyfish100/fastdfs-nginx-module/tar.gz/V1.20 [following]
--2021-05-01 20:44:22--  https://codeload.github.com/happyfish100/fastdfs-nginx-module/tar.gz/V1.20
Resolving codeload.github.com (codeload.github.com)... 13.250.162.133
Connecting to codeload.github.com (codeload.github.com)|13.250.162.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘V1.20.tar.gz’

V1.20.tar.gz                         [ <=>                                                       ]  19.36K  --.-KB/s    in 0.05s   

2021-05-01 20:44:23 (377 KB/s) - ‘V1.20.tar.gz’ saved [19825]

he@he-ThinkPad-X200:~$ ls
Desktop    dump.rdb      github-learning       Music     pycharm-2021.1    redis-6.2.2         Templates      V5.12.tar.gz
Documents  fastdfs       libfastcommon-master  Pictures  Python-3.7.9      redis-6.2.2.tar.gz  test-uwsgi.py  Videos
Downloads  fastdfs-5.12  master.zip            Public    Python-3.7.9.tgz  snap                V1.20.tar.gz
he@he-ThinkPad-X200:~$ tar -zxvf V1.20.tar.gz
fastdfs-nginx-module-1.20/
fastdfs-nginx-module-1.20/HISTORY
fastdfs-nginx-module-1.20/INSTALL
fastdfs-nginx-module-1.20/src/
fastdfs-nginx-module-1.20/src/common.c
fastdfs-nginx-module-1.20/src/common.h
fastdfs-nginx-module-1.20/src/config
fastdfs-nginx-module-1.20/src/mod_fastdfs.conf
fastdfs-nginx-module-1.20/src/ngx_http_fastdfs_module.c
he@he-ThinkPad-X200:~$ 

如何在已经安装过nginx后再次添加新的模块:

he@he-ThinkPad-X200:~$ /usr/sbin/nginx -V  # 查看nginx版本极其编译参数(即nginx已经安装了哪些模块)
nginx version: nginx/1.18.0 (Ubuntu)
built with OpenSSL 1.1.1f  31 Mar 2020
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-5J5hor/nginx-1.18.0=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module
he@he-ThinkPad-X200:~$ whereis nginx
nginx: /usr/sbin/nginx /usr/lib/nginx /etc/nginx /usr/share/nginx /usr/share/man/man8/nginx.8.gz

he@he-ThinkPad-X200:~$ wget http://nginx.org/download/nginx-1.18.0.tar.gz  # 找到安装nginx的源码根目录(即安装包存放目录),如果没有的话下载新的源码并解压。千万注意要和原来的是一个版本!!!否则后面替换的时候会有问题!
--2021-05-01 21:43:45--  http://nginx.org/download/nginx-1.18.0.tar.gz
Resolving nginx.org (nginx.org)... 3.125.197.172, 52.58.199.22, 2a05:d014:edb:5704::6, ...
Connecting to nginx.org (nginx.org)|3.125.197.172|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1039530 (1015K) [application/octet-stream]
Saving to: ‘nginx-1.18.0.tar.gz’

nginx-1.18.0.tar.gz              100%[==========================================================>]   1015K   831KB/s    in 1.2s    

2021-05-01 21:43:46 (831 KB/s) - ‘nginx-1.18.0.tar.gz’ saved [1039530/1039530]


he@he-ThinkPad-X200:~$ tar -zxvf nginx-1.18.0.tar.gz 

he@he-ThinkPad-X200:~/nginx-1.18.0$ ls
auto  CHANGES  CHANGES.ru  conf  configure  contrib  html  LICENSE  man  README  src

he@he-ThinkPad-X200:~/nginx-1.18.0$ ./configure --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-5J5hor/nginx-1.18.0=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module --add-module=/home/he/fastdfs-nginx-module-1.20/src/
# 可以看到重新添加一个--add-module,指定需要添加的模块的源码目录路径,然后执行make,特别注意,这里不需要执行make!否则就覆盖安装了!
...

he@he-ThinkPad-X200:~/nginx-1.18.0$ make
...
make[1]: Leaving directory '/home/he/nginx-1.18.0'

he@he-ThinkPad-X200:~/nginx-1.18.0$ ls 
auto  CHANGES  CHANGES.ru  conf  configure  contrib  html  LICENSE  Makefile  man  objs  README  src
he@he-ThinkPad-X200:~/nginx-1.18.0$ ls ./objs/
addon              ngx_auto_headers.h                      ngx_http_xslt_filter_module.so  ngx_stream_module_modules.c
autoconf.err       ngx_http_image_filter_module_modules.c  ngx_mail_module_modules.c       ngx_stream_module_modules.o
Makefile           ngx_http_image_filter_module_modules.o  ngx_mail_module_modules.o       ngx_stream_module.so
nginx              ngx_http_image_filter_module.so         ngx_mail_module.so              src
nginx.8            ngx_http_xslt_filter_module_modules.c   ngx_modules.c
ngx_auto_config.h  ngx_http_xslt_filter_module_modules.o   ngx_modules.o
# make(编译)完之后在nginx的objs目录下重新生成一个nginx(nginx的可执行文件),这个就是新版本的程序了。

he@he-ThinkPad-X200:~$ sudo mv /usr/sbin/nginx /usr/sbin/nginx_bak  # 备份旧的nginx程序
he@he-ThinkPad-X200:~$ sudo cp /home/he/nginx-1.18.0/objs/nginx /usr/sbin/nginx  # 把新的nginx程序覆盖原来安装nginx时生成的nginx文件

he@he-ThinkPad-X200:~$ sudo /usr/sbin/nginx -t  # 测试新的nginx程序是否正确
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

he@he-ThinkPad-X200:~$ sudo /usr/sbin/nginx -V  # 查看模块是否已安装
nginx version: nginx/1.18.0
built by gcc 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04) 
built with OpenSSL 1.1.1f  31 Mar 2020
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-5J5hor/nginx-1.18.0=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module --add-module=/home/he/fastdfs-nginx-module-1.20/src/

he@he-ThinkPad-X200:~$ sudo /usr/sbin/nginx -s reload  # 平滑启动服务
nginx: [error] invalid PID number "" in "/run/nginx.pid"
he@he-ThinkPad-X200:~$ whereis nginx
nginx: /usr/sbin/nginx /usr/lib/nginx /etc/nginx /usr/share/nginx /usr/share/man/man8/nginx.8.gz
he@he-ThinkPad-X200:~$ sudo /usr/sbin/nginx -c /etc/nginx/nginx.conf 
he@he-ThinkPad-X200:~$ sudo /usr/sbin/nginx -s reload

# 重启nginx:
he@he-ThinkPad-X200:~$ sudo /usr/sbin/nginx -s quit
he@he-ThinkPad-X200:~$ sudo /usr/sbin/nginx
he@he-ThinkPad-X200:~$ ps -aux | grep nginx
root      240129  0.0  0.0  57836  1516 ?        Ss   21:58   0:00 nginx: master process /usr/sbin/nginx
he        240133  0.0  0.0  17676   664 pts/3    S+   21:58   0:00 grep --color=auto nginx

*报错之./configure: error: the HTTP XSLT module requires the libxml2/libxslt libraries.

解决方法:

sudo apt-get install libxml2 libxml2-dev libxslt-dev
sudo apt-get install libgd2-xpm libgd2-xpm-dev

*报错之./configure: error: the HTTP image filter module requires the GD library.

解决方法:

sudo apt-get install -y libgd-dev

*报错之nginx: [error] invalid PID number “” in “/run/nginx.pid”

解决方法:

需要先执行nginx -c /etc/nginx/nginx.conf

  • nginx.conf文件的路径可以从nginx -t的返回中找到。

然后再nginx -s reload。

4.6.3 配置Nginx和fastdfs-nginx-module模块

1、配置mod-fastdfs.conf,并拷贝到/etc/fdfs文件目录下:

he@he-ThinkPad-X200:~$ cd fastdfs-nginx-module-1.20/src/
he@he-ThinkPad-X200:~/fastdfs-nginx-module-1.20/src$ ls
common.c  common.h  config  mod_fastdfs.conf  ngx_http_fastdfs_module.c
he@he-ThinkPad-X200:~/fastdfs-nginx-module-1.20/src$ sudo cp mod_fastdfs.conf /etc/fdfs/

2、进入/etc/fdfs修改mod-fastdfs.conf:
在这里插入图片描述
在这里插入图片描述

  • base_path
  • tracker_server=tracker的地址
  • url_have_group_name=true #url是否包含group名称
  • storage_server_port=23000 #需要和storage配置的相同
  • store_path_count=1 #存储路径个数,需要和store_path个数匹配
  • store_path0=文件存储的位置

3、配置nginx,在nginx.conf的http部分中添加配置信息:

在这里插入图片描述

he@he-ThinkPad-X200:~$ sudo /usr/sbin/nginx -t  # 查看nginx.conf路径
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
he@he-ThinkPad-X200:~$ cd /etc/nginx/
he@he-ThinkPad-X200:/etc/nginx$ sudo vim nginx.conf 


he@he-ThinkPad-X200:/etc/nginx$ sudo /usr/sbin/nginx -t
ngx_http_fastdfs_set pid=240339
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
he@he-ThinkPad-X200:/etc/nginx$ sudo /usr/sbin/nginx -s reload
ngx_http_fastdfs_set pid=240342

4、最后需要拷贝fastdfs解压目录中的http.conf和mime.types:

he@he-ThinkPad-X200:~/fastdfs-5.12$ ls
client  conf             docker        HISTORY  INSTALL  php_client  restart.sh  storage  tracker
common  COPYING-3_0.txt  fastdfs.spec  init.d   make.sh  README.md   stop.sh     test
he@he-ThinkPad-X200:~/fastdfs-5.12$ cd ./conf/
he@he-ThinkPad-X200:~/fastdfs-5.12/conf$ sudo cp mime.types http.conf /etc/fdfs/
he@he-ThinkPad-X200:~/fastdfs-5.12/conf$ 

4.6.4 FastDFS常用命令测试

4.6.4.1 上传文件

he@he-ThinkPad-X200:~$ cd Downloads/
he@he-ThinkPad-X200:~/Downloads$ vim test2.txt
he@he-ThinkPad-X200:~/Downloads$ /usr/bin/fdfs_upload_file /etc/fdfs/client.conf test2.txt
group1/M00/00/00/wKgAZGCNZgiAGn6KAAAABTu5NcY033.txt
he@he-ThinkPad-X200:~/Downloads$ ls /home/he/fastdfs/storage/data/00/00
wKgAZGCNRu6AYR6pAAAABEeIgU4549.txt  wKgAZGCNZgiAGn6KAAAABTu5NcY033.txt

he@he-ThinkPad-X200:~$ wget http://127.0.0.1:8888/group1/M00/00/00/wKgAZGCNZgiAGn6KAAAABTu5NcY033.txt
--2021-05-01 22:43:02--  http://127.0.0.1:8888/group1/M00/00/00/wKgAZGCNZgiAGn6KAAAABTu5NcY033.txt  # 通过wget访问成功:
Connecting to 127.0.0.1:8888... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5 [text/plain]
Saving to: ‘wKgAZGCNZgiAGn6KAAAABTu5NcY033.txt’

wKgAZGCNZgiAGn6KAAAABTu5NcY033.t 100%[==========================================================>]       5  --.-KB/s    in 0s      

2021-05-01 22:43:02 (353 KB/s) - ‘wKgAZGCNZgiAGn6KAAAABTu5NcY033.txt’ saved [5/5]

# 也可通过浏览器方式访问

4.6.4.2 下载文件

he@he-ThinkPad-X200:~/Documents$ /usr/bin/fdfs_download_file /etc/fdfs/client.conf group1/M00/00/00/wKgAZGCNZgiAGn6KAAAABTu5NcY033.txt download.txt  # 命令:/usr/bin/fdfs_download_file <config_file> <file_id> [local_filename]
he@he-ThinkPad-X200:~/Documents$ ls
download.txt  pycharmActive.txt
he@he-ThinkPad-X200:~/Documents$ vi download.txt 

  • 命令:/usr/bin/fdfs_download_file <config_file> <file_id> [local_filename]

4.6.4.3 删除文件

he@he-ThinkPad-X200:~$ ls /home/he/fastdfs/storage/data/00/00
wKgAZGCNRu6AYR6pAAAABEeIgU4549.txt  wKgAZGCNZgiAGn6KAAAABTu5NcY033.txt
he@he-ThinkPad-X200:~$ /usr/bin/fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/00/wKgAZGCNZgiAGn6KAAAABTu5NcY033.txt  # 删除
he@he-ThinkPad-X200:~$ ls /home/he/fastdfs/storage/data/00/00
wKgAZGCNRu6AYR6pAAAABEeIgU4549.txt

  • 命令:/usr/bin/fdfs_delete_file <config_file> <file_id>


【部分内容参考自】

  • fastDFS:https://www.jianshu.com/p/b7c330a87855
  • 分布式文件系统FastDFS详解:https://baijiahao.baidu.com/s?id=1588881983988039766&wfr=spider&for=pc
  • 分布式文件系统FastDFS部署:https://blog.51cto.com/wangfeng7399/1711589
  • 分布式文件系统FastDFS安装教程:https://www.cnblogs.com/handsomeye/p/9451568.html
  • fastdfs-安装fastdfs-nginx-module和配置使用:https://www.cnblogs.com/cnxkey/articles/11121376.html
  • Nginx重新编译添加模块的方法:https://www.jb51.net/article/165079.htm

以上是关于实战学习开源项目学习之FastDFS部署的主要内容,如果未能解决你的问题,请参考以下文章

实战学习开源项目学习之Celery

NodeJs随心学习之UEditor开源项目部署

java学习之即时通信项目实战

licode学习之编译篇--1

licode学习之编译篇--1

python学习之自动化运维项目